14 votes

Kill chain - on the automated bureaucratic machinery that killed 175 children

10 comments

  1. [7]
    vord
    Link
    I've been pondering the nature of beuracracy recently, and I think this foments a key thesis: Beuracracies are not supposed to be efficient. The entire point of their existence is to inject a...
    • Exemplary

    I've been pondering the nature of beuracracy recently, and I think this foments a key thesis:

    Beuracracies are not supposed to be efficient. The entire point of their existence is to inject a control to stop an efficiency because the original method was not working as intended.

    I know there is the arguement that the goal of said beuracracy should be only doing the bare minimum to accomplish intended goals, but that is optimizing for the opposite goal of the beuracracy: To insure that processes themselves work as intended.

    It'd be way more efficient to have the FDA not force drug companies to triple-blind placebo trials. But that also compromises the entire purpose.

    It is better to have a fully-functional beuracracy that is horribily inefficient that works than an efficient one that murders people. The overcorrection to 'fix' inefficiency in beuracracy is worse than the symptom.

    It does not help matters that 'efficiency' is usually used as slang for 'cutting quality and safety to the bare minimum to make more money'.

    14 votes
    1. [6]
      CptBluebear
      Link Parent
      Hence fintech and modern tech companies trying to skirt bureaucracy by redefining what already exists in a different but regulated form. I'll be the first to admit that red tape can slow things...

      Hence fintech and modern tech companies trying to skirt bureaucracy by redefining what already exists in a different but regulated form.

      I'll be the first to admit that red tape can slow things down into inefficiency, or worse irrelevance, but friction is good. It turns the decision making into a process that should (theoretically) holistically look at the outcome and whether that's desirable. Somewhat utilitarian, it should even look at if it's desirable for most.

      I'm generally for more regulation, not less. Regulation stops prediction markets from deciding they're not gambling and prevents rideshare apps from saying that they're not taxis or have CEOs claim that buy now pay later is not the same as debt, for no discernable reason other than the fact it's now done through an app or something.

      12 votes
      1. [5]
        vord
        Link Parent
        There is also at the heart of it that often to make a beuracrcy more efficient for the users, you need more money. The DMV will be much faster if they have double the staff.

        There is also at the heart of it that often to make a beuracrcy more efficient for the users, you need more money.

        The DMV will be much faster if they have double the staff.

        8 votes
        1. [4]
          CptBluebear
          Link Parent
          Vord, reading your posts on Tildes I think you're insightful and intelligent, but it appears the word bureaucracy is your nemesis haha. To your point, I agree. Caveats applied and a little nuance...

          Vord, reading your posts on Tildes I think you're insightful and intelligent, but it appears the word bureaucracy is your nemesis haha.

          To your point, I agree. Caveats applied and a little nuance out of the window I'd go as far as to say that government bodies should continually grow more expensive overtime to get more efficient in the execution while keeping the friction in decision making.

          (I also think there's a time to rebalance the budget and see what spend is no longer necessary, though I want to say that's besides the point for now)

          7 votes
          1. [2]
            Greg
            (edited )
            Link Parent
            It’s not even that things need to be expensive per se - genuine efficiency should be cheaper, you just need to have a willingness to spend up front for savings later (putting capital into newer,...

            It’s not even that things need to be expensive per se - genuine efficiency should be cheaper, you just need to have a willingness to spend up front for savings later (putting capital into newer, better systems and tools when appropriate, for example) and an understanding that the savings might not be obvious or easy to quantify. Sometimes the payoff is clear - sometimes you really do just get more done in less time with a better tool - but a lot of the time you need to consider things like aggregate total hours wasted for staff and clients before you see that the overall time cost of having more staff is actually lower.

            Obviously businesses don’t care about wasting their customers’ time as long as it doesn’t impact profits, but governments should be accounting for that, and accounting for things like “how many hours of people’s lives does the tax money we’re spending represent, and how many hours can we save them by spending it?” when determining what the net cost of a project actually is. In reality we’re lucky to get an analysis that considers the concept of externalities at all in any meaningful way, let alone one that takes the next step and maps it back to intangible benefits rather than assuming monetary value is a benefit in and of itself and not just an approach to keeping score.

            [Edit] For what it’s worth, I’m wary of accepting friction in processes as a good thing - inaction can be just as harmful as action, but people tend to have a bias towards doing harm by inaction because it feels like they’re not actually doing harm by being passive - so I’m more inclined towards the “do more, faster, just set your goals and incentives properly” approach in an ideal world. But compared to the status quo of “fast with actively bad incentives” when businesses get involved, I can understand the preference.

            5 votes
            1. CptBluebear
              Link Parent
              I wrote a reply but my computer froze and I lost it. I don't have the time to write another full reply, apologies, so I'll keep it short. Governmental bodies need to be quick to react, but slow to...

              I wrote a reply but my computer froze and I lost it. I don't have the time to write another full reply, apologies, so I'll keep it short.

              Governmental bodies need to be quick to react, but slow to act. Prevent or correct destabilizing occurrences fast, but introduce change and destabilizing elements slowly.

              “do more, faster, just set your goals and incentives properly"

              I agree, scope creep is the silent killer of any project, but their incentives should always be to make things as beneficial as possible for all. Which means taking the time to analyze all second and third order effects you didn't see coming.

              2 votes
          2. vord
            Link Parent
            Spelling was always the tool in my toolbelt that came from the 99c store.

            Spelling was always the tool in my toolbelt that came from the 99c store.

            3 votes
  2. skybrian
    Link
    From the article: [...] [...]

    From the article:

    After Google abandoned the Maven contract in 2018, Palantir took it over. In 2020, the XVIII Airborne Corps began testing the system in an exercise called “Scarlet Dragon,” which started as a tabletop wargaming exercise in a windowless basement at Fort Bragg.12 Its commander, Lieutenant General Michael Erik Kurilla, wanted to build what he called the first “AI-enabled Corps” in the Army.13 The goal was to test whether the system could give a small team the targeting capacity of a full theater operation. Over the next five years, Scarlet Dragon grew through more than ten iterations into a joint live-fire exercise spanning multiple states, with “forward-deployed engineers” from Palantir and other contractors embedded alongside soldiers.14 Each iteration was meant to provide an answer to the same question: how fast could the system move from detection to decision. The benchmark was the 2003 invasion of Iraq, where roughly two thousand people worked the targeting process for the entire theater.15 During Scarlet Dragon, twenty soldiers using Maven handled the same volume of work. By 2024, the stated goal was a thousand targeting decisions in an hour. That is 3.6 seconds per decision, or from the individual “targeteer’s” perspective, one decision every 72 seconds.

    The Maven Smart System is the platform that came out of those exercises, and it, not Claude, is what is being used to produce “target packages” in Iran. There are real limits to what a civilian like myself can know about this system, and what follows is based on publicly-available information, assembled from Palantir product demos, conferences, as well as instructional material produced for military users. But we can know quite a bit. The interface looks like a tacticool, dark mode send-up of enterprise software paired with the features of geospatial application like ArcGIS. What the operator sees are either maps with GIS-like overlays or a screen organized like a project management board. There are columns representing stages of the targeting process, with individual targets moving across them from left to right, as in a Kanban board.

    Before Maven, operators worked across eight or nine separate systems simultaneously, pulling data from one, cross-referencing in another, manually moving detections between platforms to build a targeting case. Maven consolidated and orchestrated all of these behind a single interface. Cameron Stanley, the Pentagon’s chief digital and AI officer, called it an “abstraction layer,” a common term in software engineering, meaning a system which hides the complexity underneath it.16 Humans run the targeting and the ML systems underneath produce confidence intervals. Three clicks convert a data point on the map into a formal detection and move it into a targeting pipeline. These targets then move through columns representing different decision-making processes and rules of engagement. The system evaluates factors and presents ranked options for which platform and munition to assign, what the military calls a Course of Action. The officer selects from the ranked options, and the system, depending on who is using it, either sends the target package to an officer for approval or moves it to execution.

    [...]

    Clausewitz had a word for everything the optimization leaves out. He called it “friction,” the accumulation of uncertainty, error, and contradiction that ensures no operation goes as planned. But friction is also where judgment forms. Clausewitz observed that most intelligence is false, that reports contradict each other. The commander who has worked through this learns to see the way an eye adjusts to darkness, not by getting better light but by staying long enough to use what light there is. The staying is what takes time. Compress the time and the friction does not disappear. You just stop noticing it. Clausewitz called what unfolds when you refused to notice a “war on paper,” a plan that proceeds without resistance because everything that connected it to the world it was supposed to act on has been taken out.28

    Air power is uniquely vulnerable to this. The pilot never sees what the bomb hits. The analyst works from imagery, coordinates, databases. The entire enterprise is mediated by representations of the target, not the target itself, which means the gap between the package and the world can widen without anyone in the process feeling it. The 2003 invasion of Iraq, the operation that Scarlet Dragon would later use as its benchmark, was a case in point. Marc Garlasco, the Pentagon’s chief of high-value targeting during the 2003 invasion of Iraq, ran the fastest targeting cycle the US had operated to that point. He recommended fifty leadership strikes. The bombs were precise. The intelligence behind them was not. None of the fifty killed its intended target. Two weeks after the invasion, Garlasco left the Pentagon for Human Rights Watch, went to Iraq, and stood in the crater of a strike he had targeted himself. “These aren’t just nameless, faceless targets,” he said later. “This is a place where people are going to feel ramifications for a long time.”29 The targeting cycle had been fast enough to hit fifty buildings and too fast to discover it was hitting the wrong ones.

    [...]

    Organizations that run on formal procedure need someone inside the process to interpret the rules, notice exceptions, recognize when the categories no longer fit the case. But the procedural form cannot admit this. If the organization concedes that its outcomes depend on the discretion of the people executing it, then the procedure is not a procedure but a suggestion, and the authority the organization derives from appearing rule-governed collapses. So the judgment has to happen, and it has to look like something else. It has to look like following the procedure rather than interpreting it. I’ve come to think of this as the “bureaucratic double bind,” the organization cannot function without the judgment, and it cannot acknowledge the judgment without undermining itself and being seen as “political.” One solution to this problem is replace the judgment with a number. Theodore Porter, in Trust in Numbers (1995), argued that organizations adopt quantitative rules not because numbers are more accurate but because they are more defensible.36 Judgment is politically vulnerable. Rules are not. The procedure exists to make discretion disappear, or seem to. The system’s actual flexibility lives entirely in this unacknowledged interpretive work, which means it can be removed by anyone who mistakes it for inefficiency.

    13 votes
  3. [2]
    R3qn65
    Link
    This is really interesting and quite good, but nevertheless I have some nits to pick. It's a bit silly to cite bombs not killing their targets in the Iraq war: when the US and Israel have...

    This is really interesting and quite good, but nevertheless I have some nits to pick.

    It's a bit silly to cite bombs not killing their targets in the Iraq war:

    Marc Garlasco, the Pentagon’s chief of high-value targeting during the 2003 invasion of Iraq, ran the fastest targeting cycle the US had operated to that point. He recommended fifty leadership strikes. The bombs were precise. The intelligence behind them was not. None of the fifty killed its intended target. Two weeks after the invasion, Garlasco left the Pentagon for Human Rights Watch, went to Iraq, and stood in the crater of a strike he had targeted himself. “These aren’t just nameless, faceless targets,” he said later. “This is a place where people are going to feel ramifications for a long time.”29 The targeting cycle had been fast enough to hit fifty buildings and too fast to discover it was hitting the wrong ones.

    when the US and Israel have successfully killed multiple leadership targets in this war. "It didn't work then" is not compelling evidence for why it can't work now, especially when by all appereances it is working now.

    The author's not wrong to cite the Chinese embassy bombing in the 90s or the bombing of the girl's school just a few weeks ago, but those tragedies don't mean that targeting is fundamentally ineffective. If anything, the fact that NATO struck hundreds of targets, including the embassy, in the 90s and the US struck thousands, including the school, this year would suggest that things are getting better.

    The article's thesis is, basically, that targeting has sped up so much that there's no time for judgement anymore.

    Clausewitz had a word for everything the optimization leaves out. He called it “friction,” the accumulation of uncertainty, error, and contradiction that ensures no operation goes as planned. But friction is also where judgment forms... The staying is what takes time. Compress the time and the friction does not disappear. You just stop noticing it.

    But what the author acknowledges but doesn't really, I don't know, internalize, is that almost all of what Maven eliminated isn't the judgement sort of friction, it's just lag. As s/he notes, analysts used to manually combine products from multiple different databases and now they don't. That's the biggest change, and it doesn't materially impact judgement.

    It's not like human judgement never struck the wrong target. :(

    2 votes
    1. skybrian
      Link Parent
      My guess is that the accuracy of US bombing has probably gotten better, while at the same time our standards have risen so that we are less tolerant of mistakes. Rising standards doesn't seem like...

      My guess is that the accuracy of US bombing has probably gotten better, while at the same time our standards have risen so that we are less tolerant of mistakes. Rising standards doesn't seem like a bad thing?

      Sadly, leadership quality has become considerably worse.

      2 votes