23 votes

Kill chain - on the automated bureaucratic machinery that killed 175 children

24 comments

  1. [17]
    vord
    Link
    I've been pondering the nature of beuracracy recently, and I think this foments a key thesis: Beuracracies are not supposed to be efficient. The entire point of their existence is to inject a...
    • Exemplary

    I've been pondering the nature of beuracracy recently, and I think this foments a key thesis:

    Beuracracies are not supposed to be efficient. The entire point of their existence is to inject a control to stop an efficiency because the original method was not working as intended.

    I know there is the arguement that the goal of said beuracracy should be only doing the bare minimum to accomplish intended goals, but that is optimizing for the opposite goal of the beuracracy: To insure that processes themselves work as intended.

    It'd be way more efficient to have the FDA not force drug companies to triple-blind placebo trials. But that also compromises the entire purpose.

    It is better to have a fully-functional beuracracy that is horribily inefficient that works than an efficient one that murders people. The overcorrection to 'fix' inefficiency in beuracracy is worse than the symptom.

    It does not help matters that 'efficiency' is usually used as slang for 'cutting quality and safety to the bare minimum to make more money'.

    24 votes
    1. [6]
      CptBluebear
      Link Parent
      Hence fintech and modern tech companies trying to skirt bureaucracy by redefining what already exists in a different but regulated form. I'll be the first to admit that red tape can slow things...

      Hence fintech and modern tech companies trying to skirt bureaucracy by redefining what already exists in a different but regulated form.

      I'll be the first to admit that red tape can slow things down into inefficiency, or worse irrelevance, but friction is good. It turns the decision making into a process that should (theoretically) holistically look at the outcome and whether that's desirable. Somewhat utilitarian, it should even look at if it's desirable for most.

      I'm generally for more regulation, not less. Regulation stops prediction markets from deciding they're not gambling and prevents rideshare apps from saying that they're not taxis or have CEOs claim that buy now pay later is not the same as debt, for no discernable reason other than the fact it's now done through an app or something.

      19 votes
      1. [5]
        vord
        Link Parent
        There is also at the heart of it that often to make a beuracrcy more efficient for the users, you need more money. The DMV will be much faster if they have double the staff.

        There is also at the heart of it that often to make a beuracrcy more efficient for the users, you need more money.

        The DMV will be much faster if they have double the staff.

        12 votes
        1. [4]
          CptBluebear
          Link Parent
          Vord, reading your posts on Tildes I think you're insightful and intelligent, but it appears the word bureaucracy is your nemesis haha. To your point, I agree. Caveats applied and a little nuance...

          Vord, reading your posts on Tildes I think you're insightful and intelligent, but it appears the word bureaucracy is your nemesis haha.

          To your point, I agree. Caveats applied and a little nuance out of the window I'd go as far as to say that government bodies should continually grow more expensive overtime to get more efficient in the execution while keeping the friction in decision making.

          (I also think there's a time to rebalance the budget and see what spend is no longer necessary, though I want to say that's besides the point for now)

          10 votes
          1. [2]
            Greg
            (edited )
            Link Parent
            It’s not even that things need to be expensive per se - genuine efficiency should be cheaper, you just need to have a willingness to spend up front for savings later (putting capital into newer,...

            It’s not even that things need to be expensive per se - genuine efficiency should be cheaper, you just need to have a willingness to spend up front for savings later (putting capital into newer, better systems and tools when appropriate, for example) and an understanding that the savings might not be obvious or easy to quantify. Sometimes the payoff is clear - sometimes you really do just get more done in less time with a better tool - but a lot of the time you need to consider things like aggregate total hours wasted for staff and clients before you see that the overall time cost of having more staff is actually lower.

            Obviously businesses don’t care about wasting their customers’ time as long as it doesn’t impact profits, but governments should be accounting for that, and accounting for things like “how many hours of people’s lives does the tax money we’re spending represent, and how many hours can we save them by spending it?” when determining what the net cost of a project actually is. In reality we’re lucky to get an analysis that considers the concept of externalities at all in any meaningful way, let alone one that takes the next step and maps it back to intangible benefits rather than assuming monetary value is a benefit in and of itself and not just an approach to keeping score.

            [Edit] For what it’s worth, I’m wary of accepting friction in processes as a good thing - inaction can be just as harmful as action, but people tend to have a bias towards doing harm by inaction because it feels like they’re not actually doing harm by being passive - so I’m more inclined towards the “do more, faster, just set your goals and incentives properly” approach in an ideal world. But compared to the status quo of “fast with actively bad incentives” when businesses get involved, I can understand the preference.

            8 votes
            1. CptBluebear
              Link Parent
              I wrote a reply but my computer froze and I lost it. I don't have the time to write another full reply, apologies, so I'll keep it short. Governmental bodies need to be quick to react, but slow to...

              I wrote a reply but my computer froze and I lost it. I don't have the time to write another full reply, apologies, so I'll keep it short.

              Governmental bodies need to be quick to react, but slow to act. Prevent or correct destabilizing occurrences fast, but introduce change and destabilizing elements slowly.

              “do more, faster, just set your goals and incentives properly"

              I agree, scope creep is the silent killer of any project, but their incentives should always be to make things as beneficial as possible for all. Which means taking the time to analyze all second and third order effects you didn't see coming.

              5 votes
          2. vord
            Link Parent
            Spelling was always the tool in my toolbelt that came from the 99c store.

            Spelling was always the tool in my toolbelt that came from the 99c store.

            8 votes
    2. [10]
      Grzmot
      Link Parent
      Efficiency and governmental murder are not related to each other. To pitch an inefficient beraucracy with its many frictional points as solution is the wrong approach. The (actual, WW2) Nazi...

      Efficiency and governmental murder are not related to each other. To pitch an inefficient beraucracy with its many frictional points as solution is the wrong approach. The (actual, WW2) Nazi beraucracy was horridly inefficient. It featured a staggering amount of parallel offices doing nearly the same thing, sometimes even identical, reporting to the same people, and that did not stop said empire from committing one of the worst atrocities seen in human history. In fact, there are arguments to be made that the parallel offices working in competition to each other created a fucked up free market type of competition where the goal was not to make the most money but kill the most "undesirables".

      Beraucratic friction is only sensible when the human, frictional point actually has the power to increase the friction and force a discussion if what is being done makes any sort of sense. The friction by itself is not a saving grace that stops tragedies.

      This might besides the point, because the essay in question specifically discusses beraucracy within a military kill chain, but your comment reads to me as a more general pondering. So please note that my following argument is general as well.

      There are many tasks which are necessary within a beraucracy but, especially when it comes to how people interact with the state, are rooted in such frictions which do not serve any point. They once did, because the best way to serve a need was to put a clerk into an office and have them handle citizen requests. But essentially, this clerk was the human interface to a (once) analogue database, which today is digital. This is an inefficiency which can be removed without increasing human suffering. In fact I'd argue that it reduces it, because everyone I know hates having to interact with the local county office, and it's not a stereotypical job people find personal fulfillment in.

      A lot of interactions with the government from the point of view of a private citizen are really just updating a database somewhere. It might be registering where you live (a European concept that has shocked many Americans I told this to), renewing some ID or license, or updating some other official record related to your person. A necessary task for any state. This is the typical case where the human doing this work behind a counter is really just receiving your orders and executing them by typing them into some UI, which then sends this request off to some central database. But you yourself could also do this. There is no inherent value to the friction of involving another instance, another human in this process.

      6 votes
      1. [7]
        DefinitelyNotAFae
        Link Parent
        You're supposed to register where you live in the US, updating your address with the state when you move, even intrastate, is normal. Some folks definitely don't do it immediately or until there's...

        You're supposed to register where you live in the US, updating your address with the state when you move, even intrastate, is normal. Some folks definitely don't do it immediately or until there's a need for a new ID or car registration, but it's not that odd (unless my state is some sort of extreme outlier) We just use different language for it, I guess? I don't know why people are weird about it but Americans give the government their address all the time.

        6 votes
        1. [6]
          sparksbet
          Link Parent
          As someone who is in the process of moving back to the US and thus handling these kinds of details, there is nothing remotely equivalent to registering the way you do in Germany (I can't speak for...

          As someone who is in the process of moving back to the US and thus handling these kinds of details, there is nothing remotely equivalent to registering the way you do in Germany (I can't speak for other countries). You often need to include proof of residence when registering for other things, like your driver's license or to vote or for social benefits, but you can generally use a variety of things to prove you live at an address, such as utility bills. There is nothing equivalent to just registering that you live at an address the way that is required in Germany, and you likewise do not receive a proof of registration to use for demonstrating you've registered your address when you register for other things that depends on your residency. There's an entire step in the process left out in the US (albeit a very unnecessary step imo, so I'll hardly miss it).

          If there is something like that where you live, it's state- or region-specific for you, because it definitely doesn't exist in my part of the US.

          1 vote
          1. [5]
            DefinitelyNotAFae
            Link Parent
            It sounds like the difference is that other agencies/businesses don't use the official registration as proof of address? Most states do require you to update your address within like 30 days or...

            It sounds like the difference is that other agencies/businesses don't use the official registration as proof of address?

            Most states do require you to update your address within like 30 days or less of moving with the DMV (In Illinois this applies to state IDs as well but I did not check every state on that one your state does require it for drivers at least) I get the intensity is different, , but it is law in the majority of states that you're supposed to do it.

            Probably because it isn't used the way it seems to be in Germany. It's not really checked up on unless you're having interactions with police or needing official documents from the state

            1 vote
            1. [4]
              sparksbet
              Link Parent
              Updating your other documentation (like driver's licenses and the like) is completely independent of registering your address in the sense described here, and one doesn't have to do it if you...

              Updating your other documentation (like driver's licenses and the like) is completely independent of registering your address in the sense described here, and one doesn't have to do it if you don't have a driver's license or state ID. You need to register your address in Germany even as someone without any other state identification, which is very much not a thing in the US. Even foreigners without visas are required to register their addresses (ostensibly within two weeks of moving into an address, but in practice getting an appointment usually involves a longer delay in my experience) when they're staying even medium-term. This makes the way US states handle updating your address in other systems (which afaik you typically need to independently update each relevant one) different from directly registering your address the way is required in Germany, which is not only required separately from any other identification you have but is generally a prerequisite to applying for other things like a driver's license. You also need to update your address on other identification in Germany (and in a show of German efficiency this also generally entails independently updating it in several independent state systems that don't talk to each other). This is independent of registering your address itself. You also need to *de-*register when you leave the country. It simply is not accurate that US states in general have any equivalent to this part of the process (and fwiw, this process is entirely unnecessary imo, so that's a good thing imo). The difference is quite clear when you have experience with both systems.

              2 votes
              1. [3]
                DefinitelyNotAFae
                Link Parent
                I get its different in intensity and process; I'm just saying I think the reaction from Americans to simply being told about registering one's address with the government is weird. Because while...

                I get its different in intensity and process; I'm just saying I think the reaction from Americans to simply being told about registering one's address with the government is weird. Because while different it's still a thing the vast majority of adult Americans are required to do.

                1. [2]
                  sparksbet
                  Link Parent
                  In my experience most US expats do find it weird to separately register your address with the government the way it's required here, due to how different it is compared to how it works in the US.

                  In my experience most US expats do find it weird to separately register your address with the government the way it's required here, due to how different it is compared to how it works in the US.

                  1 vote
                  1. DefinitelyNotAFae
                    Link Parent
                    Sure, I get finding the specifics of it different or whatever, I'm just saying that my response to the previous poster mentioning "registering my address with the government" was "yeah I gotta...

                    Sure, I get finding the specifics of it different or whatever, I'm just saying that my response to the previous poster mentioning "registering my address with the government" was "yeah I gotta update mine with the state when I move too" not being shocked. Maybe shocked at how difficult it is? Maybe?

                    But perhaps it was provided with all the additional context you're sharing. Idk I still find the reaction weird though.

      2. [2]
        vord
        Link Parent
        Self-service sucks. It's fine at a grocery store, wherr you only have a small handful. But if you have a full cart, a cashier + bagger will be exponentially faster. 90% of the difficulty in...

        Self-service sucks. It's fine at a grocery store, wherr you only have a small handful. But if you have a full cart, a cashier + bagger will be exponentially faster.

        90% of the difficulty in database work isn't standard cases... even in some of the worst systems these tend to be well refined.

        Having a single person who has been working with this stuff for decades can easily process someone through 3x faster than having a person who does it once every decade via self-service. Especially if there is the slightest edge case.

        And that's ultimately what the current state of every DMV (the gold standard for hated paperwork pushers) is: Simple stuff can be done online. And all the other stuff needs a human; validating authenticity of documents, weird edge cases, photo taking, etc.

        The problem boils down to organizations using the improvements to the standard path to justify cutting staff, which makes the DMV worse because for the most part, at best lines stay the same, but usually waits get worse because it was already the edge cases consuming all the time.

        DOGE closed a bunch of SSA local branches using the justifications you describe, and it's made everything worse because people going to them were already 100% unable to self-service.

        Use efficiency to improve service. Not lower costs. Worst case, your staff hates their job less.

        4 votes
        1. Grzmot
          Link Parent
          Of course the most difficulty in interacting with a database lie with the edge cases. Any system is tested by those. But 90% of the actual interactions with a system are not the edge cases. Those...

          Of course the most difficulty in interacting with a database lie with the edge cases. Any system is tested by those. But 90% of the actual interactions with a system are not the edge cases. Those are called that for a reason.

          The thing about going to the local office to get something done is that it's essentially still self-service. You're just talking to another human who than actually executes the self-service task in your name. What I mean that there is rarely a particular skill or knowledge involved that you do not have. In most of these tasks, there is little inherent knowledge required other than clicking the buttons that are labeled accordingly and filling out forms in some UI. It's not like going to the mechanic, or a doctor, where you are interacting with a human who has skills you do not posess.

          Unless the processes are oblique and cumbersome by design, or because they've grown organically over the decades and are often redundant and overly complex, the clerk at the county office doesn't really do much else. Of course if you live in a system where you need to do something, but that requires filling out 15 different forms, then yeah, of course the assistance of someone who knows their shit is appreciated. But that's not what most people need. And even complex processes can be simplified. My country recently launched a pilot project where individuals can found companies through a simple web UI, which generates valid contracts and handles a lot of complicated nuances. There are limitations, it's meant for self-employed people, but it's still a welcome simplification that covers the needs of a lot of people.

          Our arguments aren't at odds with each other, though. I think what you describe is real, and especially for folks who live in the US in the aftermath of DOGE, talking about efficiency gains is going to carry with it a foul taste.

          I don't think that there's anything wrong with having a self-service app, UI, webpage whatever, that offers you list of actions that cover the needs of most people interacting with the system, and then a way to contact some help office that has people who are actually skilled and knowledgeable and can help you solve your case. But like you say, that requires that effiency gains aren't used to fire people, they're used to train them so they are at the citizen's disposal should they need to cover an edge case.

  2. skybrian
    Link
    From the article: [...] [...]

    From the article:

    After Google abandoned the Maven contract in 2018, Palantir took it over. In 2020, the XVIII Airborne Corps began testing the system in an exercise called “Scarlet Dragon,” which started as a tabletop wargaming exercise in a windowless basement at Fort Bragg.12 Its commander, Lieutenant General Michael Erik Kurilla, wanted to build what he called the first “AI-enabled Corps” in the Army.13 The goal was to test whether the system could give a small team the targeting capacity of a full theater operation. Over the next five years, Scarlet Dragon grew through more than ten iterations into a joint live-fire exercise spanning multiple states, with “forward-deployed engineers” from Palantir and other contractors embedded alongside soldiers.14 Each iteration was meant to provide an answer to the same question: how fast could the system move from detection to decision. The benchmark was the 2003 invasion of Iraq, where roughly two thousand people worked the targeting process for the entire theater.15 During Scarlet Dragon, twenty soldiers using Maven handled the same volume of work. By 2024, the stated goal was a thousand targeting decisions in an hour. That is 3.6 seconds per decision, or from the individual “targeteer’s” perspective, one decision every 72 seconds.

    The Maven Smart System is the platform that came out of those exercises, and it, not Claude, is what is being used to produce “target packages” in Iran. There are real limits to what a civilian like myself can know about this system, and what follows is based on publicly-available information, assembled from Palantir product demos, conferences, as well as instructional material produced for military users. But we can know quite a bit. The interface looks like a tacticool, dark mode send-up of enterprise software paired with the features of geospatial application like ArcGIS. What the operator sees are either maps with GIS-like overlays or a screen organized like a project management board. There are columns representing stages of the targeting process, with individual targets moving across them from left to right, as in a Kanban board.

    Before Maven, operators worked across eight or nine separate systems simultaneously, pulling data from one, cross-referencing in another, manually moving detections between platforms to build a targeting case. Maven consolidated and orchestrated all of these behind a single interface. Cameron Stanley, the Pentagon’s chief digital and AI officer, called it an “abstraction layer,” a common term in software engineering, meaning a system which hides the complexity underneath it.16 Humans run the targeting and the ML systems underneath produce confidence intervals. Three clicks convert a data point on the map into a formal detection and move it into a targeting pipeline. These targets then move through columns representing different decision-making processes and rules of engagement. The system evaluates factors and presents ranked options for which platform and munition to assign, what the military calls a Course of Action. The officer selects from the ranked options, and the system, depending on who is using it, either sends the target package to an officer for approval or moves it to execution.

    [...]

    Clausewitz had a word for everything the optimization leaves out. He called it “friction,” the accumulation of uncertainty, error, and contradiction that ensures no operation goes as planned. But friction is also where judgment forms. Clausewitz observed that most intelligence is false, that reports contradict each other. The commander who has worked through this learns to see the way an eye adjusts to darkness, not by getting better light but by staying long enough to use what light there is. The staying is what takes time. Compress the time and the friction does not disappear. You just stop noticing it. Clausewitz called what unfolds when you refused to notice a “war on paper,” a plan that proceeds without resistance because everything that connected it to the world it was supposed to act on has been taken out.28

    Air power is uniquely vulnerable to this. The pilot never sees what the bomb hits. The analyst works from imagery, coordinates, databases. The entire enterprise is mediated by representations of the target, not the target itself, which means the gap between the package and the world can widen without anyone in the process feeling it. The 2003 invasion of Iraq, the operation that Scarlet Dragon would later use as its benchmark, was a case in point. Marc Garlasco, the Pentagon’s chief of high-value targeting during the 2003 invasion of Iraq, ran the fastest targeting cycle the US had operated to that point. He recommended fifty leadership strikes. The bombs were precise. The intelligence behind them was not. None of the fifty killed its intended target. Two weeks after the invasion, Garlasco left the Pentagon for Human Rights Watch, went to Iraq, and stood in the crater of a strike he had targeted himself. “These aren’t just nameless, faceless targets,” he said later. “This is a place where people are going to feel ramifications for a long time.”29 The targeting cycle had been fast enough to hit fifty buildings and too fast to discover it was hitting the wrong ones.

    [...]

    Organizations that run on formal procedure need someone inside the process to interpret the rules, notice exceptions, recognize when the categories no longer fit the case. But the procedural form cannot admit this. If the organization concedes that its outcomes depend on the discretion of the people executing it, then the procedure is not a procedure but a suggestion, and the authority the organization derives from appearing rule-governed collapses. So the judgment has to happen, and it has to look like something else. It has to look like following the procedure rather than interpreting it. I’ve come to think of this as the “bureaucratic double bind,” the organization cannot function without the judgment, and it cannot acknowledge the judgment without undermining itself and being seen as “political.” One solution to this problem is replace the judgment with a number. Theodore Porter, in Trust in Numbers (1995), argued that organizations adopt quantitative rules not because numbers are more accurate but because they are more defensible.36 Judgment is politically vulnerable. Rules are not. The procedure exists to make discretion disappear, or seem to. The system’s actual flexibility lives entirely in this unacknowledged interpretive work, which means it can be removed by anyone who mistakes it for inefficiency.

    14 votes
  3. [3]
    R3qn65
    Link
    This is really interesting and quite good, but nevertheless I have some nits to pick. It's a bit silly to cite bombs not killing their targets in the Iraq war: when the US and Israel have...

    This is really interesting and quite good, but nevertheless I have some nits to pick.

    It's a bit silly to cite bombs not killing their targets in the Iraq war:

    Marc Garlasco, the Pentagon’s chief of high-value targeting during the 2003 invasion of Iraq, ran the fastest targeting cycle the US had operated to that point. He recommended fifty leadership strikes. The bombs were precise. The intelligence behind them was not. None of the fifty killed its intended target. Two weeks after the invasion, Garlasco left the Pentagon for Human Rights Watch, went to Iraq, and stood in the crater of a strike he had targeted himself. “These aren’t just nameless, faceless targets,” he said later. “This is a place where people are going to feel ramifications for a long time.”29 The targeting cycle had been fast enough to hit fifty buildings and too fast to discover it was hitting the wrong ones.

    when the US and Israel have successfully killed multiple leadership targets in this war. "It didn't work then" is not compelling evidence for why it can't work now, especially when by all appereances it is working now.

    The author's not wrong to cite the Chinese embassy bombing in the 90s or the bombing of the girl's school just a few weeks ago, but those tragedies don't mean that targeting is fundamentally ineffective. If anything, the fact that NATO struck hundreds of targets, including the embassy, in the 90s and the US struck thousands, including the school, this year would suggest that things are getting better.

    The article's thesis is, basically, that targeting has sped up so much that there's no time for judgement anymore.

    Clausewitz had a word for everything the optimization leaves out. He called it “friction,” the accumulation of uncertainty, error, and contradiction that ensures no operation goes as planned. But friction is also where judgment forms... The staying is what takes time. Compress the time and the friction does not disappear. You just stop noticing it.

    But what the author acknowledges but doesn't really, I don't know, internalize, is that almost all of what Maven eliminated isn't the judgement sort of friction, it's just lag. As s/he notes, analysts used to manually combine products from multiple different databases and now they don't. That's the biggest change, and it doesn't materially impact judgement.

    It's not like human judgement never struck the wrong target. :(

    4 votes
    1. papasquat
      Link Parent
      Zooming out, efforts to improve targeting and reduce collateral damage are worthwhile, but I think we miss the forest for the trees sometimes when we decide to kill people in other countries. This...

      Zooming out, efforts to improve targeting and reduce collateral damage are worthwhile, but I think we miss the forest for the trees sometimes when we decide to kill people in other countries.

      This is a war. In every war since the beginning of time, innocent, uninvolved people are violently killed, often at a much higher rate than combatants. No matter how many times you say "surgical strikes" or "precision targeting", it will not change this fact.

      Thus, every time a leader of the country makes the decision to bomb, invade, drone strike, or fire at another country, they, and the people who support them need to ask themselves "is whatever we're trying to do worth us killing hundreds or thousands or more innocent people that are just living their lives as I am".

      Convincing yourself that this time, only the "bad guys" will die is just gross willful self delusion to soothe one's own cognitive dissonance.

      5 votes
    2. skybrian
      Link Parent
      My guess is that the accuracy of US bombing has probably gotten better, while at the same time our standards have risen so that we are less tolerant of mistakes. Rising standards doesn't seem like...

      My guess is that the accuracy of US bombing has probably gotten better, while at the same time our standards have risen so that we are less tolerant of mistakes. Rising standards doesn't seem like a bad thing?

      Sadly, leadership quality has become considerably worse.

      4 votes
  4. [2]
    Deely
    Link
    Let's apply agile to a killing. Welcome to fast paced environment, where you will thrive under pressure.

    Let's apply agile to a killing. Welcome to fast paced environment, where you will thrive under pressure.

    2 votes
    1. skybrian
      Link Parent
      Yes, that's pretty much what war is. Add to that, you will be killed if you make a mistake, or perhaps at random. (See Ukraine.) Still, it's possible to do better.

      Yes, that's pretty much what war is. Add to that, you will be killed if you make a mistake, or perhaps at random. (See Ukraine.)

      Still, it's possible to do better.

  5. balooga
    Link
    This is phenomenally researched. I don’t think I’ve seen a better writeup about Maven, Claude’s actual relationship with it, and the historical and bureaucratic context into which they’re being...

    This is phenomenally researched. I don’t think I’ve seen a better writeup about Maven, Claude’s actual relationship with it, and the historical and bureaucratic context into which they’re being deployed. For some reason I’m really curious about the names of those Kanban swim lanes.

    Also, oh boy do people ever have trouble spelling the word “bureaucracy.” I tease!

    2 votes