10 votes

Talking to the Bank of England about systemic risk and systems engineering

1 comment

  1. skybrian
    Link
    Looks like Patrick McKenzie got a chance to give a talk to people at the Bank of England. It's very long, but there's some interesting stuff in there on a three different topics. CrowdStrike He...

    Looks like Patrick McKenzie got a chance to give a talk to people at the Bank of England. It's very long, but there's some interesting stuff in there on a three different topics.

    CrowdStrike

    He talked at length about last year's CrowdStrike incident:

    I understand one of the largest financial institutions in the world to have entirely grounded their tellers for the entire day of Friday due to the inability of their tellers to log into their systems because the systems were completely down—Blue Screen of Death. And that meant that all bank functions which were gated on speaking to a teller could not happen.

    The most consequential of those functions on a Friday is large cash withdrawals, particularly by business customers. Many people may not appreciate that businesses still do large cash withdrawals to fund payday. [...] there are particular subsectors of the economy which are still heavily cash-based.

    For example, in my hometown of Chicago, for a variety of reasons, the construction industry tends to be heavily cash-based. [...]

    ...

    The Wall Street Journal reported Chase, Bank of America, Wells Fargo, and TD Bank as being heavily impacted by the CrowdStrike Falcon issue. Those account for a large share of all deposits. And in many markets, when those banks need to turn customers away because their tellers are unable to service them, that large portion of the deposit business of all banks in the city immediately falls over to the large regional banks that are in the city and the smaller financial institutions that are in the city. And those banks do not have enough standing capacity to quickly redirect all the [...] withdrawal business of the largest banks in the nation.

    So even if their bank did not personally have CrowdStrike Falcon installed, they had a very similar situation where "our tellers are unable to do withdrawal transactions because we have physically depleted just the amount of physical U.S. specie that we have on the premises, because we have the only money that is available in the city of Chicago or any other city in the United States right now."

    ...

    I collected a series of anecdotes from small financial institutions all across Chicago: "We didn't even understand why it happened, but by 10 a.m., we were out of dollars."

    Now, fortuitously, many of the automated systems for banking continued operation throughout the duration of this. They did not have the endpoint monitoring software installed. They ran on Linux systems, etc. And so the thing that banks were told to do by higher-ups, the thing bank branch staff were told to do, was to redirect customers from the teller transaction which they wanted to do and which would fail to: "If you can use the website right now or use the ATM right now, that will work for you. And so if you were going to pay someone via cash, could you try paying them via Zelle instead, because Zelle is still up?"

    ...

    I want to emphasize this: This was a near miss, and it was a near miss including for you folks in the United Kingdom.

    ...

    This happened on a randomly selected Friday, and that was astoundingly good luck for us. We might experience a technical crisis when the broader economic and business environment weather is not normal.

    Stablecoins

    He goes on to talk about stablecoins (apparently because these are topics they asked for):

    [A report about stablecoins] shows almost 3x growth in volumes from approximately $2 billion a month to approximately $6 billion a month in the last two years.

    This is the first time in 15 years of being a diligent skeptic that I've seen something in the crypto industry and said, "Wow, that almost looks like there's a there there."

    ...

    When you just aggregate the numbers, a large portion of the growth is coming from B2B payments, which went up from a negligible base of under $100 million a month in January of 2023 to over $3 billion a month by early 2025.

    Interestingly, they say that the average stablecoin payment is approximately $200,000, which might inform you as to what these payments are being used for. That's likely to be small firms buying on the order of one shipping container at a time from China. Or, well, the cryptocurrency community will be slightly vaguer as to where the money is going, but without loss of generality, it's going to China.

    ...

    If you are a small trader or a small business and you attempt to wire $200,000 to a new counterparty physically located in China, your bank is going to have all sorts of questions. And that wire might be held up in review for weeks. The factory is unlikely to release a shipment to you while the wire has not actually cleared. And so your shipment might be weeks delayed to the loading dock and weeks delayed off the ship and weeks delayed to your facility and eventually to your customers.

    Stablecoins, on the other hand, transfer in seconds. And there is, in many deployments, essentially nobody who is doing a real-time human check on who you're sending the money to.

    But compared to other payment systems, this isn't all that impressive:

    Here's the monthly payment volumes denominated in U.S. dollars, billions, in mid-2025:

    • The entire stablecoin industry, those 34 firms all wrapped up together: about $6.3 billion a month. Congratulations, that's nice business.
    • PayPay: about $10 billion a month. That's a good business.
    • And then Pix in Brazil does $90 billion a month only on person-to-business transactions—the classic payments use case.

    He also talked why Tether survived:

    The colorful part of the cryptocurrency community will say—and we will not in public agree that there was a hole, but in private, yeah, of course there was a hole, I mean, come on—"If there was a hole, that hole got made up over the years by some combination of coincidentally receiving tens of billions of dollars from FTX under circumstances that we will not elaborate about, and moreover by the interest rate environment moving massively in Tether's favor."

    But they're still shady:

    [...] Tether has filed multiple legal pleadings with their judge attempting to get them to quash the request to produce balance sheets on the basis that we can't produce balance sheets because we had no books.

    Tether was not a tiny financial institution at this time. They were, you know, nominally having tens of billions of dollars of assets under management, and they had no books to do it with.

    ...

    Tether has not gotten rid of the non-crypto assets in their reserves. They've continued to increase the amount. And they've eliminated the disclosure that they used to make that their quote-unquote "secured loans" were not to related parties.

    So the inference, given that Tether is always doing the worst thing that is possible in coherence with their regulatory statements, is that they are making loans to related parties. And of course, they're probably continuing their history of lies, frauds, and evasions on top of that.

    Going back to talking about stablecoins in general:

    Another thing—and I will credit the CEO of Ethena, which has a stablecoin-adjacent financial product, for saying this—is a thing that you can tease out from stablecoin peer-to-peer payments volume: it's heavily correlated with the price of Bitcoin and froth in the crypto market generally.

    That is not something we expect to see in payments. The amount of coffee that you consume is not correlated with the price of Bitcoin. We generally expect the real economy to only have a fairly tenuous relationship with, say, the financial markets and almost no relationship with cryptocurrency, except to the extent that cryptocurrency is shadowing the financial markets.

    So that is another data point that would suggest—and again, highlighted by someone who would bet the other way—that stablecoins are not materially used in peer-to-peer payments at the moment to the degree that has been asserted.

    AI and the Future of Trading

    [W]hen I say I'm skeptical, I think that the impact of AI will be greater than the impact of any technology up to and including the Internet, where I consider the Internet to be the most important artifact that humanity has ever created and our collective great work as a species.

    That's what makes me a skeptic. Because I'm not saying that it will fundamentally transform human life forever within the next three years.

    ...

    The most experienced cool kids in the world have about 12 months of time that they've played with these tools. And increasingly, even in only six weeks of playing with it myself, it has become physically difficult for me to navigate a codebase without having Claude Code help me to digest the code base that I wrote.

    When financial systems are on fire, every minute matters. Knight Capital collapsed over a 20-minute critical window. CrowdStrike was citing about a 90-minute window for their time to resolution.

    If during a crisis it is difficult to find engineers who can remediate the crisis because Claude Code is in a slightly degraded state on any given day, that introduces a source of risk into the financial system which didn't exist two years ago, because we didn't have computers that were so good at assisting engineers two years ago that engineers would forget how to do core parts of the job.

    I don't want to call engineers incompetent or malicious for forgetting how to do this. This is simply like a skill that one develops over the course of a long career. It's something like riding a bicycle—you never forget it per se, but it really matters if you are doing the Tour de France that you have continued riding a bicycle every day for the last 365 days versus "Bicycles? Yeah, I remember having seen one about five years ago. But why would I ever use a bicycle when I have a car available?"

    You should be aware of the fact that we are now acutely dependent on cars and acutely dependent on LLMs, even for things where LLMs are not officially making the decision that matters. And the recursive dependency graph puts larger chunks of the economy on narrower shoulders.

    Again, if there are three labs where one or three of those labs going down will idle engineers at a huge number of firms, then we have far less numerous points of failure than we did previously.

    8 votes