17 votes

During his testimony before the Senate Judiciary Committee, Peiter "Mudge" Zatko claims Twitter only has live production environment that all engineers can access

@Benjamin Powers:
Mudge walking through Twitter's construction - they only have live production environment, no test environment.

4 comments

  1. [3]
    Luna
    (edited )
    Link
    What the fuck. That is insane. It's one thing when companies without a strong engineering culture take the "we test in prod" approach, but Twitter has kernel devs on staff because they've run into...

    What the fuck. That is insane. It's one thing when companies without a strong engineering culture take the "we test in prod" approach, but Twitter has kernel devs on staff because they've run into so many weird bugs at their scale (source: read it on HN at some point), I never would have thought they'd be one of those companies.

    Edit: Arbybear's comment has the article about Twitter's kernel devs, which was linked on HN.

    Some other Tweets from the thread that I found interesting:

    https://twitter.com/benjaminopowers/status/1569690510486343680

    Grassley noting that Twitter is outsourcing moderation to other countries, and the moderators lack tools and translators in some cases.

    This (sadly) doesn't surprise me at all, Twitter is far from the only company who does this.

    https://twitter.com/benjaminopowers/status/1569691440308961280

    Mudge leads off by saying Twitter is a decade behind competitors when it comes to security standards. Says company is misleading everyone from lawmakers to board of directors.

    https://twitter.com/benjaminopowers/status/1569691968849969158

    "This kind of vulnerability is not in the abstract. It's not far fetched to say that employees inside the company could take over the accounts of all of the senators in this room."

    This tracks with what I've read about a bunch of web3 hacks:

    According to zachxbt, he has been selling access to a Twitter admin panel, which allows employee-level access to Twitter tools. This might explain how many of the accounts were compromised despite being protected by multi-factor authentication.

    I have better data security practices for my side projects. This is jaw-droppingly bad. The potential for bad actors is huge - you could dox a lot of people just by pulling the geolocation data they've logged, which is apparently wide open to every employee.

    https://twitter.com/benjaminopowers/status/1569698169478422529

    "It [Twitter] is unable (to delete data) because they do not know where it is so they are unable to comply," said Mudge.

    I think it's also safe to assume they cannot fulfill discovery obligations if they don't know where things are stored. I wonder if their legal team has ever raised this issue before.

    Edit: Apparently, the "employees accessing user info for malicious purposes" idea I floated has already happened, but not for doxxing - it was for helping the Saudi government identify political dissidents.

    Sources:

    “Several Twitter users, who either followed Mr. Al-Ahmed’s Twitter account and/or had direct contact with him through the use of Twitter’s private messaging feature, have disappeared, been arrested, or have been executed,” the complaint said.

    16 votes
    1. Omnicrola
      Link Parent
      JFC this is jaw-droppingly negligent/incompetent. It's one thing for a strappy new startup to have been caught off guard by a surge in traffic and have to scramble to improve their security....

      JFC this is jaw-droppingly negligent/incompetent. It's one thing for a strappy new startup to have been caught off guard by a surge in traffic and have to scramble to improve their security. Twitter has no excuses. At a minimum what should come out of this is a set of regulations requiring any company dealing with more than X customer records to have XYZ security procedures in place.

      12 votes
    2. Arbybear
      Link Parent
      The source for Twitter kernel devs is this Dan Luu blog post: https://danluu.com/in-house/

      The source for Twitter kernel devs is this Dan Luu blog post: https://danluu.com/in-house/

      2 votes