58 votes

Linux bans the University of Minnesota for sending intentionally buggy patches in the name of research

28 comments

  1. [2]
    whbboyd
    (edited )
    Link
    edit: I'm seeing a bunch of confusion around the Internet about the timeline of events involved here. Here's the order things happened in: Prior to February, 2021: Many people associated with the...
    • Exemplary

    edit: I'm seeing a bunch of confusion around the Internet about the timeline of events involved here. Here's the order things happened in:

    • Prior to February, 2021: Many people associated with the University of Minnesota send patches to the Linux kernel. There's no reason a priori to believe the patches to be made in bad faith or related to the problematic patches which followed. These comprise all or the vast bulk of the variously-counted 190, 68, 250, etc. large numbers of patches mentioned. I recommend referring to these as "unrelated UMN patches".
    • Prior to February, 2021: Qiushi Wu, a PhD student, and Kangjie Lu, his adviser, both at UMN, send patches to the Linux kernel which intentionally contain security flaws. They claim none of these patches entered the kernel tree.
    • February, 2021: Wu and Lu publish a preprint of their research in which they sent intentionally flawed patches. This is what I'm referring to by "the research" below. I recommend referring to Wu and Lu's patches as the Wu & Lu 2021 patches, citing the paper.
    • Prior to and on April 21, 2021 (today): Aditya Pakki, a PhD student also under Prof. Lu, sends patches which contain security vulnerabilities to the Linux kernel. He claims the vulnerabilities in these patches are accidental. I recommend referring to these as "Aditya Pakki's patches".
    • April 21, 2021: Greg Kroah-Hartman, the kernel second-in-command, concludes on the basis of the vulnerabilities in Pakki's vulnerable patches and the known bad-faith patches from other members of Lu's research group that Pakki's patches are a continuation of that research, and reverts all patches from UMN email addresses until they can be re-reviewed (this includes the unrelated UMN patches, the Wu & Lu 2021 patches, and Pakki's patches).

    Assuming I haven't missed anything and we take everyone at their word, Pakki is just an innocent bystander here: he did not intend to send vulnerable patches to the kernel mailing list, and the wildly disproportionate reaction was due to prior bad faith actions he was immediately adjacent to but not actually directly involved in.


    My employer literally just made me go through human subject research training, so I might as well put it to some use.

    The short of it is: These researchers are clearly performing human subject research, which is not in one of the categories of research which may be exempted from IRB approval. Therefore they required IRB approval which they did not receive. If the researchers are to be believed that their IRB exempted them from approval, it did so improperly, either through negligence or because the researchers misrepresented their research plan. (Personal aside: I would bet on the latter.) Furthermore, even if the IRB had reviewed it, the research is in clear violation of the Common Rule, and thus should not have been approved.

    Step-by-step:

    Human subject

    A human subject is a living individual about whom the researchers obtain data through intervention, interaction, or gathering of private and identifiable information by any means. In this case, the researchers clearly interacted with the kernel maintainers by submitting patches; therefore, the kernel maintainers are human subjects. (Plausibly in this case, the patch submissions count as intervention, as well, as they were a direct attempt to affect the actions of the maintainers.)

    Research

    This barely needs touching on, because the researchers here are publishing academic papers and, indeed, consulting their own IRB, but not all investigations count as "research" for human subject research ethics purposes. "Research" is "a systematic investigation including research development, testing, and evaluation, designed to develop or contribute to generalizable knowledge." The research in this case was systematic and included the necessary phases, and is generalizable as they are extrapolating the conclusions to all open source software.

    Possible exemptions

    Not all human subject research requires IRB approval. There are eight exemptions which could exempt the research from needing approval. I'm not going to go through them one-by-one, because there are eight of them, but this research does not fall into any of the eight. (Most relate to research performed in specific settings or for which consent has already been obtained, which obviously do not apply here.)


    So, this research is clearly non-exempt human subject research, and therefore requires IRB approval, which it did not receive. (Again, my guess as to why is that the researchers misrepresented their research to the IRB, who were not technical enough to notice. This could plausibly have been accidental, though obviously this doesn't pardon anyone.) However, if the IRB had reviewed this research, they should have rejected it.

    The Common Rule

    Individual IRBs write their own policies regarding research approval, but all must follow the Common Rule published by the Department of Health and Human Services. This research is blatantly in violation of criteria 4 regarding consent, since the researchers did not obtain any consent, and it could not have been waived due to failing conditions of minimal harm (sneaking vulnerabilities past a kernel maintainer is certainly not "minimal") and subsequent debrief.


    So, there we have it. A bona fide ethics violation. Fabulous. Instead of concluding that these researchers are unethical cranks, because to be fair they should have been prevented from engaging in this research by their IRB, I'm going to conclude the following:

    • As clearly demonstrated, kernel maintainers cannot reliably detect bad-faith patches. I'm not sure what should be done about this. "Not writing in C" would certainly mitigate the potential impacts.
    • The University of Minnesota's IRB clearly needs to do some soul searching. Something like this happening on their watch—where, if we take everyone at their word, they were consulted—is a serious dereliction of duty.
    • STEM students absolutely must learn more humanities. The ethical red flags here are a mile wide; the fact that the researchers in good faith put together this research plan at all is a damning indictment of their prior exposure (or more accurately, evident complete lack thereof) to research ethics.
    55 votes
    1. streblo
      Link Parent
      FWIW: They do clarify that they talked with the IRB after starting the research where it was determined that they were not conducting human research.

      FWIW: They do clarify that they talked with the IRB after starting the research where it was determined that they were not conducting human research.

      11 votes
  2. [9]
    petrichor
    Link
    Hacker News discussion: Thread #1 and Thread #2 Attempting to sneak bugs into open-source projects for academic clout is a particularly shitty thing to do, but unfortunately nothing new. I'm glad...

    Hacker News discussion: Thread #1 and Thread #2

    Attempting to sneak bugs into open-source projects for academic clout is a particularly shitty thing to do, but unfortunately nothing new. I'm glad the kernel maintainers are taking this seriously - making a fuss and dragging the names of those responsible through the mud seems like the best way to dissuade this type of behavior in the future.

    21 votes
    1. [8]
      streblo
      Link Parent
      Do we know that's the case? I certainly understand the response from GKH nor do I think this is a particularly ethical way of going about this but it seems an effective demonstration of how easy...

      Attempting to sneak bugs into open-source projects for academic clout

      Do we know that's the case?

      I certainly understand the response from GKH nor do I think this is a particularly ethical way of going about this but it seems an effective demonstration of how easy it is to subvert infrastructure that is propping up the modern world.

      Of course, it's wasting everyone's time without providing much additional information (that PRs are a vector for bugs/malicious code is somewhat obvious) but I think there is some value in demonstrating how easy it is to do.

      7 votes
      1. emnii
        Link Parent
        There are millions of open source projects they could've targeted to prove their point. It's not unfair to assume they chose the Linux kernel to get attention.

        There are millions of open source projects they could've targeted to prove their point. It's not unfair to assume they chose the Linux kernel to get attention.

        4 votes
      2. [6]
        petrichor
        Link Parent
        I'd consider writing a paper on sneaking bugs into open-source projects trying to gain academic clout, yeah.

        I'd consider writing a paper on sneaking bugs into open-source projects trying to gain academic clout, yeah.

        2 votes
        1. [5]
          vektor
          Link Parent
          Do you apply that same logic to all research papers? I mean, I agree their methodology was shit and unethical, but with some more transparency this could've actually been good. Pre-announce your...

          Do you apply that same logic to all research papers? I mean, I agree their methodology was shit and unethical, but with some more transparency this could've actually been good.

          Pre-announce your intent to someone high up at the kernel, then pseudonymously submit patches. Immediately retract them if approved. Clearly distinguish (in retrospect) fakes from genuine work by others. Maybe some other mechanisms I can't think of now.

          3 votes
          1. [4]
            petrichor
            Link Parent
            I was being brief. They didn't do any of that, and that's what makes it bad. My response was just addressing that that's what they did in this scenario, even though there are other, ethical...

            I was being brief. They didn't do any of that, and that's what makes it bad. My response was just addressing that that's what they did in this scenario, even though there are other, ethical options.

            Although candidly, in my opinion that'd be a fairly useless experiment anyway. There's very little to get out of it that can't be gained from observation.

            8 votes
            1. [2]
              Pistos
              Link Parent
              This is a good point. Why couldn't they have just retroactively studied previously-submitted legit patches. They'd even have a much larger pool of data to work with, which might improve...

              There's very little to get out of it that can't be gained from observation.

              This is a good point. Why couldn't they have just retroactively studied previously-submitted legit patches. They'd even have a much larger pool of data to work with, which might improve statistical significance.

              6 votes
              1. ChuckS
                Link Parent
                Well if they wanted to know what percentage of patches with bugs were accepted or rejected, then they'd have to go back through all the submitted patches and find which ones have security flaws....

                Why couldn't they have just retroactively studied previously-submitted legit patches.

                Well if they wanted to know what percentage of patches with bugs were accepted or rejected, then they'd have to go back through all the submitted patches and find which ones have security flaws.

                "That sounds like work" is a literal quote I got from undergrads working on my (unrelated) project. It might have been easier for the researchers to make their own patches with known flaws and see how many of those were accepted than to find flaws that may not exist in someone else's code.

                6 votes
            2. streblo
              Link Parent
              Have to agree there. If their goal is some improvement to the patch approval process via static analysis there is little reason I can think of to not just use historical data.

              Although candidly, in my opinion that'd be a fairly useless experiment anyway. There's very little to get out of it that can't be gained from observation.

              Have to agree there. If their goal is some improvement to the patch approval process via static analysis there is little reason I can think of to not just use historical data.

              4 votes
  3. Macil
    (edited )
    Link
    It's a bad look for UMN, but I think this highlights a real problem as brought up in https://twitter.com/eevee/status/1384991581493628929. I think this is an angle people don't talk about enough...

    It's a bad look for UMN, but I think this highlights a real problem as brought up in https://twitter.com/eevee/status/1384991581493628929. I think this is an angle people don't talk about enough when considering the benefits of using safer languages like Rust: not only does it make it harder to accidentally make exploitable mistakes yourself, but it makes it much easier to accept and judge outside contributions (from people who might be more prone to mistakes by accident or on purpose).

    When you review large contributions in memory-safe languages, you can at least be quickly sure the patch isn't opening you up for exploitation by checking to see if certain code-loading APIs are used. You don't have to follow the whole control path in the code or all the context; you can usually just inspect the spots if any where unsafe APIs are called. In memory-unsafe languages (C), practically any line of code has the potential to open you up to exploitation depending on the full context. A for-loop going the wrong number of steps could possibly make your program vulnerable to running arbitrary attacker code from the internet. I've discovered bugs like this, I've written exploits for educational purposes, and I've submitted patches to projects. Memory-unsafe codebases are minefields.

    I'm really excited that Linux is starting to accept Rust in some parts of the kernel, and I hope their Rust usage increases.

    17 votes
  4. post_below
    Link
    New apology post from the involved researchers.

    New apology post from the involved researchers.

    11 votes
  5. [2]
    streblo
    Link
    Statement from the University.
    9 votes
    1. Pistos
      Link Parent
      In case it changes as events unfold, I copy-paste here what it is at the time of this writing.

      In case it changes as events unfold, I copy-paste here what it is at the time of this writing.

      Leadership in the University of Minnesota Department of Computer Science & Engineering learned today about the details of research being conducted by one of its faculty members and graduate students into the security of the Linux Kernel. The research method used raised serious concerns in the Linux Kernel community and, as of today, this has resulted in the University being banned from contributing to the Linux Kernel.

      We take this situation extremely seriously. We have immediately suspended this line of research. We will investigate the research method and the process by which this research method was approved, determine appropriate remedial action, and safeguard against future issues, if needed. We will report our findings back to the community as soon as practical.

      Sincerely,

      Mats Heimdahl, Department Head
      Loren Terveen, Associate Department Head

      6 votes
  6. [9]
    streblo
    Link
    The researchers' response.
    8 votes
    1. [8]
      post_below
      Link Parent
      The posts and articles I've skimmed about this left me with a very different impression of what happened than the researchers response gives. If they're telling the truth, and it seems unlikely...

      The posts and articles I've skimmed about this left me with a very different impression of what happened than the researchers response gives.

      If they're telling the truth, and it seems unlikely they would lie given the investigation and the ease with which it will be able to verify or refute their claims, I'm not sure I see any serious ethical issue with what they did. Three tiny (they mentioned five lines) patches distributed via email for feedback that they immediately came clean about if they were accepted to ensure there was no chance they could make it into the codebase.

      There's no malice there, or danger, or even a significant waste of anyone's time. Seems like everyone is overreacting. Am I missing something?

      2 votes
      1. [4]
        whbboyd
        Link Parent
        Human subjects of research in the US have extraordinary protections. Broadly speaking, any research with more than minimal chance of harm requires the subjects to consent to the research...

        Human subjects of research in the US have extraordinary protections. Broadly speaking, any research with more than minimal chance of harm requires the subjects to consent to the research beforehand. The researchers here were probably not malicious, but submitting bad-faith patches to the kernel maintainers with the intention of getting them past review is certainly not "minimal harm"—there's embarrassment to the maintainers who okayed the patches, the potential for those patches to enter the kernel tree (despite the researcher's best intentions), and of course the nigh-complete breach of trust between the researchers as kernel contributors and the kernel maintainers which lead to further possibly-unrelated flawed patches being grounds to ban the entire university from kernel development and revert their commits.

        9 votes
        1. [3]
          post_below
          (edited )
          Link Parent
          Ok but putting aside ideological considerations... We're not talking about patches that had any chance of getting into the kernal (at least according to the researchers), if they were okayed they...

          Ok but putting aside ideological considerations... We're not talking about patches that had any chance of getting into the kernal (at least according to the researchers), if they were okayed they apparently immediately said some version of "no nevermind there's a bug in it, look here...".

          How is that going to get into the kernal tree?

          Three patches, each only a handful of lines, in an email thread... From a non-emotional perspective, what is the actual harm?

          And even from an emotional perspective, I write code and I'd be tickled rather than upset, if this happened with a codebase I was involved in. Again, it was never going to get into the actual code! Or so it appears.

          For clarification, there's also the more recent researcher, who may or may not have been doing it intentionally. I'm not asking about that part of it.

          Edit: clarification

          2 votes
          1. [2]
            whbboyd
            Link Parent
            From the perspective of the Common Rule, all three of those things certainly count as "harm". Even if you discount all that and you're only concerned with the possibility for vulnerabilities to...

            what is the actual harm?

            there's embarrassment to the maintainers who okayed the patches, the potential for those patches to enter the kernel tree (despite the researcher's best intentions), and of course the nigh-complete breach of trust between the researchers as kernel contributors and the kernel maintainers which lead to further possibly-unrelated flawed patches being grounds to ban the entire university from kernel development and revert their commits.

            From the perspective of the Common Rule, all three of those things certainly count as "harm".

            Even if you discount all that and you're only concerned with the possibility for vulnerabilities to enter the kernel, Wu and Lu's stated plan for keeping their patches from landing is certainly not infallible. Email is not fully reliable; and they're relying on the maintainer giving a recognizable "ACK" and leaving sufficient delay between ACK-ing and applying the patch for them to intervene.

            10 votes
            1. post_below
              Link Parent
              I just read some more about the whole situation. The original research, which the above statement refers to, still looks pretty innocuous to me. But the later patches submitted by another student...

              I just read some more about the whole situation. The original research, which the above statement refers to, still looks pretty innocuous to me.

              But the later patches submitted by another student under the professor (or part of the group, it wasn't entirely clear) definitely are not. His goal seemed to be to get patches into the actual tree, and it seems someone, somewhere along the way, did succeed in doing that.

              Which effectively taints the entire project and justifies all of the backlash. It also illustrates the ethical problem. It's one thing for a professor and one graduate student to do a very limited experiment (approved by their IRB), another to have an unknown number of students basically submitting whatever they want. Better to avoid undisclosed experiments of any kind.

              So, I get the outrage now.

              5 votes
      2. knocklessmonster
        (edited )
        Link Parent
        My opinion has changed a few times on this issue, but I think it's solidfying finally. My (non-dev) understanding is that there's an issue of trust that FOSS runs on. You're expected to submit...

        My opinion has changed a few times on this issue, but I think it's solidfying finally.

        My (non-dev) understanding is that there's an issue of trust that FOSS runs on. You're expected to submit effective code in good faith. Bugs/exploits are forgiveable if they happen by accident, but for somebody to catch even a hint of doing so intentionally constitutes a massive breach of trust (see OpenBSD losing DARPA funding because some government agency allegedly paid for a backdoor. No solid evidence has surfaced, even after a security audit, but it arguably changed the path of OpenBSD). As a user, at least, I expect my distro maintainers to maintain good code, and they expect upstream to maintain good code. Upstream expects contributors to submit and/or maintain good code. Any lapse in this trust could potentially cause an entire part of the system to collapse, so trust is extremely important.

        There's no malice there, or danger, or even a significant waste of anyone's time.

        Time is a relatively infinite resource, but you can't get it back. gregkh, as the -stable maintainer had to review these patches for submission. He's going to be understandably pissed off to find out he has been manipulated for a research paper, and even frustrated at the possibility of inserting exploits into the project he's a maintainer on, which constitutes the massive breach of trust I referred to.

        8 votes
      3. vektor
        Link Parent
        Maybe a bit more proactive transparency would've been cool, i.e. making sure that people know that there was never a chance of any of this going unnoticed. E.g. disclosing to someone even higher...

        Maybe a bit more proactive transparency would've been cool, i.e. making sure that people know that there was never a chance of any of this going unnoticed. E.g. disclosing to someone even higher up in the linux kernel or some other institution that is trusted within the FOSS community. Basically, I would want a prior disclosure of "we want to test these guys, watch this. Don't tell anyone until we're done, but make sure we're not actually causing any harm.".

        I agree, this reads so reasonable (but also quite PR-ish) that I believe someone here is (inadvertently) full of shit. Either the community is blowing this out of proportion, or the response document is PR BS that massively understates their actions.

        2 votes
      4. Octofox
        Link Parent
        People are upset because the researchers have both wasted the time of the maintainers but more importantly, embarrassed them and the whole linux community by making them participate in this study...

        People are upset because the researchers have both wasted the time of the maintainers but more importantly, embarrassed them and the whole linux community by making them participate in this study without being aware.

        I don't think what they did is really that bad but I certainly understand where the negative feelings come from.

        1 vote
  7. [2]
    emnii
    Link
    UMN CS department heads have posted their response to The Linux Foundation's requests (warning: PDF) re: their ban on contributions. UMN has also created a page linking to all of their public...

    UMN CS department heads have posted their response to The Linux Foundation's requests (warning: PDF) re: their ban on contributions.

    UMN has also created a page linking to all of their public responses.

    6 votes
    1. whbboyd
      Link Parent
      I find bullet 4 of their response to be mind-boggling. I'm assuming the CS department is essentially passing on a statement from their IRB, but their claims are straight-up directly at odds with...

      I find bullet 4 of their response to be mind-boggling. I'm assuming the CS department is essentially passing on a statement from their IRB, but their claims are straight-up directly at odds with the regulations they themselves cite. To break it down:

      • The definition of a human subject in the regulations is concise. I broke it down somewhat in my earlier comment. I'm not sure how they came to the conclusion that "the maintainer did/did not detect the vulnerability in this patch" is not "information about" the maintainer. My guess is that the IRB—or whoever wrote that statement—still does not understand the implications of the research in question.
      • The study is certainly not exempt. I didn't do this breakdown before, but I'll waste the bytes on it now, just for you, UMN! There are eight exemptions under which human subject research does not require IRB approval, and they are:
        1. Research in educational settings regarding educational practices. Obviously inapplicable.
        2. Requires the use of "educational tests", surveys, or interviews, which obviously does not apply.
        3. Requires prospective agreement of the subject, and therefore does not apply in this case.
        4. Secondary research on preexisting data. Obviously inapplicable to this novel experiment.
        5. Specific exemption for certain research by federal agencies on public benefit programs. Does not apply here.
        6. Taste testing. While kernel development probably would be made more desirable if researchers handed out free food, that's not what happened here.
        7. Requires broad consent from the subjects, making it inapplicable.
        8. Also requires broad consent from the subjects.

      I am certainly curious about which inapplicable exemption UMN's IRB believes this research falls under.

      8 votes
  8. [2]
    anothersimulacrum
    Link
    Linux Kernel report on this whole affair: https://lkml.org/lkml/2021/5/5/1244

    Linux Kernel report on this whole affair: https://lkml.org/lkml/2021/5/5/1244

    4 votes
    1. cfabbro
      Link Parent
      The comment on Patch 3 is particularly noteworthy, IMO.

      The comment on Patch 3 is particularly noteworthy, IMO.

      This patch was quickly recognized by a reviewer to be incorrect, and the reviewer offered up possible changes that the submitter could make in order to turn it into a correct change. These suggestions were ignored by the submitter and no further changes were submitted in this area.

      The maintainer was attempting to mentor an obviously junior contributor, taking time to teach the developer what the proper thing to do here would be, and what is needed in order to have them create a contribution that would be acceptable. The contributor knew that the patch was bad, showing that the researchers were willing to waste the resource that is in shortest supply in our community: the time of reviewers and maintainers. Having this waste of an "effort of someone trying to teach another" be created by an educational institution was especially hurtful to the community and caused many of the bad feelings on the community's side, further amplified by not having any idea which patches out of the hundreds sent by UMN or from new contributors using gmail accounts might be intentionally bad.

      3 votes