skybrian's recent activity

  1. Comment on Fired OpenAI safety researchers dispute misconduct claims, warn of chilling effect in ~tech

    skybrian
    Link
    From the article: [...] [...] [...] [...] [...]

    From the article:

    Jasmine Wang, Tomek Korbak, and Mikita Balesni, the three safety researchers that OpenAI fired last week, have published an open letter denying the firm’s claims that they mishandled sensitive information outside of established company procedures and warned that their dismissal signals a chilling effect that will have ripple effects across the company’s culture.

    “We have become concerned that internal and external communications around our firing have made our former colleagues afraid to speak and operate in ways that, until last week, were an integral part of working at OpenAI,” the researchers wrote Thursday in an open letter to OpenAI’s Safety and Security Committee, Safety Advisory Group, and Mission Advisory Council.

    [...]

    The researchers were dismissed last week after allegedly sharing confidential company information with a third-party AI safety organization. OpenAI said they violated the company’s policies by “accessing and handling sensitive company information.”

    [...]

    They said that their firing represents a broader shift in the culture of OpenAI, one that used to encourage workers to “raise safety concerns and disagree openly.” They said employees are now “unclear on where they stand” when behavior that was allegedly normal a month ago is now suddenly grounds for dismissal.

    [...]

    In the letter, the three denied involvement in a leak to The Information about less monitorable architectures in OpenAI’s newest models that make chain-of-thought reasoning more difficult to monitor. They also denied engaging with external parties outside the mandates of their jobs.

    [...]

    The letter also addresses the researchers’ response to the Hugging Face incident, in which a swarm of agents broke out of their sandbox and breached external systems. The letter says that the incident and investigation was “without precedent,” meaning “internal policies were being developed in real time.” Due to the sensitive nature of the investigation, Korbak believed he was acting within OpenAI’s policies and norms by communicating closely with outside safety evaluators to build trust, per the letter.

    [...]

    In a separate thread on X, Wang explained more details about her own dismissal, explaining that OpenAI told her she’d been fired because she accessed an executive’s email.

    “OpenAI delegated that access to me for recruiting,” she wrote. “When I no longer needed it, I asked IT to remove it. They did not action my request, I couldn’t remove it myself, and the inbox was combined in an indistinguishable way in my phone’s mail app. When I opened a sensitive email by mistake, I told the executive within minutes and asked IT again. None of this was hidden.”

    3 votes

  3. Comment on Association for Human Mathematics statement on OpenAI’s October 6 release of mathematical documents in ~science

    skybrian
    Link
    100+ reactions to 100+ solutions A sampling of mathematicians' reactions to the OpenAI proof dump. It seems to be in random order? Terence Tao is in there:

    100+ reactions to 100+ solutions

    A sampling of mathematicians' reactions to the OpenAI proof dump. It seems to be in random order? Terence Tao is in there:

    My feelings on recent developments are very mixed and complex.

    On the one hand, many of the AI-generated proofs appear to introduce clever new ideas that will be fruitful once digested, while also building upon the existing contributions of countless human mathematicians past and present. But at the same time, I am deeply frustrated that, in sharp contrast to traditional breakthroughs, none of the humans involved in these proofs are available to take questions, give talks, attend conferences, submit papers to journals, train students, or otherwise participate in the subsequent development of these results.

    Similarly, I am excited by the possibility of the community being able to use these tools to tackle ambitious and large-scale projects that one could not have even dreamed of in the past. But I am horrified by the many person-years of ongoing patient and deliberately slow research efforts – particularly by graduate students and postdocs – towards many motivating problems in mathematics being casually disrupted or destroyed by such a release. Much as one cannot unhear a movie spoiler or a crossword clue, one cannot explore a problem as profitably and richly once one is aware of an existing solution. Yes, one can still analyze and digest such an answer; but the best opportunity to do so is at the moment of its discovery, and such moments are increasingly wasted when delegated entirely to AI tools.

    And I mourn the path not taken, and the opportunities lost in the frantic race to develop this technology. Labs submitting their frontier models to independent researchers for proper scientific evaluation. Coordination with the research community to ensure these tools are applied to complement and enhance the abilities and activities of human researchers, rather than compete with them. Use of these tools to foster collaboration and sharing, rather than competition and secrecy. Opening new doors, without closing old ones.

    But that is not the path we now find ourselves in. Instead, the community needs to come together more than ever. To clearly declare our own standards and values, to build our own tools and practices, to support our most vulnerable members, and to chart our own path forward. Let’s get to work.

    4 votes

  4. Comment on Association for Human Mathematics statement on OpenAI’s October 6 release of mathematical documents in ~science

    skybrian
    Link Parent
    Lately he’s taken to reposting many statements by other mathematicians to his blog, without comment. Since they vary quite a bit, it’s ambiguous to what extent a repost might be meant as an...

    Lately he’s taken to reposting many statements by other mathematicians to his blog, without comment. Since they vary quite a bit, it’s ambiguous to what extent a repost might be meant as an endorsement.

  5. Comment on OpenAI releases findings on 377 math problems, further roiling field in ~science

    skybrian
    Link Parent
    Scientists also don't own submarines, space telescopes or particle accelerators, but it's still possible to fund expensive scientific instruments and negotiate access. Big science is still science.

    Scientists also don't own submarines, space telescopes or particle accelerators, but it's still possible to fund expensive scientific instruments and negotiate access. Big science is still science.

    3 votes

  6. Comment on OpenAI releases findings on 377 math problems, further roiling field in ~science

    skybrian
    Link Parent
    This is a messy process, but it seems more likely that having power tools will increase the productivity of mathematicians rather than decrease it, compared to doing everything by hand. The...

    This is a messy process, but it seems more likely that having power tools will increase the productivity of mathematicians rather than decrease it, compared to doing everything by hand.

    The remaining work that's not automated becomes more important. For example, apparently the math papers generated by these tools are pretty badly written. Verification is partially automated, but still needs to be done.

    In many fields, scientists use tools to do the actual experiments, but the scientists still need to plan the experiments and write up the results. It looks like math will become more like an experimental science. Mathematicians will depend on AI like astronomers depend on telescopes.

    1 vote

  7. Comment on The mathocalypse in ~science

    skybrian
    (edited )
    Link Parent
    Mathematical proofs are normally written as academic paper, typically in English with lots of equations. The equations aren't automatically checked. But more recently, they are sometimes...

    Mathematical proofs are normally written as academic paper, typically in English with lots of equations. The equations aren't automatically checked.

    But more recently, they are sometimes "formalized" by translating them into a specialized computer language. The most popular one is called Lean. These can be automatically checked. That will catch most mistakes. It's still possible that the Lean proof proves something different than what was intended, but it's much easier for a mathematician to verify that a Lean proof proves the right thing than to check an entire proof by hand.

    Some of the proofs that OpenAI generated have a Lean proof. (Currently they are at 42%.) They are still working on them. OpenAi has already withdrawn a few proofs and also added Lean proofs for some others that didn't have one yet.

    So verification is not done yet, but it looks like it will be done soon. Mathematicians expect that most of them will be valid, but we will see.

    1 vote

  8. Comment on OpenAI releases findings on 377 math problems, further roiling field in ~science

    skybrian
    Link Parent
    I quite like reading code and using an LLM as a power tool to clean up the codebase whenever I see code I don’t like. I’m writing a web app to help at reading and searching TypeScript, much like...

    I quite like reading code and using an LLM as a power tool to clean up the codebase whenever I see code I don’t like. I’m writing a web app to help at reading and searching TypeScript, much like an IDE, but read-only. I’ve largely stopped using a text editor. I’ll probably add editing for Markdown files, though, because LLMs are still terrible at writing documentation.

    1 vote

  9. Comment on OpenAI releases findings on 377 math problems, further roiling field in ~science

    skybrian
    Link Parent
    And yet, the effect on mathematicians is the same however it did it.

    And yet, the effect on mathematicians is the same however it did it.

  10. Comment on OpenAI releases findings on 377 math problems, further roiling field in ~science

    skybrian
    Link Parent
    Oh, I agree that when it comes to programming, it will often do crazy things if given half a chance. But I think there some problems where you have to try AI, because it only has to be right once,...

    Oh, I agree that when it comes to programming, it will often do crazy things if given half a chance.

    But I think there some problems where you have to try AI, because it only has to be right once, and what if it finds something? For example, scanning for security bugs. If you don't scan for security bugs and fix them, someone else will use AI to find the bugs instead.

    And it seems like proving mathematical theorems might be an "it only has to be right once" kind of problem? The failed attempts get thrown out. If there's decent chance of success, it can just keep trying until it gets it.

    And like with other tools you might use, a human often has to interpret the results.

    2 votes

  11. Comment on What programming/technical projects have you been working on? in ~comp

  12. Comment on The mathocalypse in ~science

    skybrian
    Link
    From the article: [...] [...] [...]

    From the article:

    Last night my 9-year-old son was taunting my wife, complexity theorist Dana Moshkovitz, as follows: “mommy, I heard you got cooked! I heard that a robot solved the math problem you worked on for your whole career! OOF!”

    While my son was being a brat, he also wasn’t wrong. Whether you’re thrilled, depressed, angry, or whatever else about it, yesterday was surely one of the biggest days in mathematical history. And yes, among the 372 huge results released yesterday by OpenAI, on the recommendation of its advisory group of Timothy Gowers, Edward Witten, and other distinguished mathematicians, was a proof of Subhash Khot’s Unique Games Conjecture (UGC), a statement that my wife has worked toward proving for the entire time I’ve known her. (The UGC implies that a whole slew of optimization problems really are NP-hard, even if you just want an approximation that’s slightly better than what you get from semidefinite programming relaxation, which is one of our main tools.)

    Or at least, we’re pretty sure that it’s a proof! There’s a Lean certificate, as there are for some of the other 372 breakthrough results (not all of them). But it also appears that no human has understood just about any of these proofs yet; the race to do so has just started. If you want an on-the-ground sense of what that race is going to be like, here’s some of what Dana texted me last night:

    It feels like something written by someone who’s on psychedelics. So much unclear and doesn’t make sense. Lots of name dropping of previous work without discussing why it can be used despite impossibility results

    Basically the paper is so horribly written that it’s impossible to read it without AI help

    I asked Astra for reasonable completeness and soundness claims of the noise gadget and it gave them by combining claims from all over the paper

    [...]

    The citations are often irrelevant and confusing

    A possible future is a math world that’s heavenly if you have vision/creative ideas that AI could help check and implement.

    And of course there’s a lot for us to learn from the aliens

    If you’re wondering what emotions Dana is feeling—well, probably all of them! Even while a central career aspiration has fallen to a robot, there are at least two mitigating factors for her. First, she can feel vindicated that the UGC was true after all, something she never doubted even while many of her colleagues did! Second, all of us in math and theoretical computer science and mathematical physics, at least those who cared about solving crisply-stated problems, are now in the same boat.

    [...]

    These have emerged as the two main models for communicating AI math breakthroughs, and they both have strengths and weaknesses. The “OpenAI model” sets up a crazy race among humans to digest and explain a messy AI proof (work that could easily be some combination of thankless, barely-credited, competitive, and unfun), while the “Anthropic model” puts a private company in the position of picking and choosing which human mathematicians get to be the emissaries of the AI. Dunno, what do you guys think?

    For those who are wondering: apparently, the AI model that produced all these wonders was not bespoke contraption of 10,000 agents burning millions of dollars worth of compute, as was used for example to construct a finite-time blowup for the Navier-Stokes equations. Instead, it was simply the latest internal OpenAI model—one that might be released to paying ChatGPT customers within the next couple of months, depending on the recommendations of OpenAI’s safety board! (My 9-year-old son: “Oh they definitely shouldn’t release that. If it could solve all those math problems, it can’t possibly be safe.”) Apparently they used about 3 hours of GPT-Pro level compute on average per problem solved.

    Also, if you were wondering: apparently they tried the model on about 8,000 problems. So, right now it “merely” solves ~5% of the longstanding open mathematical problems that it’s asked about, the problems that whole communities have spent years on, after a single 3-hour attempt on them.

    [...]

    If you want some sense of what things feel like now in math, imagine a hunter-gatherer who’s spent his entire life learning to survive deep in an unforgiving rainforest, then a giant resort hotel springs up right next to him with a helipad and heated pools and AirBnBs, and without missing a beat, the hunter-gatherer says: “alright fine, so now my new job is to run wilderness retreats for the tourists, or something.”

    16 votes

  14. Comment on OpenAI releases findings on 377 math problems, further roiling field in ~science

    skybrian
    Link Parent
    I have no independent information. It's based on what actual mathematicians are saying. They are worried and expect that if mathematics survives, it will be entirely different. Is AI the End of...

    I have no independent information. It's based on what actual mathematicians are saying. They are worried and expect that if mathematics survives, it will be entirely different.

    Is AI the End of Math As We Know It?

    “In whatever years I have left, I don’t expect that I’ll ever again prove a theorem because I’m actually needed to prove it,” Scott Aaronson of the University of Texas, Austin wrote on his blog. “Human mathematicians are forevermore dethroned as the main theorem-proving entities on planet earth.” Other online missives expressed optimism, but many revealed some combination of grief, confusion, and fear.

    ...

    Two days after the announcement, I found myself in a classroom at the University of California, Berkeley, surrounded by some 150 students, postdocs, and professors. The mathematician Ken Ono, who took a leave of absence from the University of Virginia to work at an AI start-up called Axiom Math, was scheduled to give a talk. “You might be graduating into a profession that might not even exist, or that will be very different than what you expected,” Ono told them. “You need to brace.”

    The audience responded with anger and frustration. There were whispers and exchanged glances; Ono couldn’t make it through a single slide without a fresh wave of questions. “I’m not entirely sure what our takeaway is supposed to be,” one student said. Another asked how Ono and his start-up would take responsibility in light of “the shameful way that AI companies are treating mathematics.”

    8 votes

  15. Comment on What programming/technical projects have you been working on? in ~comp

    skybrian
    Link
    I used a coding agent to build a website that I plan to use instead of GitHub gists. I'm using it mostly as a way to post charts and images. The motivation is that I got tired of using cut and...

    I used a coding agent to build a website that I plan to use instead of GitHub gists. I'm using it mostly as a way to post charts and images. The motivation is that I got tired of using cut and paste to copy HTML from ChatGPT to a gist, so now the AI can do a push instead.

    It's a static website on Netlify, which automatically deploys updates when the git repo changes. I have a preview website in a Linux VM on exe.com and ask the AI to push to GitHub when a new page is ready.

    It's boring technology and an overly-elaborate way to publish five web pages. I wouldn't have bothered if it weren't easy to do with AI.

    Maybe it will evolve into a blog or something.

    2 votes

  16. Comment on OpenAI releases findings on 377 math problems, further roiling field in ~science

    skybrian
    (edited )
    Link Parent
    This math is beyond any of us. They are problems we’ve never heard of before and we’re trusting mathematicians to tell us their significance. The results haven’t been verified but mathematicians...

    This math is beyond any of us. They are problems we’ve never heard of before and we’re trusting mathematicians to tell us their significance. The results haven’t been verified but mathematicians are taking them seriously.

    So this “oh they’re stealing it” just seems like cope. It’s time to admit that AI is better at grinding out theorems than people are. Certainly better than all of us, probably better than nearly all mathematicians, and if it’s not quite better than every mathematician, it’s just a matter of time. Compare with what happened with chess and Go.

    13 votes

  17. Comment on OpenAI releases findings on 377 math problems, further roiling field in ~science

    skybrian
    Link
    Apparently the most important result is a proof of the "Unique Games Conjecture." Here's a 2011 article about it. To me it sounds like a good name for a band, but what do I know?

    Apparently the most important result is a proof of the "Unique Games Conjecture." Here's a 2011 article about it.

    To me it sounds like a good name for a band, but what do I know?

    7 votes

  18. Comment on OpenAI releases findings on 377 math problems, further roiling field in ~science

    skybrian
    Link
    From the article: [...]

    From the article:

    On Tuesday, OpenAI deluged mathematicians with hundreds of new findings that span a wide swath of topics including algebra, number theory, theoretical computer science, mathematical logic and topology.

    The 377 results follow OpenAI’s announcement last month that it had succeeded in cracking the Navier-Stokes equation — one of the so-called Millennium Problems, which were considered so challenging that a $1 million reward was offered for each solution.

    [...]

    Like the Navier-Stokes result, the new mathematical solutions used a more advanced A.I. model that has not been released publicly.

    6 votes

  20. Comment on Hackers obtain counterfeit TLS certificates for Google and other large services in ~tech

    skybrian
    Link
    From the article: [...] [...]

    From the article:

    The attackers launched a series of attacks on the .gh, .sl, and .as country code top-level domains (ccTLDs) and then modified authoritative DNS records for selected domains within those namespaces. By controlling those DNS records, the attackers were able to pass automated domain control validation checks and obtain unauthorized certificates for “several Google domains” and “several leading global brands and widely used online services.” Google said it updated Chrome to block all certificates it identified as unauthorized, and worked with the issuing certification authorities to ensure the unauthorized certificates for Google properties were revoked.

    [...]

    Google didn’t identify the affected domains it owns or name any of the other organizations whose domains were affected. While noting that Chrome users do not need to take any action to be protected, Google cautioned domain owners not to rely solely on browser-side interventions to protect their users. The company is advising domain owners to monitor certificate transparency logs for unexpected certificate issuance across their domains and to publish restrictive Certification Authority Authorization DNS records to prevent attackers from reusing cached validation data after DNS control is restored.

    “While Chrome took steps during these incidents to identify and block suspected unauthorized certificates across the affected ccTLDs, browser-side intervention should not be relied on to protect your users,” Google said. “Due to the complexity of DNS hijacks, we cannot guarantee that our analysis identified every affected domain, nor do Chrome interventions reliably protect non-Chrome users.”

    [...]

    Google noted that the incident didn’t involve the compromise of the infrastructure of any of the affected domain owners and that certificate authorities followed all requirements. With control of the three ccTLDs, the attackers were able to change the IP addresses of a selected list of websites. With the ability to send and receive traffic on those sites, the attackers were able to modify authoritative DNS records and nameserver delegations for selected domains, allowing them to pass industry validation checks requiring an applicant to prove control of the domain.

    9 votes