psi's recent activity

  1. Comment on Overview of the proofs behind the Navier-Stokes $1 million millennium prize problem in ~science

    psi
    Link
    A couple dozen Fields Medal recipients have since issued a statement spurred by this development. "A Severe Misalignment of AI in Mathematics"

    A couple dozen Fields Medal recipients have since issued a statement spurred by this development.

    Over the last few months, the mathematical capabilities of LLMs have improved dramatically, to the point that they can solve major outstanding problems in many fields of mathematics. However, the push by AI companies to solve mathematical problems as a benchmark is detrimental to the science of mathematics, and to the mathematical community. The goals of the AI companies and the goals of the mathematical community are severely misaligned. We see these as part of broader alignment issues impacting other scientific and creative professions, as well as the whole of society.

    4 votes

  2. Comment on Overview of the proofs behind the Navier-Stokes $1 million millennium prize problem in ~science

    psi
    (edited )
    Link Parent
    But most people wouldn't know "the right inputs". That's almost entirely the controversy. Buckman and Alpöge allege (in short) that OpenAI stole their research program and scaled it up 10000 times.

    But most people wouldn't know "the right inputs". That's almost entirely the controversy. Buckman and Alpöge allege (in short) that OpenAI stole their research program and scaled it up 10000 times.

    6 votes

  3. Comment on Overview of the proofs behind the Navier-Stokes $1 million millennium prize problem in ~science

    psi
    Link Parent
    There are no other computed-based tools comparable to an LLM. It's a totally different paradigm. Probably the previous largest technological breakthrough for mathematics was the internet, but the...

    There are no other computed-based tools comparable to an LLM. It's a totally different paradigm. Probably the previous largest technological breakthrough for mathematics was the internet, but the internet encouraged collaboration, not hoarding. I also wouldn't agree that simply having a solution is necessarily good for mathematics as a whole. As Terence Tao puts it:

    [...] The incentives may now be pointing in the direction of no longer sharing any promising research directions with the broader community, which would reverse centuries of traditions of open science and do serious long-term damage to the future of the field.

    In short, the indiscriminate use of powerful solution-extraction tools can achieve the immediate short-term goal of solving problems at hand, but at the cost of sustaining the ecosystem for the next wave of progress, or in understanding the progress already obtained. [...]

    There is much more to doing mathematics than just having the answer. Indeed, in mathematics the journey is usually at least as important as the destination. However, LLM-steered mathematics tends towards inelegant and clunky intermediate results, meaning that one loses much of the value that would normally be gained from having a solution to a difficult problem.

    14 votes

  4. Comment on Overview of the proofs behind the Navier-Stokes $1 million millennium prize problem in ~science

    psi
    Link Parent
    These are good points, but I think you risk underselling the importance of the universal approximation theorem. For a small neural network (e.g. reproducing a 1-dimensional function), you don't...

    These are good points, but I think you risk underselling the importance of the universal approximation theorem. For a small neural network (e.g. reproducing a 1-dimensional function), you don't even need backpropagation; you can use whatever minimizer you'd like.

    1 vote

  5. Comment on Overview of the proofs behind the Navier-Stokes $1 million millennium prize problem in ~science

    psi
    Link Parent
    The larger issue is that OpenAI used millions of dollars of compute to scoop the researchers who were honing in on a solution. Admittedly those researchers were potentially still years away, but...

    The larger issue is that OpenAI used millions of dollars of compute to scoop the researchers who were honing in on a solution. Admittedly those researchers were potentially still years away, but OpenAI's behavior must be massively demotivating for mathematicians. Why spend your career working on a difficult problem if a lab will just throw 10,000 agents at that problem and solve it over the weekend? The entire episode shows that these labs are extremely capable but also not responsible stewards of their own technology.

    Also, regarding the citations: even in OpenAI's most recent draft of the proof, Levent Alpöge (Buckman's collaborator) still hasn't been cited.

    16 votes

  6. Comment on US judge to baby: file an asylum application in ~society

    psi
    (edited )
    Link Parent
    Consider this today's daily reminder that immigration judges are executive branch employees, not Article III judges. (Or as I've heard one immigration lawyer put it: immigration court is just...

    Consider this today's daily reminder that immigration judges are executive branch employees, not Article III judges. (Or as I've heard one immigration lawyer put it: immigration court is just traffic court but with life and death consequences.)

    To be more generous than @gryfft's read (albeit perhaps less generous than yours, @skybrian), the Board of Immigration Appeals comprises almost exclusively Trump appointees (the Biden appointees having been purged), so terrible rulings like these are permitted and championed, and when immigration judges try to resist, they are often fired. (For another example of unjust policy, see the BIA's ruling on mandatory detention.) More likely, the judge either supports this administration's immigration objectives or feels obligated to rule as if they do, lest they lose their job.

    11 votes

  7. Comment on What's happening in Kansas should alarm us all (gifted link) in ~lgbt

    psi
    Link Parent
    The article mentions trans folk who tried to have their markers changed to the new legal requirement but were denied because they didn't bring the notice letter with them. (Indeed, many never even...

    The article mentions trans folk who tried to have their markers changed to the new legal requirement but were denied because they didn't bring the notice letter with them. (Indeed, many never even received the notice.)

    But then there's also this:

    On May 5, Kris Ripper, a trans woman, was stopped while driving home from work for failing to turn on her headlights in the rain. The police officer repeatedly asked her why her license indicated that she was male; she explained that the law required it. Almost two months later she found out that she had been charged with a misdemeanor for driving without a valid license. (Ripper talked to another trans reporter about her predicament, but declined to speak with me.) The charge was eventually dropped, but not before she was threatened with a $1,000 fine and jail time.

    6 votes

  8. Comment on What happened: OpenAI and Hugging Face in ~comp

    psi
    Link Parent
    "15 attorneys general have instructed OpenAI to preserve all materials related to the Hugging Face hack." Business Insider. No consequences yet, but hacking into other people's computers is a...

    "OpenAI's unprecedented and alarming misconduct demands an immediate and significant response," the group wrote, adding that the company may have violated state and federal law, including consumer protection and data privacy statutes.

    They instructed OpenAI to immediately preserve all materials related to the Hugging Face hacking incident, as well as any prior instances in which its agents similarly engaged in unauthorized intrusions into computer systems or databases.

    No consequences yet, but hacking into other people's computers is a crime. It will at least be interesting to see how this plays out (even if, like you, I'm not optimistic much of anything will come from it).

    3 votes

  9. Comment on Share your favourite dal recipe! in ~food

    psi
    Link Parent
    My recipe is similar to yours, so I'll just post mine here (as well as the video I stole it from). There is a fair amount of flexibility in how one prepares the tadka: like you, usually I leave...

    My recipe is similar to yours, so I'll just post mine here (as well as the video I stole it from). There is a fair amount of flexibility in how one prepares the tadka: like you, usually I leave out the curry leaves and add smoked paprika instead, though I use butter instead of ghee (as I always have butter lying around). Also a warning: it's rather easy to burn the tadka if you aren't being careful.

    Ingredients

    For the dal

    Amount Item
    1 cup toor (split pea) lentils
    2 medium potatoes
    5 cups water
    2 tsp tumeric
    1/2 tsp fenugreek leaves
    1 clove grated garlic
    1 tsp grated ginger
    to taste salt

    For the tadka

    Amount Item
    a handful curry leaves
    half medium onion
    1/2 tsp mustard seeds
    1/2 tsp cumin seeds
    3 cloves garlic
    2-3 small peppers
    small piece cinnamon
    blob butter
    some oil

    Instructions

    For dal

    1. Wash lentils and set aside.
    2. Heat oil in (lentil) pot at medium heat.
    3. Chop garlic and ginger, setting aside a smaller portions for dal and a larger portion for tadka
    4. If using, dice potatoes.
    5. Add smaller portions of garlic & ginger to oil.
    6. After a minute, add dal to toast. If using, add potatoes, too.
    7. Add tumeric.
    8. Add water and salt. Cover lid.
    9. Let cook for ~30 minutes.
    10. Add tadka (see below).
    11. Add fenugreek leaves.

    For tadka

    1. Remove curry leaves from stem.
    2. Cut peppers at an angle
    3. In a separate pan, set to medium heat and add blob of butter + oil (to prevent burning)
    4. Add mustard seeds, cumin seeds, and cinnamon. Cover lid.
    5. After the popping begins to subside, add curry leaves and onion. Add more butter if needed.
    6. After a couple minutes, add garlic, ginger, peppers, and a bit of oil (to prevent burning).
    7. Just before the garlic starts to brown, pour tadka into dal. Add a little bit of dal to tadka pan then pour back into dal bowl to ensure that none of the buttery tadka goodness is lost.

    References

    1 vote

  10. Comment on A skeptical overview of AI and consciousness in ~humanities

    psi
    Link Parent
    Chapter 11 has a section on "strange intelligence" that is very much worth a read, if you haven't read it yet. In particular, we can imagine that in order for an entity to be conscious, it should...

    Chapter 11 has a section on "strange intelligence" that is very much worth a read, if you haven't read it yet. In particular, we can imagine that in order for an entity to be conscious, it should have a set of essential properties, but that we could in principle build machines that contain only a subset of those properties. They wouldn't technically be conscious, but they would be almost conscious -- what the author calls a form of strange intelligence.

    Unfortunately, rather than resolving the ethical implications, it just raises a host of new ones. For instance, is it moral to intentionally lobotomize a machine that would otherwise be conscious? Certainly nobody would think it moral to intentionally lobotomize an otherwise viable embryo. Similarly, perhaps consciousness is not a necessary condition for feeling, and despite not being conscious such machines would still feel suffering when being forced to do their user's bidding. Probably comparisons to anesthesia are appropriate here. I once read a particularly harrowing account of a colonoscopy in which a patient screamed bloody murder the whole time, but as soon as the operation was done (and the anesthesia stopped), the patient calmly thanked the doctor and said it was the easiest colonoscopy of his life; I don't think people would be okay with the digital equivalent of that.

    1 vote

  11. Comment on A skeptical overview of AI and consciousness in ~humanities

    psi
    Link Parent
    I assume the author discusses LLMs because it's concrete, existing technology, and it truly does make the philosophical implications more pressing compared to, say, 15 years ago when something...

    I assume the author discusses LLMs because it's concrete, existing technology, and it truly does make the philosophical implications more pressing compared to, say, 15 years ago when something like ChatGPT would have sounded like a lazy plot device. However, most of the author's arguments concern AI in the abstract, not LLMs in particular.

    It's a little risky to assume what the author thinks, given that the central tenet of his book is that we should practice epistemic modesty with respect to AI consciousness. Nevertheless, if we read between the lines, it seems pretty likely to me that the author does not believe LLMs are conscious, but rather that future AI models where LLMs are a component potentially could be.

  12. Comment on A skeptical overview of AI and consciousness in ~humanities

    psi
    (edited )
    Link Parent
    I think you might have misunderstood the purpose of the draft. The book intends to point out that we are sprinting straight into an epistemological nightmare, and its "fog" may well become the...

    I think you might have misunderstood the purpose of the draft. The book intends to point out that we are sprinting straight into an epistemological nightmare, and its "fog" may well become the source of massive social upheaval. Further, if consciousness is private (one of its possible essential features), then it may well be impossible to know whether the next generation of AI models will be conscious, regardless of whether they are or aren't. Meanwhile, frontier labs race each other to create AGI.

    I mean, consider the social ramifications (chapter 11). Suppose in the year 2100 we create some machine that could plausibly be called conscious, enough so that it's the social issue of the era (comparable to abortion or LGBT rights in the US currently). If such machines were conscious, then forcing them to do our bidding would amount to a form of slavery. But if they were not conscious, then we risk expending a huge amount of future political capital -- say, granting them land and rights -- to something that's just a fancy calculator, mimicking conscious behavior but totally lacking an inner world. Some people will choose to marry AI models; others will attempt to outlaw the practice. Financial interests will corrupt the search for truth and justice absolutely.

    It's a huge unfolding mess.

    3 votes

  13. Comment on A skeptical overview of AI and consciousness in ~humanities

    psi
    (edited )
    Link Parent
    My understanding of the author's use of "skeptical" is not that we should be skeptical that AI is conscious, but rather we should be skeptical of anyone who claims to know whether AI is/can become...

    My understanding of the author's use of "skeptical" is not that we should be skeptical that AI is conscious, but rather we should be skeptical of anyone who claims to know whether AI is/can become conscious or not. That is, we should be skeptical of arguments both for and against AI consciousness. If you haven't yet, you should really read the first couple chapters in which the author elaborates on how "consciousness" and "artificial intelligence" are load-bearing terms that are basically impossible to nail down. (As if to prove the point, some philosophers argue that rocks are conscious.) When it comes to consciousness, there are no obvious answers, just as there is no obvious divide between "artificial intelligence" and "real intelligence", so to speak.

    Suppose we take the argument at face value. There is no LLM declaring this, so the argument is entirely moot.

    Well, as I said, the author is not so much concerned about current generation LLMs as he is future iterations. Nevertheless, it's worth examining some of the points you made. You are declaring that consciousness must contain some "essential" properties (see chapters 3 and 4), specifically that it must have "access" and "specious presence" (see text or my footnote [1]). However, neither of these properties are necessarily essential. For instance, it could be that we have conscious experiences that are not available for further processing (experiences that don't backpropagate, so to speak), but because we wouldn't process these experiences, we wouldn't remember them, either.

    Moreover, generalizing any essential property risks suffering from a certain type of "sampling bias", as the author calls it -- we risk making a classification error in which we equivocate "consciousness" (whatever that is) with the human experience of having consciousness. For example, it could be that some creatures experience the world in a way outside time (like Kurt Vonnegut's Tralfamadorians or Ted Chiang's Heptapods), which would render "specious presence" unnecessary.

    In general, I would really just recommend you read Schwitzgebel's book. He's certainly thought more about this subject than either of us.

    [1] Taken from chapter 3:

    (4) Access. To be conscious, an experience must be available for “downstream” cognitive processes like inference and planning, verbal report, and memory. No conscious experience can simply occur in a cognitive dead end, with no possible further cognitive consequences.

    (9) Specious presence. All conscious experiences are felt to be temporally extended, smeared across a small interval of time (a fraction of a second to a few seconds) – generally called the “specious present” – rather than being strictly instantaneous or wholly atemporal.

    6 votes

  14. Comment on A skeptical overview of AI and consciousness in ~humanities

    psi
    (edited )
    Link Parent
    I think your question amounts to asking whether AI models can "feel" or merely mimic feeling , which is addressed in Chapter 7 ("The Mimicry Argument Against AI Consciousness"). First, it's worth...

    I think your question amounts to asking whether AI models can "feel" or merely mimic feeling , which is addressed in Chapter 7 ("The Mimicry Argument Against AI Consciousness"). First, it's worth establishing what the author means by mimicry; these excerpts are probably sufficient for you to get the drift.

    When you know that something has been designed or has evolved as a mimic, you cannot infer from the readily observed feature to the further feature in the way you ordinarily would in the model. At least you can’t do so without further evidence. Once you know that the viceroy [butterfly] mimics the monarch, you cannot infer from its wing pattern to its toxicity. Maybe the viceroy is toxic, but that would need to be separately established. Similarly, knowing that [a toy doll's] “hello” mimics a human greeting, you cannot infer that the toy actually intends to greet you.

    [...]

    A consciousness mimic is an entity that mimics some superficial or readily observable features that, in some set of model entities, reliably indicate consciousness. But because the mimic has been designed or selected specifically to display those superficial features, we the receivers cannot justifiably infer underlying consciousness – not in the same way we can when we see those same features in the model entity. This is obvious for the “hello” toy, less obvious but still true for entities specifically designed to pass the Turing test or otherwise mimic the surface features of human language. An important class of AI systems are consciousness mimics in this sense.

    Searle and Bender aim for a stronger conclusion, inviting us positively to conclude that the mimics do not have conscious linguistic understanding. I don’t think we can know this from their arguments. But both thought experiments successfully describe consciousness mimics whose outputs we should reasonably mistrust. The case for consciousness is undercut. It does not follow that the case against consciousness is established.

    [...]

    Ordinarily, if you’re having what seems to be a meaningful conversation, you can infer that your conversation partner is conscious and understands the meaning of your words. But if you know that the entity is designed to mimic human text outputs, you ought no longer be so sure.

    The author mentions later in the chapter that "the large majority of experts on consciousness agree that classic pure transformers are not conscious", so if a current generation LLM were to declare, "Now I am just as conscious, just as full of experience and feeling, as any human", we are probably correct to dismiss the statement as mimicry. However, post-training complicates this assessment: tomorrow's frontier models will not be evaluated exclusively on their ability to produce plausible-sounding text. Indeed, "feeling" will likely be an explicit (or at least auxiliary) training goal (emphasis added):

    Recent Large Language Models such as ChatGPT and Claude build upon the mimicry structures of pure transformer models but also receive post-training. Reinforcement learning from human feedback “rewards” human-approved outputs, strengthening the associated weights. Models can also be reinforced for being “right” by external standards, and some can access tools like calculators. To the extent the machines move beyond pure mimicry, the Mimicry Argument applies less straightforwardly. For now, mimicry-based skepticism still seems warranted, since their core architecture remains close to that of pure transformers, and their humanlike outputs are still best explained by their pretraining on word co-occurrence in human texts.

    In the longer term, we might imagine architectures more thoroughly trained on the rights and wrongs of the world itself – maybe like AlphaGo but with the larger world, or some significant portion of it, as its playground. Outputs would be shaped primarily by success in real-world complex tasks, perhaps including communicative tasks, rather than by resemblance to humans. The Mimicry Argument would then no longer apply. Skepticism about their consciousness, if warranted, would need a different basis.

    In general, this book is quite comprehensive and comprehensible. There is also an entire chapter on whether biological substrate matters (chapter 10), which will likely address your question from another angle.

    4 votes

  15. Comment on European Parliament achieves upgrade to air passenger rights in ~transport

    psi
    Link Parent
    Yeah, given Ryanair's history of dark patterns, this is what I suspect as well. Ryanair's basically subsidizing the cost of tickets for some customers by banking on other customers making honest...

    Honestly the "higher margin" sounds like them charging people 60€ for not having the app installed.

    Yeah, given Ryanair's history of dark patterns, this is what I suspect as well. Ryanair's basically subsidizing the cost of tickets for some customers by banking on other customers making honest mistakes, which is a pretty shitty way to treat the people you're supposed to be serving.

    6 votes

  16. Comment on Arch User Repository compromised, 1500+ packages affected in ~tech

    psi
    Link Parent
    On the other hand, the compromised PKGBUILDs do two things: add npm as a dependency and use it to install atomic-lockfile in a post-install script. So unless someone is the type of user who...

    On the other hand, the compromised PKGBUILDs do two things: add npm as a dependency and use it to install atomic-lockfile in a post-install script. So unless someone is the type of user who actually scrutinizes every line of code from a git repo (or is rightfully distrustful of npm dependencies), that person would not notice the malware injection.

    It's like blaming Tesla drivers for crashes when the autopilot randomly fails and the driver didn't respond in the 3 second window between autopilot's disengagement and the accident. The larger issue is that the AUR is fundamentally insecure but works well in >99% of cases, luring users into a false sense of security; it's a design problem.

    6 votes

  17. Comment on US FBI says Google engineer used internal search data to win $1.2M on Polymarket in ~tech

    psi
    Link Parent
    Well, yes, but that's sort of like saying the point of drug dealing is to sell drugs. Barring the proper legal frameworks, such activities are not lawful.

    Well, yes, but that's sort of like saying the point of drug dealing is to sell drugs. Barring the proper legal frameworks, such activities are not lawful.

    6 votes

  18. Comment on If you let AI do your writing, I will come to your house and kill you in ~tech

    psi
    Link Parent
    In a recent interview, Ezra Klein referred to such constructions as "attachment hacking". (I'm not sure where he got the term from; possibly here.)

    In a recent interview, Ezra Klein referred to such constructions as "attachment hacking". (I'm not sure where he got the term from; possibly here.)

    Klein: I find it interesting. “Attachment hacking” is the idea that one thing happening in A.I. which is different than, as you note, social media algorithms, is that the A.I.s have been designed — they didn’t come up with this on their own — to hack the way we attach to other people.

    So when I’m talking to Claude, it’s constantly saying to me: Well, if you want my honest opinion — or: The best piece I read on this is — or: That’s a great point.

    There’s no reason it has to pretend to have a first-person pronoun with me.

    Claude is not an “I” in that way. Nor is ChatGPT or Gemini or Grok, or any of them, but they speak to you as if they are, and that’s a design choice to attach you to them.

    [...]

    I can feel it working before I shut that down, or I try to shut that down. Who knows if I’m actually being successful.

    But it’s amazing to read these moments in which this algorithm is posing as another entity offering me an emotionally connected response. Giving me praise I might want or offering me candor that I might admire.

    And I know it’s [expletive], and yet my brain is tuned to recognize that as connection.

    7 votes

  19. Comment on US Pentagon puts building blocks in place for Cuba invasion in ~society

    psi
    Link Parent
    I think you're focusing on the wrong bit. It's worth emphasizing that the Trump admin specifically called Cuba a failed state, not an economically weak one. Objectively this isn't true (a failed...

    I think you're focusing on the wrong bit. It's worth emphasizing that the Trump admin specifically called Cuba a failed state, not an economically weak one. Objectively this isn't true (a failed state would have no functioning government, e.g. Libya or Haiti), so that should already be sufficient to dismiss the explanation as nonsense.

    However, we might be charitable and assume they mean that Cuba is on its way to becoming a failed state. But the rolling blackouts that have recently impacted Cuba were directly caused by the US's embargo, so if the Trump administration were truly concerned about preventing Cuba from becoming a failed state, they could just (partially) lift the embargo. But obviously they won't do that, from which we can conclude that the justification is not merely nonfactual but also pretextual.

    20 votes

  20. Comment on Weekly US politics news and updates thread - week of May 18 in ~society

    psi
    Link Parent
    Of course, the fact that it specifies Democratic elected officials already suggests the settlement is not politically neutral. And depending on how you parse that sentence, you could distribute...

    Of course, the fact that it specifies Democratic elected officials already suggests the settlement is not politically neutral. And depending on how you parse that sentence, you could distribute the adjective Democrat[ic] across the whole list of items, i.e. Democrat[ic] elected officials, Democrat[ic political and career federal employees, etc, which would necessarily mean that people would presumptively preclude victims from suing Republican officials and agents. Obviously such an arrangement cannot be legal.

    Regarding (4B), another complaint is that, despite Congress having power of the purse, only one of the five seats is assigned to Congress, and (as you said) Trump can unilaterally remove anyone at any time, which makes that one token seat functionally useless.

    4 votes