post_below's recent activity

  1. Comment on The machines are fine. I'm worried about us. in ~science

    post_below
    Link Parent
    I've seen the original post floating around but wasn't really interested. The idea that LLMs can contribute to skill athrophy, or not building skills in the first place, is low hanging fruit...

    I've seen the original post floating around but wasn't really interested. The idea that LLMs can contribute to skill athrophy, or not building skills in the first place, is low hanging fruit that's been discussed from every angle over the last couple years.

    But after I saw your comment I gave it a skim to see if it was LLM generated and, yes, it definitely is. The author gave it at least one human editing pass after generation which makes it a bit harder to detect. And they did a better job than many do of editing, but as always happens the human editing gets progressively lazier as you go.

    If you wonder if something is AI written, ignore the first two paragraphs. They'll often be the only human written part, or the most carefully post edited. Instead start halfway through.

    Anyway the tells in the post are so numerous and frequent that anyone who's used frontier agents in any volume will recognize the tropes.

    I imagine the "author" convinced themselves they were being profoundly ironic by using AI to write the post. But I suspect that really they're just lazy and cynical.

    In the context of the point their LLM is trying to make for them: The prompter robbing themselves of the opportunity to develop skills and deep mental models around writing. They're also missing out on the rewarding dynamic created by respecting your audience.

    3 votes

  2. Comment on My polish costs nothing in ~tech

    post_below
    Link Parent
    As it happens, if you tell them not to use em-dashes, they develop a newfound love of semicolons!

    (and semi-colon user, though I don't think I've ever seen that one in LLM writing?)

    As it happens, if you tell them not to use em-dashes, they develop a newfound love of semicolons!

    1 vote

  3. Comment on My polish costs nothing in ~tech

    post_below
    Link Parent
    It wouldn't be the first time, or even the 50th, that people with non-mainstream tech insight influenced culture. A pretty high percentage of life has been happening online for most of two...

    It wouldn't be the first time, or even the 50th, that people with non-mainstream tech insight influenced culture. A pretty high percentage of life has been happening online for most of two decades.

    Not to say that I necessarily like the odds. Slop is going to have an impact one way or another.

    1 vote

  4. Comment on My polish costs nothing in ~tech

    post_below
    Link Parent
    Likely there are different RF segments for these but the important thing is the prompt/context. "Translate this" results in inference that's going to try harder to stay close to the source...

    I don't know if LLMs go through different training pipelines between these tasks,

    Likely there are different RF segments for these but the important thing is the prompt/context. "Translate this" results in inference that's going to try harder to stay close to the source material. "Write about this" is asking for fabrication, which is a free pass for all the tropes introduced in fine tuning to come out.

  5. Comment on My polish costs nothing in ~tech

    post_below
    Link Parent
    Something to think about for that conversation: IMO the conversation we should be having isn't "human coding VS agentic coding". Instead we should be talking about how to incorporate agents in...

    Something to think about for that conversation: IMO the conversation we should be having isn't "human coding VS agentic coding". Instead we should be talking about how to incorporate agents in ways that make sense and don't suck for humans. There's no denying that, used well, agents can speed things up and soften pain points without lowering quality (too much) or driving people insane. The problem is that there's not yet conventional wisdom about what that should look like. We're still figuring out what coding agents are best used for, and what they aren't good at. And then re-figuring it out every model generation or two.

    I think the answer for the time being is to be extremely intentional when incorporating agents. And expect a lot of experience and feedback driven iteration of the process. Scaffolding makes a huge difference and I've seen a lot of attempts that miss this entirely. Formalization is another big one but I won't digress. Most importantly: until things have settled and best practices start to emerge, every org should take the time to carefully create processes that work for them. Preferrably driven largely by senior engineers rather than management.

    It sounds like maybe the process isn't particularly intentional at your company, so humans are left trying to adapt, rather than adapting agents to what works for humans. It doesn't have to be that way. Though of course I know nothing about your org, so I might have that wrong. And maybe you're at a scale where intentionally building processes is really hard. The principle is sound though.

    3 votes

  6. Comment on My polish costs nothing in ~tech

    post_below
    Link Parent
    It's got an old school dark children's story vibe to it

    It's got an old school dark children's story vibe to it

  7. Comment on My polish costs nothing in ~tech

    post_below
    Link Parent
    Cory Doctorow has a term for this: reverse centaur. He's alarmist sometimes but he coins great terms.

    There's something utterly disgusting in becoming a sort of biological appendage to systems that become more prevalent and more opaque with each passing day, and I have a hard time articulating exactly why.

    Cory Doctorow has a term for this: reverse centaur. He's alarmist sometimes but he coins great terms.

    21 votes

  8. My polish costs nothing

    ~tech Ask
    Hopefully the Tilderen aren't completely sick of the topic of AI writing. To summarize my take: I think LLM agents are profoundly useful. And problematic. And I really really dislike AI writing...

    Hopefully the Tilderen aren't completely sick of the topic of AI writing.

    To summarize my take: I think LLM agents are profoundly useful. And problematic. And I really really dislike AI writing when I encounter it outside of documentation. It's not only that it usually has a poor signal to noise ratio and a lot of annoying tropes, it also breaks a contract that has existed since writing was invented. And I dislike that it displaces human writing. Human writing is beautiful, we need more of it, not less.

    By breaking a contract I mean that human writing implies an investment. The author invested time in creating it and has (sometimes) invested their reputation by publishing it. Whereas LLM prose requires minimal effort and thought. One great summary is:

    "If you didn't take the time to write it, why should I take the time to read it?"

    But it's not just about time or preference. We rely on signals of investment and competence to judge the value of information both personally and professionally. LLMs short circuit that completely. They incorporate signals of effort despite there being no effort involved. Signals that we could once rely on that someone has a deep understanding of a subject, or did extensive research, or is reasonably intelligent, or interesting, don't have any meaning when it's LLM prose.

    Recently, while investigating Fable failure modes (while working in Claude Code), it outputted a summary that I think captures the situation really well: "My polish costs nothing".

    Which is to say that LLM prose folds in signals of authority, competence, rigor and insight without any of those things needing to be present in the content. A fluent human is often worth listening to while a fluent LLM is just the default, the fluency carries no information. I think that's part of the reason for AI psychosis, I think it effectively uses, now outdated, expectations to hijack perception.

    The reason that I sometimes take the time to investigate why an agent went wrong is partly to refine the scaffolding and partly to remind myself how dumb they are. Each generation is a little bit better, confidently right a little more than it's confidently wrong, so I make a point not to let myself fall into the trap of trusting their output too much.

    I suppose I'm just posting to pass on the reminder.

    38 votes

  9. Comment on A skeptical overview of AI and consciousness in ~humanities

    post_below
    Link Parent
    It seems to me that the word feeling was coined to describe a biological process. Probably we'd use a different term for something else. But I take your point, as long as we don't perfectly...

    That may be true for the biological implementation of feelings. How do you know it's equally true for all other implementations?

    It seems to me that the word feeling was coined to describe a biological process. Probably we'd use a different term for something else.

    But I take your point, as long as we don't perfectly understand what feeling is, we can't say what is or isn't feeling with confidence.

    2 votes

  10. Comment on A skeptical overview of AI and consciousness in ~humanities

    post_below
    Link Parent
    Might be useful to make the distinction between feelings and qualia. For qualia, sure, we have essentially no way to quantify. With feelings, when we experience them there are various predictable,...

    Might be useful to make the distinction between feelings and qualia. For qualia, sure, we have essentially no way to quantify. With feelings, when we experience them there are various predictable, measurable things that happen in our biology which coincide with the feeling. Indeed with the right chemicals you can even turn feelings off, or turn them up.

    In a lot of cases you could argue that the things we're measuring are only establishing correlation, but not in all cases. It seems to me that you'd need pretty extraordinary evidence to make a case that a significant part of feeling is not biological.

    That doesn't mean feeling couldn't be produced some other way of course, but it's part of a very complex, interdependent, system that co-evolved for a specific set of circumstances over millennia. An AI system achieving a particular threshold of intelligence is not just going to manifest that entire system spontaneously. Unless the claim is that feelings, and related areas of experience, are actually mediated by a soul, and the machine gains a soul at some inflection point.

    We'd instead need to specifically set it on a course where the goal was for it to evolve a system analogous to our own. Which isn't unimaginable, but we definitely haven't even started down that path.

    3 votes

  11. Comment on A skeptical overview of AI and consciousness in ~humanities

    post_below
    Link Parent
    The biggest difference is that we understand our cognition far less well than we understand transformer models. Absolutely, I just think it will look a lot different than LLMs and so probably the...

    How different is that

    The biggest difference is that we understand our cognition far less well than we understand transformer models.

    Again, I’m most definitely not saying we’re there with current technology, just that it’s a discussion probably worth preparing right now.
    Who knows what’ll happen in the not-so-distant future?

    Absolutely, I just think it will look a lot different than LLMs and so probably the conversation shouldn't be couched in LLM tech.

    4 votes

  12. Comment on A skeptical overview of AI and consciousness in ~humanities

    post_below
    Link Parent
    There's some truth to that, about pretty much everything, but stated that strongly it's just FUD. We don't know way more than we know, generally speaking, but the universe isn't that...
    • Exemplary

    "The experts do not know, and you do not know, and society collectively does not and will not know, and all is fog"

    There's some truth to that, about pretty much everything, but stated that strongly it's just FUD. We don't know way more than we know, generally speaking, but the universe isn't that incomprehensible. LLMs certainly aren't.

    The insistence that any large neural network running in vram is definitely NOT conscious, or x, y, or z, is also bullshit because no one knows how they work

    This isn't true. We absolutely know how they work.

    but the engineers that are working on this will tell you the actual mechanism within these matrices is a giant black box

    That's misleading. We don't have great visibility into how embeddings encode a particular response/behavior. But the mechanism, the math, is well understood. There are good reasons to want to call it a black box, but it's not strictly true. Remember that much of the discourse is influenced by companies with a vested interest in it feeling more magical than it is. Even a lot of individuals are excited about LLMs in an oddly irrational way. The utility and the potential are real, and paradigm changing, but they also get exaggerated a lot.

    We are running neural networks that program themselves with code that is more complicated than we can currently understand.

    That's misleading too. LLM outputs are used in training, yes, but "program themselves" paints a picture of recursive self improvement that stretches the meaning of the term way too far.

    There is no code anywhere in the process that we don't understand. The inference engines (the runtimes) are written by humans, at this point LLMs are no doubt helping but they're not producing anything people can't understand.

    The model weights themselves, the part that sometimes gets called a black box, are not code. They're parameters that represent the training data (pre and post). We run gradient descent on the parameters and something remarkable happens that was never coded by anyone, human or AI. But it's not right to say that it's code or even really that we don't understand it. The math isn't particularly complicated and the ouput is exactly what you'd expect.

    12 votes

  13. Comment on A skeptical overview of AI and consciousness in ~humanities

    post_below
    Link Parent
    I think creating conscious servants is wrong and we should never do it. I think it's a valuable conversation to have, if maybe premature. I like to think the bulk of humanity would agree that...

    I think creating conscious servants is wrong and we should never do it. I think it's a valuable conversation to have, if maybe premature. I like to think the bulk of humanity would agree that creating slaves is bad so we can probably get away with waiting until AGI is at least visible on the horizon if you squint before there's any urgency.

    Meanwhile my point: The author talked a whole lot about LLMs, but LLM tech doesn't offer a path to consciousness without making near spirtitual leaps beyond what we currently know. So propping a thesis, even in part, on LLMs undermines the whole thing.

    Maybe the implicit alarmism will contribute to convincing people to support legislation that puts guardrails on the big model labs. Which would be a good thing. But I hope we can do that without being disingenuous.

    4 votes

  14. Comment on A skeptical overview of AI and consciousness in ~humanities

    post_below
    Link Parent
    It's hard to read when the framing is based on the idea that there's a reasonable debate to be had about AI and consciousness. To even begin to have that conversation you need to first make a case...

    It's hard to read when the framing is based on the idea that there's a reasonable debate to be had about AI and consciousness. To even begin to have that conversation you need to first make a case for the possibility that any technology we currently have could be advanced to the point that it could result in consciousness. Which presumably would first require AGI. That's a big ask.

    And without that the whole premise falls apart and it's just speculation for its own sake.

    2 votes

  15. Comment on A skeptical overview of AI and consciousness in ~humanities

    post_below
    Link Parent
    This captures some of my sentiment about consciousness conversations around LLMs. You first have to misunderstand the technology (willfully or not) and then speculate heavily about what could...

    This captures some of my sentiment about consciousness conversations around LLMs. You first have to misunderstand the technology (willfully or not) and then speculate heavily about what could happen in the future once being forced to acknowledge that current tech almost definitely isn't headed there. It's science fiction, which I generally think is great, but it's often framed as rational or academic when it's neither.

    If a paper, or in this case a book, were to lead with "Here's how the technology works... Given how it works, even with our incomplete understanding of what consciousness is, it's safe to say that consciousness is not going to happen here. With that in mind, let's speculate about what technology that reasonably could lead in that direction might look like... That I could appreciate.

    But so far what keeps coming out is attempts to overfit LLM tech into a path to consciousness because a significant percentage of the population really wants conscious machines.

    3 votes

  16. Comment on A skeptical overview of AI and consciousness in ~humanities

    post_below
    Link Parent
    We don't know what feeling physically is in a comprehensive sense. But we have quantified enough parts of feeling to confidently say that it's complicated, it's both neurological and biochemical,...

    We don't know what feeling physically is in a comprehensive sense. But we have quantified enough parts of feeling to confidently say that it's complicated, it's both neurological and biochemical, it's not exclusively controlled or originated by the brain, and it evolved through natural selection to help organisms behave in ways that would cause them to have a better chance of survival.

    We don't attribute feelings to toasters and rocks because they lack both the equipment and the impetus to produce feelings, not because it would be inconvenient. It's a bit muddier when it comes to, for example, insects.

    4 votes

  17. Comment on A skeptical overview of AI and consciousness in ~humanities

    post_below
    Link Parent
    How would that work? Feeling is a complicated biochemical process rather than a side effect of intelligence. Intelligence and feeling do not somehow spontaneously come in the same package. In...

    “Now I am just as conscious, just as full of experience and feeling, as any human”

    How would that work? Feeling is a complicated biochemical process rather than a side effect of intelligence. Intelligence and feeling do not somehow spontaneously come in the same package.

    In order to replicate feeling it would need to be the explicit goal, it won't happen accidentally. The idea that we "won't know" is fun for science fiction, but with current technology the accurate sentence is "won't happen".

    Maybe someday, but only as a result of an intentional initiative. Even if we imagine recursive self improvement, feeling wouldn't be a logical target, we'd need to ask for it

    7 votes

  18. Comment on Apple sues OpenAI for stealing trade secrets to build AI hardware in ~tech

    post_below
    Link Parent
    I don't have insight into their financials, it's hard to see exactly what's happening with all the smoke from the burning piles of cash. But over the last year or so, for business use cases, and...

    I don't have insight into their financials, it's hard to see exactly what's happening with all the smoke from the burning piles of cash.

    But over the last year or so, for business use cases, and particularly coding, they've seemingly given up on leading and have instead been copying what Anthropic does in terms of harnesses and tooling. Their models themselves are usually a little behind Anthropic. In some situations a lot behind in my experience.

    It's not an enviable position to be in when it's a tight race despite outspending the competition. Their head start is long gone.

    I think they're well into profit on inference though. Which doesn't make up for their insane costs but it does make the business model look viableish. I think they've got some time yet before they need to worry about investors turning on them dramatically. Everyone is incentivized to hold on for the IPO.

    And then at the same time, Chinese open weight models are getting dangerously close to the frontier. I'm surprised it hasn't caused a bigger correction. It's potentially really bad for both Open AI and Anthropic's profitability future, and therefore the public companies they're propping up.

    I think I'd say they're in a precarious position rather than struggling. 5.6 Sol is a success from what I've seen so far, with a little help from the federal Fable shutdown. They're still undeniably in the top 2 with 3rd place quite a bit behind. As long as that doesn't change, and as long as the markets don't get spooked, they can probably keep the bonfire going for a while.

    6 votes

  19. Comment on Control the ideas, not the code in ~comp

  20. Comment on Control the ideas, not the code in ~comp

    post_below
    Link Parent
    Given how people react to AI generated prose, me included, I find myself feeling much more forgiving towards an ESL writer who didn't do an agent editing pass.

    Given how people react to AI generated prose, me included, I find myself feeling much more forgiving towards an ESL writer who didn't do an agent editing pass.

    1 vote