11 votes

Control the ideas, not the code

16 comments

  1. [12]
    bme
    Link
    Maybe antirez is some god-tier prompter, but I find myself constantly dissatisfied by the decisions that anthropic's frontier models make. I think they are 100% great at automated reviews...

    Maybe antirez is some god-tier prompter, but I find myself constantly dissatisfied by the decisions that anthropic's frontier models make.

    I think they are 100% great at automated reviews (relatively low cost to verify, I can overrule without negotiation, endless attention any time of the day, can be fed house review checklists). I think they are great sounding boards for design. Exploiting their encyclopedic knowledge of general techniques to direct research yields better designs faster than I could unassisted. I am just struggling with the idea that you can be so sure that your ideas are being faithfully reflected if you look never. That's not at all my experience. I am constantly finding inconsistencies and stupidity any time I take my eye off the underlying evolution of what's being spit out.

    12 votes
    1. [4]
      stu2b50
      Link Parent
      The key is design docs. Lots of design docs. Always have the agent make a design doc first. Only have it start generating code once the details in the docs are agreeable. As a bonus it’s really...

      The key is design docs. Lots of design docs. Always have the agent make a design doc first. Only have it start generating code once the details in the docs are agreeable. As a bonus it’s really nice having that kind of doc easily available.

      6 votes
      1. bme
        Link Parent
        Do you not find that docs produced by ai tend to miss the forest from the trees over indexing on minutiae to the point of rendering them nearly useless for humans (probably very good for other...

        Do you not find that docs produced by ai tend to miss the forest from the trees over indexing on minutiae to the point of rendering them nearly useless for humans (probably very good for other agents)?

        Also, I don't know about you, but personally I find I arrive at leaner designs by stress-testing them against existing code. Sure, if it's something that fits an obvious archetype or design pattern you can probably anticipate a fair amount.

        I find that AI basically never simplifies ever. Neither in design or code and without active anti entropy you end up in a bad spot, and no amount of design gets you lower entropy.

        2 votes
      2. pocketry
        Link Parent
        What kind of design docs are you advocating for? Technical, functional, architecture, or something else? Do you mean for each piece of functionality you build or high level system things? How big...

        What kind of design docs are you advocating for? Technical, functional, architecture, or something else? Do you mean for each piece of functionality you build or high level system things? How big are the docs you prefer?

        Im curious how it compares to our setup and my preferred method

        1 vote
      3. bme
        Link Parent
        Do you not find that docs produced by ai tend to miss the forest from the trees over indexing on minutiae to the point of rendering them nearly useless for humans (probably very good for other...

        Do you not find that docs produced by ai tend to miss the forest from the trees over indexing on minutiae to the point of rendering them nearly useless for humans (probably very good for other agents)?

        Also, I don't know about you, but personally I find I arrive at leaner designs by stress-testing them against existing code (less turns planning more turns refining). Sure, if it's something that fits an obvious archetype or design pattern you can probably anticipate a fair amount.

        I mean maybe I am just outing myself here: I am complaining about models not reading my mind haha.

        I find that AI basically never simplifies ever. Neither in design or code and without active anti entropy you end up in a bad spot, and no amount of design gets you lower entropy.

    2. [3]
      DrStone
      (edited )
      Link Parent
      I'm a software engineer with 15+ years in the industry. Over the years, I have occasionally tried AI through whatever the Intellij IDEA Ultimate IDE offered when I was really stuck on something...

      I'm a software engineer with 15+ years in the industry. Over the years, I have occasionally tried AI through whatever the Intellij IDEA Ultimate IDE offered when I was really stuck on something fairly self contained (e.g. sql query optimization). Every time I was disappointed and wondered how anyone got anything worthwhile out of AI.

      Earlier this year, my boss got us all a 1-month trial to Claude Code (Opus 4.6 I think), and told us genuinely no-pressure "give it a shot, see if it helps". I went in very skeptically, with a baseline zero trust for the AI output. The first few tasks were self-contained, dead simple, and easily verifiable - stuff like "translate this legacy Twirl template to React". I reviewed every line it produced and tested every result. It did pretty well, so I started using it a bit more.

      It's been a few months now and I find that I use Claude Code (now Opus 4.8, thinking high) for the majority of my work. I've created some tight agent files for the projects and a short global one based on actual usage and results. I've heard my process called "architecture driven development", though I don't really participate in the AI industry? culture? fandom? Detailed design docs / plans vastly improve the AI implementation results and subsequent review, like @stu2b50 mentioned. Anyway, here's what's gotten me good results:

      1. Personally draft as detailed of a requirement set as I can, with high level design/architecture. Reference local code precedent where possible.
      2. Give that to Claude and prompt it to draft a plan. Anything beyond a known simple fix, always draft a plan. It'll ask me various questions during the planning, and state any assumptions or decisions it made at the end.
      3. In a clean session (so it's a cold-read), prompt Claude to critically review the plan it just wrote. I created a skill for this after a while based on how I would usually prompt it, refined with Claude itself. This digs in to everything from verifying all claims against live source / data, reviewing high level architecture and cost, preemptively looking for contradictions and ambiguities and edge cases, etc. It'll come back to me with a list of findings, organized by estimated severity. Giving it a read-only user for the database and several other key integrations has helped a lot here, and sometimes even surfaced bad assumptions on my part.
      4. Manually walk through each point in the plan review findings with Claude, prompting a resolution or further investigation, until all are resolved.
      5. Depending on the nature of what was found and how much the resolutions altered the plan, repeat step #3 as many times as reasonable.
      6. Manually skim the final plan. At this point, I'm pretty familiar with it from the drafting and reviews, so it's mostly a sanity check.
      7. In a new session, have Claude implement the plan and check against whatever automated verification the specific thing might have (compile, linting, the rare tests, etc.)
        8 - 10. Same as step #3-5, except for the implementation. In a clean session audit the implementation. I've got another skill for this based on what I'd usually ask it to do. Faithfulness to plan, trace impacted code flows, etc. Walk through findings, rinse and repeat as necessary.
      8. A skim of the actual code diff. Working with a large, mature codebase gives Claude a lot of precedent to follow, so it usually organizes and writes things consistent with our existing code. Sometimes I don't like exactly how it organized things, or the specific implementation of something, and I'll either directly change it or prompt Claude to redo that bit.
      9. Manual end to end testing. Usually some adjustments fall out of this.

      Note that the various clean sessions are important, or having claude run the review skill with subagents in clean sessions. Aside from growing large contexts eventually degrading results, a cold reading of the plan and the implementation are key. They don't let old assumptions skip over investigating potential faults, dropped branches from investigation get brought back in, or anything like that.

      If nothing else, it's a good way to force yourself to articulate and record all of your requirements, all of the desired flows, and account for "all" edge cases. Like extreme rubber-ducking.

      EDIT:
      Oh, and one other thing that I don't think people often consider: Maybe once every few months, ask the AI to review all of it's memories and agent files for consistency, redundancy, verbosity, and organization (different project memories, memory vs agent file, project agent file vs global agent file). Over time, especially if you've got multiple related projects, it'll accumulate a lot. Cleaning up can make a difference in result quality and context token usage.

      5 votes
      1. [2]
        post_below
        Link Parent
        This is an excellent workflow! Mine isn't a lot different. In general I'd argue for more review, both human and agent, but it really depends on the code/application. At least for production/user...

        This is an excellent workflow! Mine isn't a lot different. In general I'd argue for more review, both human and agent, but it really depends on the code/application. At least for production/user facing code.

        One thing that's different... I don't use auto memory and I don't have agents write skills or subagent definitions. I've never been satisfied with their results in either making memories or building scaffolding. Personally I think the "self improving" angle is a trap, at least for the time being. Agents suck at it. Instead I manually formalize things that come up repeatedly into context files that are read/injected in various ways.

        Working with a large, mature codebase gives Claude a lot of precedent to follow

        The value of this cannot be overstated, and this is more true with each generation of models. If you give them access to a comprehensive codebase with patterns, utilities and established architecture, they will do a reasonably good job of following conventions. Which removes a lot of opportunities to go off the rails.

        For greenfield projects, even though you don't have to write code, it doesn't mean you can't! It's a great way to give agents a pile of conventions, patterns and design philosophies very quickly.

        One thing I'd add to the list:

        1. Create design docs about larger areas of the codebase. This is one area where I let the agents do most of it because it's essentially summarization, which they're great at, and there aren't too many opportunities for mistakes that might matter. It's very useful to be able to say "read X file" and have the agent immediately understand a particular feature, functionality or system and where it exists in the codebase. Cuts down on both time and tokens.
        1 vote
        1. DrStone
          Link Parent
          Always comforting to hear people arrived at a similar usages and conclusions. Either we're doing something right, or we're at least in the same wrong boat together! Yeah, memories are a bit goofy....

          Always comforting to hear people arrived at a similar usages and conclusions. Either we're doing something right, or we're at least in the same wrong boat together!

          Yeah, memories are a bit goofy. They seem to be useful short term, helping to bridge the gap between workdays and task switches, but can come back to bite weeks down the line when they're no longer in sync with the current state of the world. I try to get it to write as much as possible and useful to the plan files, and much more rarely to the agent files after hitting the same problem or deriving the same information over and over again.

          For skills and the agent files, I had Claude write the prose, but built them with nearly the same flow as the feature flow. Detailed objective scope and requirements, known bits of information or where to gather them from, desired behavior, etc. up front, iterative back and forth, cold reading reviews by both me and Claude, and tweaks over time based on their performance. I think it's important for people to use the AI tools and harness as vanilla as possible for a while to find what fits you personally rather than cargo-culting a bunch of skills and prompts and agent file "hacks".

          Definitely agree about the design docs for large areas of the code base, as well as the boundaries and interactions between code bases if your project has multiple repos/applications. Before I did that, anything non-trivial would either get confused if I didn't tell it explicitly where things lived, or would spend time and tokens poking around every time. For the same reason, I keep a database schema dump on hand so it can do some quick grepping to find most things.

          Oh, and I just remembered another tidbit, agents are great with CLI tools. I rarely need to reach for some feature-subset MCP when the thing has a mature cli tool I can install. Plus I don't have to keep authentication info in some plaintext MCP config.

    3. [4]
      skybrian
      (edited )
      Link Parent
      My tip is to add this to AGENTS.md: And in the first prompt I always ask a question. If it's about a bug it's "why did this happen?" If it's a feature request, I describe the change and say...

      My tip is to add this to AGENTS.md:

      If a prompt contains questions, stop after answering the questions, so the user has a chance to reply before you edit any more code.

      And in the first prompt I always ask a question. If it's about a bug it's "why did this happen?" If it's a feature request, I describe the change and say "what's involved in making this change?" If there's an API change then I ask "How should we change the API?" I don't tell it to go ahead until I'm pretty sure I understand what it will do.

      Also, if it's a new project, I write a plan.md first and we go over it, making multiple passes. Then I ask "are there any important questions to answer before starting work?" And I edit the plan myself until it's good enough to start Phase 1, because coding agents tend to put in excessive detail.

      I guess Claude Code has a /plan mode, but my coding agent doesn't and I find I don't need it. I just make sure to ask a question if I don't want it to start right away.

      Then again I'm just writing web apps, not anything complicated.

      1 vote
      1. [2]
        bme
        Link Parent
        Interesting idea. I'll give it a whirl. I tend to just roll with auto-mode because I've found empirically (literally drag racing the same problem) that by the time my colleagues have finished...

        Interesting idea. I'll give it a whirl. I tend to just roll with auto-mode because I've found empirically (literally drag racing the same problem) that by the time my colleagues have finished their glorious round of dialectics with some planning mode i have iterated my way to a better design and solution just by hammering on the problem and getting deeper into the problem than can be discovered by looking at from afar.

        1. skybrian
          Link Parent
          Well, I'm a hobbyist programmer so I have no deadlines and I'm not competing with anyone. So that's a different situation.

          Well, I'm a hobbyist programmer so I have no deadlines and I'm not competing with anyone. So that's a different situation.

          2 votes
      2. Daedalus_1
        Link Parent
        Also make sure to look into gitnexus.

        Also make sure to look into gitnexus.

  2. post_below
    Link
    The coding agent landscape has changed a lot in the past year, but in my experience it's still true that agents make dumb decisions. Less than before, but it's a long way from going away. Because...

    The coding agent landscape has changed a lot in the past year, but in my experience it's still true that agents make dumb decisions. Less than before, but it's a long way from going away.

    Because of that, if you're not keeping track of what it's doing, dumb decisions are going to end up coded into your project. You can mitigate some of the problem by creating detailed enough plans that there aren't a lot of decisions to make, but as soon as an agent runs into a blocker, it will start making decisions. Which you'll have no idea is happening if you aren't in the loop. This is actually more true of the current frontier models because they're trained to keep working through issues more aggressively than previous generations.

    I agree with the author that's it's no longer necessary to review every single line of code, at least in some circumstances (boilerplate mostly). The latest models have much better taste than models from even 6 months ago. But I completely disagree with the framing that reviewing code slows down your velocity and therefore makes you worse. Velocity is not the only metric! I feel like this comes from a corporate "engineers are replaceable cogs" mindset. There are a lot of situations where going slow enough to maintain a mental model, reason sufficiently about choices and tradeoffs and continually update and maintain decent architecture are really really important. In those situations you're not "falling behind" by not cramming the maximum amount of coding into an 8 window, you're being a responsible engineer.

    7 votes
  3. skybrian
    Link
    From the article:

    From the article:

    So mine is a trick. People feel more and more programming is completely modified by AI and don’t know what they should do, if they can really start coding in a completely different way, without looking much at the code as their main output. They feel like they are betraying their own field. So my intention is to arrive and say “look at me, In can write code, you know, I’m not hiding behind AI: yet, things changed, it’s not your weakness, it’s not that you are AI-pilled. It is just that our field is evolving in an incredible *and* painful (but also joyful) direction”.

    This is why yesterday, on X, I said that I believe many programmers at this point have less impact they could have because they look at the code. I truly believe into that. And note that this does not mean to vibe code something just asking for the final product. The point is: if you control the ideas of your software, looking at the code itself is suboptimal and often pointless. For the following reasons:

    4 votes
  4. all_summer_beauty
    (edited )
    Link
    Speaking as an amateur, my approach has been to not touch LLMs for code help unless I'm absolutely desperate, and part of his closing vindicates that choice, I feel: I take that to mean, roughly,...

    Speaking as an amateur, my approach has been to not touch LLMs for code help unless I'm absolutely desperate, and part of his closing vindicates that choice, I feel:

    I have a doubt only regarding young programmers that don't have enough experience, and can't build a mental model. We don't know, yet, if they will require or not to understand very well how a given piece of code works, but I believe they should learn how to write programs.

    I take that to mean, roughly, "If you couldn't write the code/solve the problem yourself, you shouldn't be asking the LLM to do it for you." So while I find a lot of what he says in there disturbing (e.g. "I believe many programmers at this point have less impact they could have because they look at the code. [...] The working day is 8 hours. If you read the code, it is a tradeoff."), the ending bit kind of qualifies the rest of it in a way that I'm slightly less uncomfortable with, I guess?

    Idk. Looking forward to hearing what the professionals on here think.

    2 votes
  5. fxgn
    (edited )
    Link
    There is one type of cancerous coding pattern all models make, no matter how good the model is, and no matter how detailed the AGENTS.md. I found it's best described here: The models would rather...

    There is one type of cancerous coding pattern all models make, no matter how good the model is, and no matter how detailed the AGENTS.md. I found it's best described here:

    Present-day models tend to produce code that is too defensive, too complex, too local in its reasoning. They avoid strong invariants. They add fallbacks instead of making bad states impossible.
    [...]
    Furthermore it’s well understood that models tend to observe some local failure and add a local defense. Karpathy mentioned how they are “mortally terrified of exceptions”. In systems with important invariants, especially persisted data formats or core infrastructure, the right fix is not “handle every malformed case.” The right fix is to make the malformed case unrepresentable or impossible to write in the first place. Yet even with a lot of manual steering, that type of code does not come out of LLMs naturally, and even if the code comes out naturally like that, they will still attempt to handle now impossible errors.

    The models would rather add a if (property !== null) check all over the code than just add that to the type definition and handle the null case in the single place it could possibly arise.

    Until this is somehow improved, there is no way I'd ever prompt AI without fixing the generated code. Sometimes the model does a bad job and I have to refactor most of it myself, sometimes it does a good job and I dont. In any case, I don't think I've ever had a situation where I didn't have to first remove a bunch of useless defensive checks from the generated code.

    It seems like the "best" models are even worse in this regard. I've tried GPT-5.5, but I keep going back to DeepSeek v4, because GPT keeps insisting on adding that stuff all over even when I remove it.