11 votes

The day I taught AI to read code like a Senior Developer

Topic removed by site admin

21 comments

  1. [15]
    creesch
    Link
    This isn't groundbreaking. In fact I feel it almost shouldn't even be worthy of a blog post. Or at the very least it lacks the introspection it needs. Because the underlying issue is that you have...

    This isn't groundbreaking. In fact I feel it almost shouldn't even be worthy of a blog post. Or at the very least it lacks the introspection it needs. Because the underlying issue is that you have been using LLM systems without properly thinking about how to use them properly. Effectively the "throw stuff at the wall and see if it sticks" attitude.

    That you need to provide an LLM with context is something so fundamental. That it amazes me that it wasn't given some initial consideration before implementing it.

    This might sound a bit harsh, but I am also having your previous blog post in mind. Where you outline your blind reliance on tools like cursor.

    25 votes
    1. [12]
      DawnPaladin
      Link Parent
      I found the post helpful. I'm not a senior developer, so the technique nmn describes for how to review PRs was new to me; I think it will come in handy. I'm surprised that grouping the files in a...

      I found the post helpful. I'm not a senior developer, so the technique nmn describes for how to review PRs was new to me; I think it will come in handy.

      I'm surprised that grouping the files in a PR by function would make such a difference. I'm going to have to experiment with my prompting more.

      Finally:

      This isn't groundbreaking. In fact I feel it almost shouldn't even be worthy of a blog post.

      What the fuck? Don't ever tell people their discoveries aren't good enough to blog about. Everyone is at different stages of learning. Something that reads as uninsightful to you is helpful for someone like me, because we're at different points in our learning journey. That's doubly true for a field like LLMs where tons of stuff is brand new and people are figuring out the basics every day.

      23 votes
      1. [10]
        creesch
        Link Parent
        Context is important here, as I said OP also posted a different blog post a few days ago. You are right that the field is brand new. It also is a field where people are jumping in head first...

        What the fuck? Don't ever tell people their discoveries aren't good enough to blog about.

        Context is important here, as I said OP also posted a different blog post a few days ago. You are right that the field is brand new. It also is a field where people are jumping in head first without thinking. Not to mention overselling what AI can do.

        If you look at OPs blog posts they very much position themselves as an expert on AI implementations and innovations. If at this point they haven't done their basic homework they are part of a what I consider a problem in how AI is shoehorned into everything these days without much care or consideration.

        Finally, whenever they post their own blog on here they post it and bail without any interaction.

        I very much will call that sort of stuff out, thank you very much.

        12 votes
        1. [6]
          DawnPaladin
          Link Parent
          Why would they want to interact here when this is the kind of comment they get? If you don't like the way someone is posting (or not posting) on Tildes, it's fine to say so. If you object to what...

          Finally, whenever they post their own blog on here they post it and bail without any interaction.

          Why would they want to interact here when this is the kind of comment they get?

          If you don't like the way someone is posting (or not posting) on Tildes, it's fine to say so. If you object to what they're saying, talk about that - we are here for discussion, after all. But telling someone their ideas aren't worthy of blogging about makes the internet worse. When you tell people not to blog, you get fewer blogs. Blogs are the best place to find humans writing in depth and at length to share knowledge. When they go away, all that's left is spam and corporations.

          You might think "okay, but we only want to encourage the good blogs, not the bad ones." That doesn't work. As with every other creative endeavor, you usually have to write a bunch of garbage before you can attain brilliance. If you discourage bad bloggers, you will never get any good ones.

          I don't think this blog post is bad. But even if I did, I would never tell the author they shouldn't have written it. Telling someone "this blog post isn't a good fit for our discussion forum" or even "your idea needs refinement" can be part of valid, constructive criticism that helps people improve. Telling someone "you shouldn't have bothered writing this" is not constructive, and it's out of line.

          3 votes
          1. [2]
            danke
            Link Parent
            This seems to be overly charitable and confusing cause with effect. Their other post here wherein they didn't engage at all wasn't a one-off, it's in line with their activity on HN. After...

            Why would they want to interact here when this is the kind of comment they get?

            This seems to be overly charitable and confusing cause with effect. Their other post here wherein they didn't engage at all wasn't a one-off, it's in line with their activity on HN. After achieving moderate success with a single blog post on HN, OP proceeded to spam a followup metastory 3 times [1] [2] [3] in four days, another blog post 3 times [4] [5] [6] in six days, two more meta posts about their one successful post [7] [8], and didn't engage with a single comment in their 117+ comment post 10 days ago [9]. I don't think @creesch's hostility is unwarranted, this just looks like a spammer chasing that fleeting high of internet virality.

            11 votes
            1. DawnPaladin
              Link Parent
              Fair enough. I haven't examined their HN behavior.

              Fair enough. I haven't examined their HN behavior.

              1 vote
          2. [3]
            creesch
            Link Parent
            They must get awful comments on all platforms then, not just Tildes. As I said here they seem to rarely really engage with any feedback on their writings. Which is their own choice, but on me...

            Why would they want to interact here when this is the kind of comment they get?

            They must get awful comments on all platforms then, not just Tildes. As I said here they seem to rarely really engage with any feedback on their writings. Which is their own choice, but on me impresses a certain look, not a good one at that.

            But telling someone their ideas aren't worthy of blogging about makes the internet worse.

            I mean, that isn't exactly what I wrote though. I said that I almost feel this is not worth a blog post. Simply because it lacks the introspection I would expect also given the context of their post from a few days ago. Which hasn't much to do with the idea itself, but the way they frame it, certainly as they position themselves as somewhat of an AI expert and senior developer with man-years of experience. From someone in that position I would expect better and that is what I am calling out.
            Again, context matters a lot here.

            Blogs are the best place to find humans writing in depth and at length to share knowledge. When they go away, all that's left is spam and corporations.

            I guess we are at a fundamental disagreement here. Good blogs are a minority and in my experience a large portion of blogs, certainly dev blogs, are blogs with corporate interests and many of them are borderline spam. Of those who don't have direct corporate ties many of them are very much written with the goal of reputation and perception building.

            Just because someone decides to put their thoughts on a blog doesn't make them more worthy or more insightful. In fact, I have seen it achieve the opposite. As soon as people start posting on a blog there is a higher potential they start doing it for their audience regardless of them actually having anything to say on a subject.
            The reverence of bloggers, certainly in the space of IT, is always something that I have found odd in that regard.

            Personally I find more value in communities like Tildes than in most blogs. That's not to say there aren't any good blogs out there. But it isn't a precious resource that should be protected at all cost, it already is a thoroughly polluted environment with all the worse aspects of the internet fully present and accounted for.

            But even if I did, I would never tell the author they shouldn't have written it. Telling someone "this blog post isn't a good fit for our discussion forum" or even "your idea needs refinement" can be part of valid, constructive criticism that helps people improve. Telling someone "you shouldn't have bothered writing this" is not constructive, and it's out of line.

            Again, I said it almost doesn't feel worthy of a blog post. Anyway, I think I have been more than clear enough.

            5 votes
            1. [2]
              DawnPaladin
              (edited )
              Link Parent
              I think "almost" is a weasel word that lets people say what they think without needing to stand behind their words. If you literally meant "it's good that you wrote this, because its quality is...

              I think "almost" is a weasel word that lets people say what they think without needing to stand behind their words. If you literally meant "it's good that you wrote this, because its quality is very slightly above the threshold of not being worth the electrons it's printed on," then fine, but that's not how it comes across.

              I can see where you're coming from about blogs. When you make your living as a developer, the line between altruistic knowledge-sharing and cynical self-promotion can be blurry. I too have seen lots of low-quality blogs. But I get enormous value out of the good ones, and I think you can't get the good without the bad.

              2 votes
              1. creesch
                Link Parent
                I get where you are coming from, I honestly do. At the same time I guess I have to keep repeating myself, context is important. I said more than the one sentence where you considered the word...

                I get where you are coming from, I honestly do. At the same time I guess I have to keep repeating myself, context is important. I said more than the one sentence where you considered the word "almost" to be a weasel word. I certainly added a lot more context in reply to others as well before you replied again.

                It seems to me that you have been focussing on the opening sentence alone with little regard to what followed. Yes, if all I had said was the initial sentence then I would have agreed that it is out of line. But it was not.

                4 votes
        2. [3]
          Lapbunny
          Link Parent
          My sibling in christ, they did it once before and this post has been up less than a day.

          Finally, whenever they post their own blog on here they post it and bail without any interaction.

          My sibling in christ, they did it once before and this post has been up less than a day.

          1 vote
          1. [2]
            creesch
            Link Parent
            I suppose that's on me for not clarifying that further. To be clear, even if it is only twice I still find it odd. However, I also noticed that on their blog they proudly display the scores of...

            I suppose that's on me for not clarifying that further. To be clear, even if it is only twice I still find it odd. However, I also noticed that on their blog they proudly display the scores of their posts on reddit and hackernews. As it turns out they do the same thing there and have done so more often.

            Enough for me to say that it looks like a pattern of self promotion and not participation in communities.

            5 votes
            1. Lapbunny
              Link Parent
              Aah, OK. Yeah, I'd agree on that then.

              Aah, OK. Yeah, I'd agree on that then.

      2. ali
        Link Parent
        I agree. Everyone should blog about whatever they like. It’s unnecessarily dismissive. Especially because it’s a good insight. Not everything is written for subject matter experts. I’ve written a...

        I agree. Everyone should blog about whatever they like. It’s unnecessarily dismissive.

        Especially because it’s a good insight. Not everything is written for subject matter experts.

        I’ve written a blog post on an insanely basic task with a library before and saw some company put it in their manual.

        8 votes
    2. [2]
      TonesTones
      Link Parent
      I think your post came across as more than a bit harsh. I’m effectively going to double down on what @DawnPaladin said. Note that this is less because of exactly what you wrote, and more because...

      I think your post came across as more than a bit harsh. I’m effectively going to double down on what @DawnPaladin said. Note that this is less because of exactly what you wrote, and more because it received more upvotes than the post itself. I’m speaking both to you and those that are endorsing the language of your critique.

      This isn't groundbreaking. In fact I feel it almost shouldn't even be worthy of a blog post.

      The implicated idea is bizarre to me? If you want to read novel work, you can read papers published in academic journals covering LLMs. That writing is held to that standard. Blog posts aren’t supposed to be groundbreaking.

      Blogs are generally one person’s experience. Writing and publishing both act as tools to further your own learning and potentially help others learn. We’re all on our own journey. If you feel like it’s not a post worth reading, just don’t vote on it.

      Because the underlying issue is that you have been using LLM systems without properly thinking about how to use them properly. Effectively the "throw stuff at the wall and see if it sticks" attitude.

      Most people use the “throw stuff at the wall and see if it sticks” attitude. It’s a good attitude to have! When someone is learning to play chess, you don’t tell them to study all of the professional lines and build out a principled approach.

      You tell them to go play chess. Do the thing! Make mistakes! Hang a queen! Or twenty! Levy Rozman (GothamChess) has a series where he laughs at bad games of chess. He makes it very explicit that: “We aren’t laughing at you. We’re laughing because we’ve all been there.” We’re all going to have to learn and struggle; maybe not with using LLMs, but with something.

      I recognize the author is a consultant in the software space, and that there is some responsibility that comes with that career position. I don’t think that experience means we can’t make mistakes and learn from them.

      This might sound a bit harsh, but I am also having your previous blog post in mind. Where you outline your blind reliance on tools like cursor.

      I suspect either you or some of your upvoters instinctually rejected this piece on grounds that it covers using AI in SWE to do “even more”. Reading it and recognizing that it wasn’t necessarily novel to you reinforced that bias.

      I understand that AI is scary, and that for some, it sucks that we can’t close Pandora’s Box. AI will solve some problems, but it might cause a lot more. The technology might simultaneously cause people to lose their jobs while making the people with the remaining jobs work harder to solve AI’s mistakes. I’m writing not because AI is good and “let’s all use it” to read code like senior developers.

      I’m writing because our dislike of a technology shouldn’t get in the way of how we treat other human beings. At the end of the day, @nmn is just another person doing their best. I might be fired if one of my students or colleagues wrote the above and I responded by saying they shouldn’t have wrote it at all. It’s a good thing I’m held to that standard. We should hold ourselves to that standard on Tildes as well.

      Some developers will use AI going forward. I’ve said it before and I’ll say it again; if you are able to use LLMs to do your job, the capitalists will find a way to replace you. That doesn’t mean we should mock others learning how to use the new tech. It’s a part of our world now. We will accept that and find a way to be good to each other anyway.

      6 votes
      1. creesch
        (edited )
        Link Parent
        As I said in reply to the other comment, context is important. As I am not about to repeat myself I invite you to read that comment first:...

        As I said in reply to the other comment, context is important. As I am not about to repeat myself I invite you to read that comment first: https://tildes.net/~comp/1lq4/the_day_i_taught_ai_to_read_code_like_a_senior_developer#comment-eryl

        As an added note, I don't dislike LLMs on principle. I use them on a regular basis and find them quite useful tools. In fact, one quick look at my history on Tildes will show you that I talk about them on a regular basis.

        What I do dislike is an industry trend of shoehorning them into every single application people can think of, often without too much thought and then overselling their capabilities and value.

        OPs attitude here, if you take their entire blog in account, very much falls within that category as far as I am concerned. I am not going to apologize for calling that out, if anything I should have made that context more clear.

        6 votes
  2. BroiledBraniac
    Link
    I often find anything using terms like "senior" in this way a bit derogatory. I've seen mids and juniors review code in the "right" way too. A lot of dev creators online like to write about skill...

    I often find anything using terms like "senior" in this way a bit derogatory. I've seen mids and juniors review code in the "right" way too. A lot of dev creators online like to write about skill levels like they are these little boxes, and it reads in this very dehumanizing way. This is coming from someone that's been titled in senior+ positions for years. I don't get why people write like this and don't know how to make them realize who they hurt along the way.

    16 votes
  3. [4]
    snake_case
    Link
    I think I missed some basic context here, are people really using copilot as a way of testing code? So far Ive only heard of people using it to write new code, not to review PRs. My company...

    I think I missed some basic context here, are people really using copilot as a way of testing code? So far Ive only heard of people using it to write new code, not to review PRs.

    My company doesn’t allow any stuff like this so Im kinda falling behind here.

    4 votes
    1. [3]
      creesch
      Link Parent
      You are indeed missing some basic context. CoPilot is just one of many tools that make use of LLMs (large language models). Can I assume you are familiar with chatGPT? This is more like giving...

      You are indeed missing some basic context. CoPilot is just one of many tools that make use of LLMs (large language models). Can I assume you are familiar with chatGPT? This is more like giving chatGPT your code with a nice prompt and context to review it. But then through APIs of course.

      2 votes
      1. [2]
        snake_case
        Link Parent
        Haha yeah I’m familiar with chat gpt I work on LLMs for a living, we just aren’t allowed to use copilot at work so I rely on news to keep me informed on how people are using it. I thought that...

        Haha yeah I’m familiar with chat gpt

        I work on LLMs for a living, we just aren’t allowed to use copilot at work so I rely on news to keep me informed on how people are using it.

        I thought that copilot just reads the code and gives you suggestions, I didn’t know that you could prompt it and make it run tests like this.

        4 votes
        1. creesch
          Link Parent
          They don't mention copilot though? Where are you getting that from?

          They don't mention copilot though? Where are you getting that from?

  4. Deely
    Link
    Hm, will be very interesting to run this model against some big open-source project. Why author didn't do it?

    Hm, will be very interesting to run this model against some big open-source project. Why author didn't do it?