I use caveman mode unironically, and care not about performance or token-saving, but because without it the pychophantic babble-chatter is unbearable. I've noticed if I send Claude on a bug hunt...
I use caveman mode unironically, and care not about performance or token-saving, but because without it the pychophantic babble-chatter is unbearable.
I've noticed if I send Claude on a bug hunt without caveman, and I give even the slightest hint what I think the problem might be, it presumes I'm a supergenious and blows smoke up my bum from here to Timbuktu.
I switched to GPT-5.5 recently, because I can connect the $20/month ChatGPT subscription that I already had to the coding agent I'm using, with the result that there's no extra charge for me. It...
I switched to GPT-5.5 recently, because I can connect the $20/month ChatGPT subscription that I already had to the coding agent I'm using, with the result that there's no extra charge for me. It seems okay and is a lot more no-nonsense. One thing that takes a bit of getting used to is that it's somewhat more careful to follow whatever instructions you give it.
My current stack is Claude (Opus + Fable for a few more hours) for the hard stuff. pi.dev + Codex for clear tasks and Opencode + GLM-5.2 via z.ai, which seems to be somewhere between Sonnet 4.6...
My current stack is Claude (Opus + Fable for a few more hours) for the hard stuff. pi.dev + Codex for clear tasks and Opencode + GLM-5.2 via z.ai, which seems to be somewhere between Sonnet 4.6 and Opus in "intelligence"
In my experience, property-based testing usually finds Unicode bugs. String handling is hard to get right for all of Unicode, and parsing is also pretty error-prone. But for a low-stakes personal...
In my experience, property-based testing usually finds Unicode bugs. String handling is hard to get right for all of Unicode, and parsing is also pretty error-prone.
But for a low-stakes personal website that's not internationalized, the Unicode bugs are often using characters that I might not ever use, and I haven't found other bugs with fuzz testing.
I was really hoping to get some insight into testing LLMs, but its just about using LLMs to test. Very insightful regardless, would have been useful to me years ago.
I was really hoping to get some insight into testing LLMs, but its just about using LLMs to test. Very insightful regardless, would have been useful to me years ago.
I disagree, it is also about testing LLMs and a critique of most benchmarks. Unfortunately, it is also about why this is hard and sort of unsatisfying and inconclusive. Still, two people who have...
I disagree, it is also about testing LLMs and a critique of most benchmarks. Unfortunately, it is also about why this is hard and sort of unsatisfying and inconclusive.
Still, two people who have read this article should have a slightly easier time discussing LLM benchmarking with each other.
I mean just that it’s pretty common knowledge that open benchmarks are useless, even when testing on specific data benchmarks are useless. What Ive been doing is designing questions in a way that...
I mean just that it’s pretty common knowledge that open benchmarks are useless, even when testing on specific data benchmarks are useless.
What Ive been doing is designing questions in a way that it gives me a functionally correct answer that I can pass/fail since I work on a RAG system. Things like “the response to this query should contain this type of graph”. It woulda been neat for him to talk about something close to what Ive come up with. I’m sure I’m not the only one doing it.
That’s going to be most of us in 5 years, I suspect. The companies are in market capture mode right now, selling access at a loss (subsidized temporarily by investor capital) to get everyone...
That’s going to be most of us in 5 years, I suspect. The companies are in market capture mode right now, selling access at a loss (subsidized temporarily by investor capital) to get everyone hooked. Eventually shareholders are going to demand profitability which means enshittification and higher prices.
Hopefully by the time that happens, local open-source alternatives will be a viable pathway out of the trap. But I’m not holding my breath for that.
I use caveman mode unironically, and care not about performance or token-saving, but because without it the pychophantic babble-chatter is unbearable.
I've noticed if I send Claude on a bug hunt without caveman, and I give even the slightest hint what I think the problem might be, it presumes I'm a supergenious and blows smoke up my bum from here to Timbuktu.
I switched to GPT-5.5 recently, because I can connect the $20/month ChatGPT subscription that I already had to the coding agent I'm using, with the result that there's no extra charge for me. It seems okay and is a lot more no-nonsense. One thing that takes a bit of getting used to is that it's somewhat more careful to follow whatever instructions you give it.
My current stack is Claude (Opus + Fable for a few more hours) for the hard stuff. pi.dev + Codex for clear tasks and Opencode + GLM-5.2 via z.ai, which seems to be somewhere between Sonnet 4.6 and Opus in "intelligence"
This is a super insightful and interesting post. Hard to summarize or quote something as it touches on so many things.
In my experience, property-based testing usually finds Unicode bugs. String handling is hard to get right for all of Unicode, and parsing is also pretty error-prone.
But for a low-stakes personal website that's not internationalized, the Unicode bugs are often using characters that I might not ever use, and I haven't found other bugs with fuzz testing.
I was really hoping to get some insight into testing LLMs, but its just about using LLMs to test. Very insightful regardless, would have been useful to me years ago.
I disagree, it is also about testing LLMs and a critique of most benchmarks. Unfortunately, it is also about why this is hard and sort of unsatisfying and inconclusive.
Still, two people who have read this article should have a slightly easier time discussing LLM benchmarking with each other.
I mean just that it’s pretty common knowledge that open benchmarks are useless, even when testing on specific data benchmarks are useless.
What Ive been doing is designing questions in a way that it gives me a functionally correct answer that I can pass/fail since I work on a RAG system. Things like “the response to this query should contain this type of graph”. It woulda been neat for him to talk about something close to what Ive come up with. I’m sure I’m not the only one doing it.
Honestly I can’t afford an AI subscription.
That’s going to be most of us in 5 years, I suspect. The companies are in market capture mode right now, selling access at a loss (subsidized temporarily by investor capital) to get everyone hooked. Eventually shareholders are going to demand profitability which means enshittification and higher prices.
Hopefully by the time that happens, local open-source alternatives will be a viable pathway out of the trap. But I’m not holding my breath for that.