One of the issues Daniel identifies has been poignant for me when it comes to AI. For the last several decades, users of computers have gotten very used to “I do X, and Y happens”. Every piece of...
One of the issues Daniel identifies has been poignant for me when it comes to AI.
As a lot of these reporters seem to genuinely think they help out, apparently blatantly tricked by the marketing of the AI hype-machines, it is not certain that removing the money from the table is going to completely stop the flood.
For the last several decades, users of computers have gotten very used to “I do X, and Y happens”. Every piece of software developed for the average user is supposed to behave consistently, and the computer should do what you tell it to do. If that doesn’t happen, it’s a bug.
I’m unsurprised that many users genuinely believe they can use these tools to find real vulnerabilities in software and think they are novel for doing so. In my day-to-day interactions, people want to use AI and get so, so frustrated when the AI model gets something wrong.
I studied probabilistic and randomized algorithms in undergrad, and while the math is incredibly interesting, no software developer ever uses them; even when you can’t get a correct answer all the time, you’d rather have an algorithm that outputs within a known error bound than one that generates output correctly a known percentage of the time.
In the modern world of consumer-facing software, AI is very quickly becoming just marketing. The tools now have a chatbot frontend, but the chatbot just calls some traditionally-written software tool that has consistent, non-probabilistic outputs. If it cannot find a tool, say that you can’t do the task. Years of consumer habit dictate that the computer can’t be wrong, and while eventually users might get used to AI’s errors, I’d be surprised if it happens anytime soon.
I know this is completely tangential to your main point (which I agree with), but this tickled me because I do use random algorithms all the time (coordinating distributed systems). Almost any...
no software developer ever uses them
I know this is completely tangential to your main point (which I agree with), but this tickled me because I do use random algorithms all the time (coordinating distributed systems). Almost any time you want to coordinate "who goes next" over a lossy channel randomness can dramatically simplify what you are doing and improve latency by reducing messaging round trips. After all, you don't need to exchange information to draw a random number :)
Thanks for clarifying and going more in depth! I suppose I could have been a bit more precise when I said “random algorithms”. I didn’t mean to imply randomness is never used in algorithm design...
Thanks for clarifying and going more in depth! I suppose I could have been a bit more precise when I said “random algorithms”. I didn’t mean to imply randomness is never used in algorithm design or that randomness isn’t a useful concept in software.
Specifically, I meant the class of algorithms that sample a random number and with correctness that depends on which random number was sampled.
It’s also untrue that randomized algorithms aren’t used in software development, but pretty much anytime they are used, they have a deterministic fallback. Raft is interesting because it doesn’t do so, but just repeats the algorithm so that the probability of non-consensus goes to zero over time. This is slightly different than lots of randomized algorithms since you know a consensus exists, just not what it is. Randomized algorithms for problems in RP or co-RP can never determine if a solution does not exist or exists, respectively.
Either way, having errors is so unacceptable that you’d rather detect them and fix them than leave them be. Thanks for adding on and providing resources for people to learn more!
A follow up to the blog post of last year by Daniel Stenberg, lead developer for cURL. He laments the difficulty of triaging bug bounty reports now that LLM can be leveraged to make fake reports...
He laments the difficulty of triaging bug bounty reports now that LLM can be leveraged to make fake reports look far more professional.
In early July, about 5% of the submissions in 2025 had turned out to be genuine vulnerabilities. The valid-rate has decreased significantly compared to previous years.
While we are not going to do anything rushed or in panic immediately, there are reasons for us to consider changing the setup. Maybe we need to drop the monetary reward?
People mention charging a fee for the right to submit a security vulnerability (that could be paid back if a proper report). That would probably slow them down significantly sure, but it seems like a rather hostile way for an Open Source project that aims to be as open and available as possible. Not to mention that we don’t have any current infrastructure setup for this – and neither does HackerOne. And managing money is painful.
He also provides a list of bug bounty example cases that are frustrating to read even when you are not forced to deal with this on a daily basis
They were already struggling with problems adjacent to this before LLMs became widespread and sounded convincing-enough… I can only imagine the sheer amount of time wasted at a project of cURL’s...
They were already struggling with problems adjacent to thisbefore LLMs became widespread and sounded convincing-enough… I can only imagine the sheer amount of time wasted at a project of cURL’s scale.
One of the issues Daniel identifies has been poignant for me when it comes to AI.
For the last several decades, users of computers have gotten very used to “I do X, and Y happens”. Every piece of software developed for the average user is supposed to behave consistently, and the computer should do what you tell it to do. If that doesn’t happen, it’s a bug.
I’m unsurprised that many users genuinely believe they can use these tools to find real vulnerabilities in software and think they are novel for doing so. In my day-to-day interactions, people want to use AI and get so, so frustrated when the AI model gets something wrong.
I studied probabilistic and randomized algorithms in undergrad, and while the math is incredibly interesting, no software developer ever uses them; even when you can’t get a correct answer all the time, you’d rather have an algorithm that outputs within a known error bound than one that generates output correctly a known percentage of the time.
In the modern world of consumer-facing software, AI is very quickly becoming just marketing. The tools now have a chatbot frontend, but the chatbot just calls some traditionally-written software tool that has consistent, non-probabilistic outputs. If it cannot find a tool, say that you can’t do the task. Years of consumer habit dictate that the computer can’t be wrong, and while eventually users might get used to AI’s errors, I’d be surprised if it happens anytime soon.
I know this is completely tangential to your main point (which I agree with), but this tickled me because I do use random algorithms all the time (coordinating distributed systems). Almost any time you want to coordinate "who goes next" over a lossy channel randomness can dramatically simplify what you are doing and improve latency by reducing messaging round trips. After all, you don't need to exchange information to draw a random number :)
Probably the most widely deployed random algorithm is the leader election phase of raft, but a very fun non networking example is improving heap usage in native programs using randomised layouts and page merging
Thanks for clarifying and going more in depth! I suppose I could have been a bit more precise when I said “random algorithms”. I didn’t mean to imply randomness is never used in algorithm design or that randomness isn’t a useful concept in software.
Specifically, I meant the class of algorithms that sample a random number and with correctness that depends on which random number was sampled.
It’s also untrue that randomized algorithms aren’t used in software development, but pretty much anytime they are used, they have a deterministic fallback. Raft is interesting because it doesn’t do so, but just repeats the algorithm so that the probability of non-consensus goes to zero over time. This is slightly different than lots of randomized algorithms since you know a consensus exists, just not what it is. Randomized algorithms for problems in RP or co-RP can never determine if a solution does not exist or exists, respectively.
Either way, having errors is so unacceptable that you’d rather detect them and fix them than leave them be. Thanks for adding on and providing resources for people to learn more!
A follow up to the blog post of last year by Daniel Stenberg, lead developer for cURL.
He laments the difficulty of triaging bug bounty reports now that LLM can be leveraged to make fake reports look far more professional.
He also provides a list of bug bounty example cases that are frustrating to read even when you are not forced to deal with this on a daily basis
They were already struggling with problems adjacent to this before LLMs became widespread and sounded convincing-enough… I can only imagine the sheer amount of time wasted at a project of cURL’s scale.