Interesting to see them shift away from Swift and towards Rust. They didn't mention it but I wonder if Rust's C++ interop is a reason why they switched to Rust. I read that Google donated $1M to...
Interesting to see them shift away from Swift and towards Rust. They didn't mention it but I wonder if Rust's C++ interop is a reason why they switched to Rust. I read that Google donated $1M to the Rust foundation with the goal of improving Rust/C++ interop so I assume the progress on that is going better than Swift.
Their use of AI here is interesting to see too. It's not a blind trust of AI to do everything but careful dispatch of AI to do things step by step with careful human validation.
I don't know Rust that well, but based on everything I've heard, it's entirely possible it's well-suited to AI code generation if you could insure the AI was never allowed to use the unsafe...
I don't know Rust that well, but based on everything I've heard, it's entirely possible it's well-suited to AI code generation if you could insure the AI was never allowed to use the unsafe keyword.
My gut says that the best-possible code generator is specifically trained on a single highly opinionated language which favors 1 way to do things.
I’m a web frontend guy who should keep his mouth shut about Rust, but I’ve had good results telling my AI agent (Codex) not to consider any work complete until all linter errors are resolved. In...
I’m a web frontend guy who should keep his mouth shut about Rust, but I’ve had good results telling my AI agent (Codex) not to consider any work complete until all linter errors are resolved. In my case that’s ESLint which is eminently flexible and I can codify pretty much whatever rules I want to be enforced… I assume similar tools are available for Rust?
the problem that has periodically appeared in the news is that nothing you say to the agent really prevents the agent from completly ignoring the rules. Git commit hooks with the configs...
the problem that has periodically appeared in the news is that nothing you say to the agent really prevents the agent from completly ignoring the rules. Git commit hooks with the configs untouchable might do the trick though.
Yeah, something like that is what I had in mind. I’ve got a pretty sparse AGENTS.md file which only includes the most essential rules so they don’t get lost in a crowded context dump. The...
Yeah, something like that is what I had in mind. I’ve got a pretty sparse AGENTS.md file which only includes the most essential rules so they don’t get lost in a crowded context dump. The instruction is as simple as “IMPORTANT: Don’t consider any work complete until a build completes successfully.” Then I’m relying on traditional tooling that I’ve crammed into the build pipeline, to require strict TypeScript typing (AI tends to play fast and loose with that by default), my configured lint rules, Prettier formatting, a WCAG a11y scan, and full E2E tests.
My project is tiny so that’s not egregious… it would be for a big codebase. And I have no idea what the workflow is like for actual compiled languages like Rust, maybe this whole suggestion is nonsense for that use case. All I know is, it’s been working for me. The AI is only tasked with kicking off the build and responding to failures. I don’t trust it with doing its own linting, type checking, code formatting, etc. but it works with those dedicated tools and they can be relied on to make sure nothing slips through the cracks.
Nah, not really. It won't output trivial buffer overflows and RCEs like it would in C, but the entire rich universe of logic errors remains open to it, and in a web browser, nearly every logic...
it's entirely possible [safe Rust] is well-suited to AI code generation
Nah, not really. It won't output trivial buffer overflows and RCEs like it would in C, but the entire rich universe of logic errors remains open to it, and in a web browser, nearly every logic error is a serious security issue.
^ I'm not the person you're replying to, but fwiw, we literally just had a severe (8.8) remote code execution CVE drop last week for Chromium that was caused by a use-after-free error. This is...
^ I'm not the person you're replying to, but fwiw, we literally just had a severe (8.8) remote code execution CVE drop last week for Chromium that was caused by a use-after-free error. This is more Rust advocacy than AI-Rust advocacy, but reducing the number of available footguns when authoring sensitive code is extremely helpful.
I'd make the argument that Rust is well suited to AI code generation because of human factors: devs who are knowledgeable in sensitive areas (rendering engines, virtualization frameworks, kernels, etc.) also despise Rust, and thoroughly reject using it on a day-to-day basis. If an LLM is able to do enough of the legwork for them that they don't feel as though their hands have been soiled by the dirty Rustaceans -- and that they're able to exercise their Supreme Intellect as the Senior Chief Technical Architect Engineer of their little kingdom -- then perhaps it'll incentivize further adoption, and therefore a reduction in severe CVEs caused by their preferred language.
... I may have a bit of a bone to pick, or twenty, I realize 😅
Oh, don't get me wrong, there is ample evidence that humans cannot write correct C or C++, that the errors they tend to write are catastrophic (arbitrary memory exposure or code execution), and...
Oh, don't get me wrong, there is ample evidence that humans cannot write correct C or C++, that the errors they tend to write are catastrophic (arbitrary memory exposure or code execution), and that Rust practically prevents those errors. Continuing to use C/C++ for security-sensitive projects is indefensible. I'm just pushing back against the idea that Rust prevents all bugs¹.
If the correctness of your program doesn't really matter², then I guess an LLM is more likely to produce output which produces wrong output but doesn't have surprise RCEs in it using Rust than using C. But for a web browser, the correctness of your program is a security issue. And so the fact that Rust effectively saves you from one particular class of very, very bad issues doesn't make it suitable as a target for LLM output.
¹ In my experience, this idea is not very prevalent among Rust users, but does seem to be quite prevalent among the hypothetical Rust users who exists in the minds of people who use phrases like "Rust Evangelism Strike Force".
² Though of course, if the correctness of your program doesn't really matter, surely exit(1) is simpler, clearer, and more robust.
I'm actually vibecoding a hot mess where the correctness doesn't matter much. In essence, capturing web requests and user entered data, and then cross-refrencing the api spec and spitting out a...
I'm actually vibecoding a hot mess where the correctness doesn't matter much.
In essence, capturing web requests and user entered data, and then cross-refrencing the api spec and spitting out a vendor-proprietary testing framework format.
The whole intent is that a user could reproduce their bug while this is running, and it spits out a 'close enough' representation that the person actually writing regression testing can clean it up, make it legible, and add it to the test suite.
I'm reminded of significant figures. I don't need down to the decimals, only to the 1,000s. The advantage of being able to hack together quickly outweighed the disadvantages, while still saving considerable time relative to parsing incomplete user reproduction steps with missing data.
Oof, yeah, agreed. I haven't personally run into that, but I hope I'm not coming across that way! Mmhm, fair enough. I suppose one thing that this project has in its favour -- re. suitability for...
I'm just pushing back against the idea that Rust prevents all bugs.
Oof, yeah, agreed. I haven't personally run into that, but I hope I'm not coming across that way!
But for a web browser, the correctness of your program is a security issue.
Mmhm, fair enough. I suppose one thing that this project has in its favour -- re. suitability for code generation -- well-ish defined standards? That way, one can ensure that the generated code can be tested against the spec, as opposed to the typical vibe coded application which kinda works, sometimes.
The failure mode on that was super interesting: So the AI's ability to follow instructions was dependent on the amount of data processed, resulting in different behavior in the test environment...
The failure mode on that was super interesting:
I said “Check this inbox too and suggest what you would archive or delete, don’t action until I tell you to.” This has been working well for my toy inbox, but my real inbox was too huge and triggered compaction. During the compaction, it lost my original instruction 🤦♀️
So the AI's ability to follow instructions was dependent on the amount of data processed, resulting in different behavior in the test environment versus the real thing. I can imagine ways to protect against that, such as prepending non-compacted prompts to every summary. But if you don't foresee needing to do that, you won't find out until it's too late.
(Also: more operations should be undoable, just in general. That helps humans, too!)
(sidebar: I really think an AI security researcher should probably have seen this coming) I think you raise an interesting point, somewhat tangentially -- perhaps that the root cause was not the...
(sidebar: I really think an AI security researcher should probably have seen this coming)
I think you raise an interesting point, somewhat tangentially -- perhaps that the root cause was not the skipped prompt, but the agent UX (whatsapp in particular)! We've known for ages that LLMs will just ignore instructions sometimes (mere minutes ago I was fighting with one to implement a refactoring just the way I wanted), and the systems they interface with need to guard against that. If WhatsApp was genuinely the best way to interact with this agent, having a "kill word" or whatnot that's hooked up to a watchdog would've been terribly easy to add as a requirement to OpenClaw! Ideally that would come up during a risk analysis of an engineering project (i.e. what happens if the LLM starts doing unaligned shenanigans; I might not be beside my Mac Mini). That much feels predictable -- it's the same sort of concern which keeps me from leaving a 3d printer on when I'm out of the house.
I've had to fight developers on this with MCP integrations. I've repeatedly heard, "we'll give it sudo access to our database with clear instructions on which commands to run!" Instead, they...
I've had to fight developers on this with MCP integrations. I've repeatedly heard, "we'll give it sudo access to our database with clear instructions on which commands to run!" Instead, they should to give it tightly limited APIs with limited read access, etc.
Somehow, LLMs have made people unlearn everything we know about security.
Hah, very true XD sometimes -- in my dark moments -- I wonder if anyone was ever learning anything, or if they perceived themselves as jumping through arbitrary hoops that "the security team" told...
Somehow, LLMs have made people unlearn everything we know about security.
Hah, very true XD sometimes -- in my dark moments -- I wonder if anyone was ever learning anything, or if they perceived themselves as jumping through arbitrary hoops that "the security team" told them to. I may not have the highest opinion of most software developers 😅
Oh turned the how have tables
Swiftbros it has never been more over
I really struggled to read that
Interesting to see them shift away from Swift and towards Rust. They didn't mention it but I wonder if Rust's C++ interop is a reason why they switched to Rust. I read that Google donated $1M to the Rust foundation with the goal of improving Rust/C++ interop so I assume the progress on that is going better than Swift.
Their use of AI here is interesting to see too. It's not a blind trust of AI to do everything but careful dispatch of AI to do things step by step with careful human validation.
I don't know Rust that well, but based on everything I've heard, it's entirely possible it's well-suited to AI code generation if you could insure the AI was never allowed to use the unsafe keyword.
My gut says that the best-possible code generator is specifically trained on a single highly opinionated language which favors 1 way to do things.
I’m a web frontend guy who should keep his mouth shut about Rust, but I’ve had good results telling my AI agent (Codex) not to consider any work complete until all linter errors are resolved. In my case that’s ESLint which is eminently flexible and I can codify pretty much whatever rules I want to be enforced… I assume similar tools are available for Rust?
the problem that has periodically appeared in the news is that nothing you say to the agent really prevents the agent from completly ignoring the rules. Git commit hooks with the configs untouchable might do the trick though.
Yeah, something like that is what I had in mind. I’ve got a pretty sparse
AGENTS.mdfile which only includes the most essential rules so they don’t get lost in a crowded context dump. The instruction is as simple as “IMPORTANT: Don’t consider any work complete until a build completes successfully.” Then I’m relying on traditional tooling that I’ve crammed into the build pipeline, to require strict TypeScript typing (AI tends to play fast and loose with that by default), my configured lint rules, Prettier formatting, a WCAG a11y scan, and full E2E tests.My project is tiny so that’s not egregious… it would be for a big codebase. And I have no idea what the workflow is like for actual compiled languages like Rust, maybe this whole suggestion is nonsense for that use case. All I know is, it’s been working for me. The AI is only tasked with kicking off the build and responding to failures. I don’t trust it with doing its own linting, type checking, code formatting, etc. but it works with those dedicated tools and they can be relied on to make sure nothing slips through the cracks.
Nah, not really. It won't output trivial buffer overflows and RCEs like it would in C, but the entire rich universe of logic errors remains open to it, and in a web browser, nearly every logic error is a serious security issue.
^ I'm not the person you're replying to, but fwiw, we literally just had a severe (8.8) remote code execution CVE drop last week for Chromium that was caused by a use-after-free error. This is more Rust advocacy than AI-Rust advocacy, but reducing the number of available footguns when authoring sensitive code is extremely helpful.
I'd make the argument that Rust is well suited to AI code generation because of human factors: devs who are knowledgeable in sensitive areas (rendering engines, virtualization frameworks, kernels, etc.) also despise Rust, and thoroughly reject using it on a day-to-day basis. If an LLM is able to do enough of the legwork for them that they don't feel as though their hands have been soiled by the dirty Rustaceans -- and that they're able to exercise their Supreme Intellect as the Senior Chief Technical Architect Engineer of their little kingdom -- then perhaps it'll incentivize further adoption, and therefore a reduction in severe CVEs caused by their preferred language.
... I may have a bit of a bone to pick, or twenty, I realize 😅
Oh, don't get me wrong, there is ample evidence that humans cannot write correct C or C++, that the errors they tend to write are catastrophic (arbitrary memory exposure or code execution), and that Rust practically prevents those errors. Continuing to use C/C++ for security-sensitive projects is indefensible. I'm just pushing back against the idea that Rust prevents all bugs¹.
If the correctness of your program doesn't really matter², then I guess an LLM is more likely to produce output which produces wrong output but doesn't have surprise RCEs in it using Rust than using C. But for a web browser, the correctness of your program is a security issue. And so the fact that Rust effectively saves you from one particular class of very, very bad issues doesn't make it suitable as a target for LLM output.
¹ In my experience, this idea is not very prevalent among Rust users, but does seem to be quite prevalent among the hypothetical Rust users who exists in the minds of people who use phrases like "Rust Evangelism Strike Force".
² Though of course, if the correctness of your program doesn't really matter, surely
exit(1)is simpler, clearer, and more robust.I'm actually vibecoding a hot mess where the correctness doesn't matter much.
In essence, capturing web requests and user entered data, and then cross-refrencing the api spec and spitting out a vendor-proprietary testing framework format.
The whole intent is that a user could reproduce their bug while this is running, and it spits out a 'close enough' representation that the person actually writing regression testing can clean it up, make it legible, and add it to the test suite.
I'm reminded of significant figures. I don't need down to the decimals, only to the 1,000s. The advantage of being able to hack together quickly outweighed the disadvantages, while still saving considerable time relative to parsing incomplete user reproduction steps with missing data.
Oof, yeah, agreed. I haven't personally run into that, but I hope I'm not coming across that way!
Mmhm, fair enough. I suppose one thing that this project has in its favour -- re. suitability for code generation -- well-ish defined standards? That way, one can ensure that the generated code can be tested against the spec, as opposed to the typical vibe coded application which kinda works, sometimes.
The AI tremendously helped but was heavily guided. This is what happens when you leave AI unsupervised
The failure mode on that was super interesting:
So the AI's ability to follow instructions was dependent on the amount of data processed, resulting in different behavior in the test environment versus the real thing. I can imagine ways to protect against that, such as prepending non-compacted prompts to every summary. But if you don't foresee needing to do that, you won't find out until it's too late.
(Also: more operations should be undoable, just in general. That helps humans, too!)
(sidebar: I really think an AI security researcher should probably have seen this coming)
I think you raise an interesting point, somewhat tangentially -- perhaps that the root cause was not the skipped prompt, but the agent UX (whatsapp in particular)! We've known for ages that LLMs will just ignore instructions sometimes (mere minutes ago I was fighting with one to implement a refactoring just the way I wanted), and the systems they interface with need to guard against that. If WhatsApp was genuinely the best way to interact with this agent, having a "kill word" or whatnot that's hooked up to a watchdog would've been terribly easy to add as a requirement to OpenClaw! Ideally that would come up during a risk analysis of an engineering project (i.e. what happens if the LLM starts doing unaligned shenanigans; I might not be beside my Mac Mini). That much feels predictable -- it's the same sort of concern which keeps me from leaving a 3d printer on when I'm out of the house.
I've had to fight developers on this with MCP integrations. I've repeatedly heard, "we'll give it sudo access to our database with clear instructions on which commands to run!" Instead, they should to give it tightly limited APIs with limited read access, etc.
Somehow, LLMs have made people unlearn everything we know about security.
Hah, very true XD sometimes -- in my dark moments -- I wonder if anyone was ever learning anything, or if they perceived themselves as jumping through arbitrary hoops that "the security team" told them to. I may not have the highest opinion of most software developers 😅