So ... not trying to sound like an AI apologist, but does this suggest that LLMs would be fantastic as a cheap red team to throw at nearly everything for hardening? I'm aware that software...
So ... not trying to sound like an AI apologist, but does this suggest that LLMs would be fantastic as a cheap red team to throw at nearly everything for hardening? I'm aware that software vulnerability detection with LLMs has been a dud (see the curl bug bounty conversation)), but maybe network security is a better fit? They're probably better capable of launching spearfishing and social engineering attacks than existing tools, at least.
OpenAI just announced a security product. It's an arms race. Can the good guys secure their code faster than the bad guys can find exploits? The AI companies are trying to tilt the odds in favor...
It's an arms race. Can the good guys secure their code faster than the bad guys can find exploits? The AI companies are trying to tilt the odds in favor of defense, but they are also unwittingly helping the bad guys.
Hrm. Perhaps? I see that as a distinct product from what occurred here (but it does seem promising in its own right!). Without further details, I'd imagine that the Mexican agency "hacks" were the...
Hrm. Perhaps? I see that as a distinct product from what occurred here (but it does seem promising in its own right!).
Without further details, I'd imagine that the Mexican agency "hacks" were the old standard stuff: ancient PHP servers running default passwords, easily spearfished employees with misconfigured email servers, critical services that don't rely on mutual authentication etc. Finding critical CVEs, like with Codex Security, is pretty cool! But more important is ensuring that your customers don't freely publish internal credentials on their homepage, allowing for hundreds of millions of SSNs to be leaked. For example :3
Even during curl's frustrations, there were good AI-assisted bug finds for the project. It was always that people who had no competency at finding bugs (and probably writing software) were abusing...
Even during curl's frustrations, there were good AI-assisted bug finds for the project. It was always that people who had no competency at finding bugs (and probably writing software) were abusing these tools in search of a quick payday.
Here's yet another troubling story about this "golden" era of AI. A hacker has exploited Anthropic's Claude chatbot to carry out attacks against Mexican government agencies, according to a report by Bloomberg. This resulted in the theft of 150GB of official government data, including taxpayer records, employee credentials and more.
The hacker used Claude to find vulnerabilities in government networks and to write scripts to exploit them. It also tasked the chatbot with finding ways to automate data theft, as indicated by cybersecurity company Gambit Security. This started in December and continued for around a month.
It looks like the hacker was able to essentially jailbreak Claude with prompts, finally bypassing the chatbot's guardrails. Claude originally refused the nefarious demands until eventually relenting.
[...]
Anthropic has investigated the claims, disrupted the activity and banned all of the accounts involved, according to a company representative. The spokesperson also said that its latest model, Claude Opus 4.6, includes tools to disrupt this kind of misuse.
It's also been reported that this hacker used ChatGPT to supplement the attacks, using OpenAI's chatbot to gather information on how to move through computer networks, determine which credentials were needed to access systems and how to avoid detection. OpenAI says it has identified attempts by the hacker to violate its usage policies and that the tools refused to comply.
The hacker remains unidentified. The attacks haven't been attributed to a specific group, but Gambit Security did suggest they could be tied to a foreign government. It's also unclear what the hacker wants to do with all of that data.
So ... not trying to sound like an AI apologist, but does this suggest that LLMs would be fantastic as a cheap red team to throw at nearly everything for hardening? I'm aware that software vulnerability detection with LLMs has been a dud (see the curl bug bounty conversation)), but maybe network security is a better fit? They're probably better capable of launching spearfishing and social engineering attacks than existing tools, at least.
OpenAI just announced a security product.
It's an arms race. Can the good guys secure their code faster than the bad guys can find exploits? The AI companies are trying to tilt the odds in favor of defense, but they are also unwittingly helping the bad guys.
Hrm. Perhaps? I see that as a distinct product from what occurred here (but it does seem promising in its own right!).
Without further details, I'd imagine that the Mexican agency "hacks" were the old standard stuff: ancient PHP servers running default passwords, easily spearfished employees with misconfigured email servers, critical services that don't rely on mutual authentication etc. Finding critical CVEs, like with Codex Security, is pretty cool! But more important is ensuring that your customers don't freely publish internal credentials on their homepage, allowing for hundreds of millions of SSNs to be leaked. For example :3
Even during curl's frustrations, there were good AI-assisted bug finds for the project. It was always that people who had no competency at finding bugs (and probably writing software) were abusing these tools in search of a quick payday.
From the article:
[...]
It’s incredibly ironic that this was posted literally within a few hours of me creating an account for Claude.
What do you mean? What sort of account?
Claude pro. It was just a funny bit of synchronicity.