Okay, we've seen this before. Blackhat hacker isn't careful with their opsec, leaks personal data, and gets caught. What is interesting here, is how not only the opsec problems but the entire...
What ultimately led to EncryptHub’s unmasking was a catastrophic series of operational security failures, (...)
Okay, we've seen this before. Blackhat hacker isn't careful with their opsec, leaks personal data, and gets caught.
What is interesting here, is how not only the opsec problems but the entire malware came to be in the first place:
The most fascinating aspect of this case is EncryptHub’s extensive reliance on ChatGPT as a “partner in crime.”
The AI assistant was leveraged to create nearly every component of his malicious infrastructure, from writing malware code to configuring Telegram bots, command and control servers, phishing sites, and onion services.
This kinda nicely ties into yesterday's thread about the problems of vibe coding. Of course stuff like password-reuse is not to be blamed on any LLM, but apparently the infrastructure has been mostly (and insecurely) setup by ChatGPT as well, so the whole vibe-coding aspect at least contributed to that guy's downfall.
But beyond that, the fact we're seing AI generated malware in the wild now imo also raises the question wether attempting to 'lock down' what kind of responses LLMs may give is even worth it. So much energy has been poured into preventing AIs from giving certain answers, and yet people who use them maliciously are usually the ones who find the workarounds anyway, while average users suffer from misguided restrictions on rather normal topics, or from not being given answers that could be used both legitimately and maliciously.
Finally I find it very funny how we see a malicious actor turning to an LLM for moral guidance here:
In one particularly revealing conversation, EncryptHub asked the AI to evaluate whether he was better suited to be a “black hat or white hat” hacker, even confessing to criminal activities and exploits he had developed.
Okay, we've seen this before. Blackhat hacker isn't careful with their opsec, leaks personal data, and gets caught.
What is interesting here, is how not only the opsec problems but the entire malware came to be in the first place:
This kinda nicely ties into yesterday's thread about the problems of vibe coding. Of course stuff like password-reuse is not to be blamed on any LLM, but apparently the infrastructure has been mostly (and insecurely) setup by ChatGPT as well, so the whole vibe-coding aspect at least contributed to that guy's downfall.
But beyond that, the fact we're seing AI generated malware in the wild now imo also raises the question wether attempting to 'lock down' what kind of responses LLMs may give is even worth it. So much energy has been poured into preventing AIs from giving certain answers, and yet people who use them maliciously are usually the ones who find the workarounds anyway, while average users suffer from misguided restrictions on rather normal topics, or from not being given answers that could be used both legitimately and maliciously.
Finally I find it very funny how we see a malicious actor turning to an LLM for moral guidance here:
xD