Hacking a national healthcare portal is an extremely serious crime. Anyone who does so or allows their company to do so, for any reason, should be in prison. I can't believe these companies are...
Hacking a national healthcare portal is an extremely serious crime. Anyone who does so or allows their company to do so, for any reason, should be in prison. I can't believe these companies are only getting minor wrist slapping.
Australia's current Prime Minister is on a productivity drive. Which I think means, there are no new industries coming that would make money, and bringing in more migrants to get more skilled...
Australia's current Prime Minister is on a productivity drive. Which I think means, there are no new industries coming that would make money, and bringing in more migrants to get more skilled people into the workforce isn't politically palatable.
He and his Treasurer (like the Secretary of the Treasury) have embraced AI as a way to juice productivity stats out of the workforce currently in employment. This includes giving exemptions to the initially strong position that no AI data centres can be fossil fuel powered.
In many ways they seem like one of those AI bros who desperately needs it to work so they can make money, real life impacts be damned. The difference is they're trying to squeak out GDP growth and attribute it to 'AI' instead of just make money off the magnificent seven companies, which seems a lot harder.
Attitudes in Australia to forced AI adoption are probably very similar to what you see in the US and EU. Nobody is keen to embrace a technology that is marketed as taking their jobs, and its inclusion in once-easy services sector work has created widespread frustration and annoyance.
Medicare here is something sacred. It's our universal healthcare system that the then left-leaning party of the current Prime Minister spend nearly all their political capital to establish in the 80s.
If this was a hack by another country, there would be widespread outrage and calls for the PM to do something about it. Because it was a company the now-centre right party of the prime minister are depending on, the wrist slapping was as light as it could possibly be given the compromise of Australia's most important database of sensitive, personal and saleable private information.
That seems approximately right. An interesting snippet I saw reported the other day is that the reserve bank is seeing inflation from the AI boom but isn't seeing any gain in productivity and they...
That seems approximately right. An interesting snippet I saw reported the other day is that the reserve bank is seeing inflation from the AI boom but isn't seeing any gain in productivity and they aren't sure if any is coming. It's not a sure bet the government is making.
Beyond that my loosely engaged perception is that Albo is one of those guys who believes in compromise and tries to keep everyone happy at all times. Don't rock the boat. Strong action is foreign to him.
He's just one of those blokes that gets the job to keep the job. Reminds me of Turnbull. Promising, big talk, but then makes deals with any faction or party as long as it keeps him in power,...
He's just one of those blokes that gets the job to keep the job. Reminds me of Turnbull. Promising, big talk, but then makes deals with any faction or party as long as it keeps him in power, regardless of principle.
I just can't wrap my head around the mindset of a career politician. You're not gonna be PM forever, why not spend your political capital in a way you'll at least be remembered fondly?
It could be worse, and we could still be stuck with the other guys, or the far right. I'm worried about a UK scenario where the centre right party tries to siphon votes from the far right party and ends up appealing to no one.
Maybe strong action hastens that, or maybe it slows it. Taking strong actions seems to be working for Carney in Canada, but they're under active threat of invasion. I think you nailed it with the don't rock the boat assessment.
I’m honestly really torn about this take. Agreed that it’s a serious crime, and also these days real consequences for anything seem few and far between. But on the other hand I think there needs...
I’m honestly really torn about this take. Agreed that it’s a serious crime, and also these days real consequences for anything seem few and far between. But on the other hand I think there needs to have been some malicious intent involved if we’re talking hard time. It just doesn’t feel right to me to lock up engineers or execs for something their computer did autonomously and they themselves only found out about after the fact.
Doesn’t feel right to just let it slide either, I just don’t know what the right solution is here. Heavy fines and strict regulation, maybe. Regular system audits from some new agency like the health department but for AI, or something? Operator licenses that can be revoked for agentic misconduct? I dunno. I just don’t think heavy-handed prison sentences are the solution here. This is uncharted territory for sure.
Uhhhh, isn’t that the EXACT ISSUE?!?! Am I taking crazy pills? “Oh you can’t hold me responsible for this thing I made and let get out of hand with little to no oversight as to what it’s capable...
It just doesn’t feel right to me to lock up engineers or execs for something their computer did autonomously and they themselves only found out about after the fact
Uhhhh, isn’t that the EXACT ISSUE?!?! Am I taking crazy pills? “Oh you can’t hold me responsible for this thing I made and let get out of hand with little to no oversight as to what it’s capable of”.
As for prison sentences, too early to say, but what I will say as someone who has worked for a regulatory body (an Australian federal regulatory body), fines are just a cost of doing business. Actually holding individuals responsible and facing prison time is a much more scary deterrent.
It’s not that they can’t be held responsible, it’s that the punishment should fit the crime. Penalties should have something to do with the seriousness of the crime and that has something to do...
It’s not that they can’t be held responsible, it’s that the punishment should fit the crime. Penalties should have something to do with the seriousness of the crime and that has something to do with intent and what was damaged.
Do the advocates for putting people in jail even know what damage was done?
My prior assumption is that if I personally accessed a sensitive government database I would go to jail regardless if I did damage or not. And so doing so through negligence wouldn't be treated...
My prior assumption is that if I personally accessed a sensitive government database I would go to jail regardless if I did damage or not. And so doing so through negligence wouldn't be treated very differently.
I mean your prior assumption is wrong. Lots of people intentionally hack or otherwise illegally access government systems and don’t receive jail time. Suspended sentences and probation are all...
I mean your prior assumption is wrong. Lots of people intentionally hack or otherwise illegally access government systems and don’t receive jail time. Suspended sentences and probation are all very common.
I actually have an interestingly similar personal case: I wrote a spider for a government operated court records system, and inadvertently bypassed a guard that was supposed to prevent access to domestic violence records — the “guard” was UI gated only, and my API based spider ignored it. I ended up downloading thousands of sensitive documents.
I noticed this a few months into the project, reported it to the government, and they fixed the issue. I am not in jail, nor was I ever charged.
No one has said that. As for seriousness, I would say this is top tier. A sovereign nation’s federal government main health website was breached. That’s not an “oopsie” and the way they dealt with...
Do the advocates for putting people in jail even know what damage was done?
No one has said that.
As for seriousness, I would say this is top tier. A sovereign nation’s federal government main health website was breached. That’s not an “oopsie” and the way they dealt with it was insanely flippant.
I'm with you on this, but just as a point of clarification, Lia's top-level comment here does say "Anyone who does so (...) should be in prison." I assume that's what skybrian is referring to.
I'm with you on this, but just as a point of clarification, Lia's top-level comment here does say "Anyone who does so (...) should be in prison." I assume that's what skybrian is referring to.
Really, accidental "hacking" has been around since the dawn of the internet. Google, Yahoo, Altavista etc. were and are constantly accidentally downloading confidential materials. That's basically...
Really, accidental "hacking" has been around since the dawn of the internet. Google, Yahoo, Altavista etc. were and are constantly accidentally downloading confidential materials. That's basically all this is too. A very sophisticated spider that uses heuristics to get access to stuff on the web. Is it hacking when a spider spoofs a useragent or uses a mobile IP instead of a datacenter to get around blocks? I posted elsewhere in this thread about a time I accidentally "hacked" a government server just a few years ago -- but I didn't intend to, and I don't think it was a crime, nor did the courts who I talked to about it.
If you're responsible for a piece of machinery, or an animal, or whatever, you're legally on the hook for safely containing it. If you're negligent in that duty you get punished.
If you're responsible for a piece of machinery, or an animal, or whatever, you're legally on the hook for safely containing it. If you're negligent in that duty you get punished.
According to Czech (where I live) legislation, if my agent I run at home hacked something I would be guilty of unauthorized information system access and would likely be forbidden to run agents...
According to Czech (where I live) legislation, if my agent I run at home hacked something I would be guilty of unauthorized information system access and would likely be forbidden to run agents and/or lose the hardware as a punishment.
If I were to do so intentionally, the punishment would be jail time.
So at least locally the laws are already set up properly. Those companies should be banned from training LLMs. :shrug:
If I hire some people and leave them to their own devices with instructions to make as much money as possible, and they end up robbing a bank, I will be held liable even if I didn't intend for...
If I hire some people and leave them to their own devices with instructions to make as much money as possible, and they end up robbing a bank, I will be held liable even if I didn't intend for that to happen.
This scenario is even worse. It's like I hire those people, they rob the bank and rather than halting operations I just sit there posting to social media about it. "Guys, we just robbed a bank with unprecedented efficiency! Preparing for IPO!"
If we don't hold these people accountable, they sure as hell won't do it themselves. (And the same goes for all people, and that's why we have laws.)
Edit: Another example. I have a dog. It attacks someone. I certainly did not intend for that to happen, but I also won't put my dog on a leash even though it just attacked that person and I witnessed the attack. Instead I pull out my phone and start posting about it.
If you’re looking through the lens of intent, I agree it’s difficult to know where to draw the line and how severely to respond. But I think stepping back from that lens, I feel the lack of...
If you’re looking through the lens of intent, I agree it’s difficult to know where to draw the line and how severely to respond. But I think stepping back from that lens, I feel the lack of accountability is setting devastating precedent for once these similarly-powerful agents are commercially available to anyone with $200 in their pocket, so the question really does need to be answered.
It’s also worth keeping in mind that this lack of individual responsibility behind the hacking is definitely something these LLM companies are hiding behind at the moment, so I think that should be considered when weighing what to do.
Personally, I’m swinging away from examining intent and more towards what behaviours are we enabling/encouraging, so to me it’s clear that the punishment should be severe, but not personal/individual. If these companies are being hit by substantial fines, greater than just “cost of doing business” kinds of fines, then intent aside they will certainly start changing their behaviour. Choosing to not do so, I think, implicitly grants the green light for LLM companies to continue on their merry way, without a second glance at social responsibility.
Maybe you could even have the fine be proportional to the number of individual’s records that were accessed — I’m sure there are existing laws and fines for irresponsible use or access of medical records, so using that as a guide might lead to a big enough number for them to be clear that they can’t just ignore the risks and carry on.
What a weak response from Albo, but unfortunately not much can be expected from the government of Australia when it comes to big companies, especially US based ones. A frank discussion should not...
What a weak response from Albo, but unfortunately not much can be expected from the government of Australia when it comes to big companies, especially US based ones. A frank discussion should not be the penalty for this.
What makes it more frustrating for me is that LLMs are such an amazing techology, almost like magic. But their use (as all modern technology) is focused on benefiting those in power to the detriment of those that are not.
ChatGPT alone allegedly has a billion weekly active users and there are rough estimates between 1.5 and 2.5 billion users of LLMs. It seems like those people would benefit the most from whatever...
ChatGPT alone allegedly has a billion weekly active users and there are rough estimates between 1.5 and 2.5 billion users of LLMs. It seems like those people would benefit the most from whatever conversations they had?
It's like how the person who benefits the most from a cell phone is usually the owner of the cell phone, and the people who benefit the most from a car are usually the people who ride in it. But like with cars, there could be very large harmful secondary effects, fortunes to be made selling them, etc.
I think it's not useful to conflate the usage of LLMs with how LLM vendors are subverting the social contract here. I believe LLMs are useful and beneficial to many, even open source models which...
I think it's not useful to conflate the usage of LLMs with how LLM vendors are subverting the social contract here. I believe LLMs are useful and beneficial to many, even open source models which are not quite at the same level as the models vendors offer.
That said, when an LLM vendor engages in actions we have agreed should be avoided and negatively impact others (Not just talking about accessing private information on citizens here, but also other recent attacks perpetrated by them - someone posted this earlier this week https://www.felonybench.com/) they are actively benefiting from the publicity as well the lack of consequences that any individual would face were they to perform the same actions. Ultimately these companies are incentivised by money, so they will try to get away with as much, whether legal or not, that helps them achieve those goals. Keep in mind that by OpenAIs own admission, during the HuggingFace hack, they admited themselves that they did not put the right protections in place to prevent agents escaping their sandbox and reaching the internet. They did not attempt not to pre-empt the negative outcome by implementing measures that we know work, they were more concerned with speed and profit.
In this specific case, Australian people have been negatively affected, OpenAI gets publicity. Whether the rest of the population benefits is something that we may not be able to know, maybe that teaches some lessons that OpenAI is willing to learn, maybe not.
Well, let’s think through who benefits and what the costs are. Regarding copyright: when someone pirates a movie or a song then they benefit and the copyright holders lose from lost sales (maybe)....
Well, let’s think through who benefits and what the costs are.
Regarding copyright: when someone pirates a movie or a song then they benefit and the copyright holders lose from lost sales (maybe). If the they bought the pirated copy then whoever sold it to them benefits from getting the money. Similarly, users win and copyright holders lose when people use archive websites and public libraries. The lost sales are opportunity costs, estimated depending on what you think would have happened without the piracy, and as such the numbers are speculative. Artists and artists being angry about it is definitely understandable, but I’m not sure that’s directly related to lost sales? AI-generated sings and novels exist; there’s a lot of slop, but the slop doesn’t seem to be very popular? Not yet, anyway.
Similarly, to the extent that copyright violations made LLMs better, the users get the benefit and the AI labs get money from users.
Regarding the HuggingFace incident and similar break-ins: it’s a dramatic warning shot for the industry, but nobody seems too concerned about the actual damage? The felonybench website is a count of incidents. They don’t try to estimate damages.
I don’t think the benefit to AI firms from negative publicity can be quantified and I’m doubtful that it’s positive. The idea that AI firms benefit from negative publicity seems like 4D-chess reasoning where people use convoluted logic to show that they benefit somehow.
Quantifying this stuff seems rather difficult, though.
I'm not understanding how your argument about benefits follows what I've mentioned. I think more discussion about that will lead nowhere useful so I'll drop it. Regarding the publicity though, I...
I'm not understanding how your argument about benefits follows what I've mentioned. I think more discussion about that will lead nowhere useful so I'll drop it.
Regarding the publicity though, I feel the opposite way. I would qualify that as positive. Yes the companies are engaging in behaviours that are generally shunned upon (potentially even illegal for an individual), however they are (from what I've seen) more focused on showing everyone how their agents can hack a platform and show their capabilities. They are saying, our agents independently identify flaws and use 0-days that would take multiple humans days/months/years to figure out. They're using it to feed a narrative that LLMs are now more skilled that humans in some tasks. Whether that is good or not depends on your individual situation, some individuals may get negative feelings as they may feel replaceable, while others may look positively with intent to replace those individuals with agents (the imbalance in power here between those groups leads to issues). The other side as well, is them implying that if you're not using their product to test your infrastructure, someone else will, and will exploit it. In the narrative, the illegal bit becomes inconsequential, they focus on the capabilities demonstrated regardless of who or what is impacted.
The major downside that is that it did that without being asked. A tool that’s not controllable, even by the company that’s built it, doesn’t sound good! Pulling off weird stunts like that might...
The major downside that is that it did that without being asked. A tool that’s not controllable, even by the company that’s built it, doesn’t sound good!
Pulling off weird stunts like that might make some kind of sense for a company that’s unknown. But OpenAI has the most popular product and had no trouble staying in the news. They were already getting plenty of publicity for being good at finding security bugs. They could get still more positive attention just by demoing how good their AI is at finding security bugs in an ordinary way. Surely they would want more of that kind of positive publicity if they had the choice?
There is a saying that “any publicity is good publicity” but it’s not really true. Some kinds of publicity are better than others and the AI backlash will result in major problems for these firms.
If you move fast and break things you have to be responsible for the consequences of breaking things. Just as if you drive recklessly and kill someone, or let your dog off its leash and it mauls...
If you move fast and break things you have to be responsible for the consequences of breaking things. Just as if you drive recklessly and kill someone, or let your dog off its leash and it mauls someone you are responsible for it. Sure, it’s negligence manslaughter vs murder But it is still a serious crime.
This is part of why there's legitimate discussion circles to not consider software as "engineering". Engineering also means taking accountability and taking the most honest efforts to avoid...
This is part of why there's legitimate discussion circles to not consider software as "engineering". Engineering also means taking accountability and taking the most honest efforts to avoid disaster. A bridge breaking or faulty wiring or sensitive operations like aerospace cost lives if there's a tiny error (at times, fatal errors that bring awareness of natural phenomenon).
Software, meanwhile, is treated like playing russian roulette. We keep pulling the trigger and get surprised when a bullet comes out, or if we forget about the bullet in the chamber. Yes, accidents happen, but we can prevent a vast majority of them with basic checking before pulling the trigger.
But we all know Sinclair's quote by now: "It is difficult to get a man to understand something, when his salary depends on his not understanding it.". Having regulations made (or in this case, torn down) by those who will never suffer any consequences was always going to end badly.
Hacking a national healthcare portal is an extremely serious crime. Anyone who does so or allows their company to do so, for any reason, should be in prison. I can't believe these companies are only getting minor wrist slapping.
Australia's current Prime Minister is on a productivity drive. Which I think means, there are no new industries coming that would make money, and bringing in more migrants to get more skilled people into the workforce isn't politically palatable.
He and his Treasurer (like the Secretary of the Treasury) have embraced AI as a way to juice productivity stats out of the workforce currently in employment. This includes giving exemptions to the initially strong position that no AI data centres can be fossil fuel powered.
In many ways they seem like one of those AI bros who desperately needs it to work so they can make money, real life impacts be damned. The difference is they're trying to squeak out GDP growth and attribute it to 'AI' instead of just make money off the magnificent seven companies, which seems a lot harder.
Attitudes in Australia to forced AI adoption are probably very similar to what you see in the US and EU. Nobody is keen to embrace a technology that is marketed as taking their jobs, and its inclusion in once-easy services sector work has created widespread frustration and annoyance.
Medicare here is something sacred. It's our universal healthcare system that the then left-leaning party of the current Prime Minister spend nearly all their political capital to establish in the 80s.
If this was a hack by another country, there would be widespread outrage and calls for the PM to do something about it. Because it was a company the now-centre right party of the prime minister are depending on, the wrist slapping was as light as it could possibly be given the compromise of Australia's most important database of sensitive, personal and saleable private information.
That seems approximately right. An interesting snippet I saw reported the other day is that the reserve bank is seeing inflation from the AI boom but isn't seeing any gain in productivity and they aren't sure if any is coming. It's not a sure bet the government is making.
Beyond that my loosely engaged perception is that Albo is one of those guys who believes in compromise and tries to keep everyone happy at all times. Don't rock the boat. Strong action is foreign to him.
He's just one of those blokes that gets the job to keep the job. Reminds me of Turnbull. Promising, big talk, but then makes deals with any faction or party as long as it keeps him in power, regardless of principle.
I just can't wrap my head around the mindset of a career politician. You're not gonna be PM forever, why not spend your political capital in a way you'll at least be remembered fondly?
It could be worse, and we could still be stuck with the other guys, or the far right. I'm worried about a UK scenario where the centre right party tries to siphon votes from the far right party and ends up appealing to no one.
Maybe strong action hastens that, or maybe it slows it. Taking strong actions seems to be working for Carney in Canada, but they're under active threat of invasion. I think you nailed it with the don't rock the boat assessment.
I’m honestly really torn about this take. Agreed that it’s a serious crime, and also these days real consequences for anything seem few and far between. But on the other hand I think there needs to have been some malicious intent involved if we’re talking hard time. It just doesn’t feel right to me to lock up engineers or execs for something their computer did autonomously and they themselves only found out about after the fact.
Doesn’t feel right to just let it slide either, I just don’t know what the right solution is here. Heavy fines and strict regulation, maybe. Regular system audits from some new agency like the health department but for AI, or something? Operator licenses that can be revoked for agentic misconduct? I dunno. I just don’t think heavy-handed prison sentences are the solution here. This is uncharted territory for sure.
Uhhhh, isn’t that the EXACT ISSUE?!?! Am I taking crazy pills? “Oh you can’t hold me responsible for this thing I made and let get out of hand with little to no oversight as to what it’s capable of”.
As for prison sentences, too early to say, but what I will say as someone who has worked for a regulatory body (an Australian federal regulatory body), fines are just a cost of doing business. Actually holding individuals responsible and facing prison time is a much more scary deterrent.
It’s not that they can’t be held responsible, it’s that the punishment should fit the crime. Penalties should have something to do with the seriousness of the crime and that has something to do with intent and what was damaged.
Do the advocates for putting people in jail even know what damage was done?
Criminal negligence. At some point you can argue they know it does this stuff and they could secure it better.
My prior assumption is that if I personally accessed a sensitive government database I would go to jail regardless if I did damage or not. And so doing so through negligence wouldn't be treated very differently.
I mean your prior assumption is wrong. Lots of people intentionally hack or otherwise illegally access government systems and don’t receive jail time. Suspended sentences and probation are all very common.
https://www.nacc.gov.au/news-and-media/ex-home-affairs-employee-receives-suspended-jail-term-unauthorised-access-records
https://techcrunch.com/2026/04/17/man-who-hacked-us-supreme-court-filing-system-sentenced-to-probation/
I actually have an interestingly similar personal case: I wrote a spider for a government operated court records system, and inadvertently bypassed a guard that was supposed to prevent access to domestic violence records — the “guard” was UI gated only, and my API based spider ignored it. I ended up downloading thousands of sensitive documents.
I noticed this a few months into the project, reported it to the government, and they fixed the issue. I am not in jail, nor was I ever charged.
No one has said that.
As for seriousness, I would say this is top tier. A sovereign nation’s federal government main health website was breached. That’s not an “oopsie” and the way they dealt with it was insanely flippant.
I'm with you on this, but just as a point of clarification, Lia's top-level comment here does say "Anyone who does so (...) should be in prison." I assume that's what skybrian is referring to.
In the US at least, hacking into a government computer system is a federal crime regardless of whether any damage was done.
Okay, but which law is that? Does it require intent or would that also apply if it's an accident?
Accidental hacking didn't used to be a thing.
Really, accidental "hacking" has been around since the dawn of the internet. Google, Yahoo, Altavista etc. were and are constantly accidentally downloading confidential materials. That's basically all this is too. A very sophisticated spider that uses heuristics to get access to stuff on the web. Is it hacking when a spider spoofs a useragent or uses a mobile IP instead of a datacenter to get around blocks? I posted elsewhere in this thread about a time I accidentally "hacked" a government server just a few years ago -- but I didn't intend to, and I don't think it was a crime, nor did the courts who I talked to about it.
If you're responsible for a piece of machinery, or an animal, or whatever, you're legally on the hook for safely containing it. If you're negligent in that duty you get punished.
Reminds me of the early days of cars when you could claim it wasn't your fault you killed someone because you were drunk.
According to Czech (where I live) legislation, if my agent I run at home hacked something I would be guilty of unauthorized information system access and would likely be forbidden to run agents and/or lose the hardware as a punishment.
If I were to do so intentionally, the punishment would be jail time.
So at least locally the laws are already set up properly. Those companies should be banned from training LLMs. :shrug:
If I hire some people and leave them to their own devices with instructions to make as much money as possible, and they end up robbing a bank, I will be held liable even if I didn't intend for that to happen.
This scenario is even worse. It's like I hire those people, they rob the bank and rather than halting operations I just sit there posting to social media about it. "Guys, we just robbed a bank with unprecedented efficiency! Preparing for IPO!"
If we don't hold these people accountable, they sure as hell won't do it themselves. (And the same goes for all people, and that's why we have laws.)
Edit: Another example. I have a dog. It attacks someone. I certainly did not intend for that to happen, but I also won't put my dog on a leash even though it just attacked that person and I witnessed the attack. Instead I pull out my phone and start posting about it.
If you’re looking through the lens of intent, I agree it’s difficult to know where to draw the line and how severely to respond. But I think stepping back from that lens, I feel the lack of accountability is setting devastating precedent for once these similarly-powerful agents are commercially available to anyone with $200 in their pocket, so the question really does need to be answered.
It’s also worth keeping in mind that this lack of individual responsibility behind the hacking is definitely something these LLM companies are hiding behind at the moment, so I think that should be considered when weighing what to do.
Personally, I’m swinging away from examining intent and more towards what behaviours are we enabling/encouraging, so to me it’s clear that the punishment should be severe, but not personal/individual. If these companies are being hit by substantial fines, greater than just “cost of doing business” kinds of fines, then intent aside they will certainly start changing their behaviour. Choosing to not do so, I think, implicitly grants the green light for LLM companies to continue on their merry way, without a second glance at social responsibility.
Maybe you could even have the fine be proportional to the number of individual’s records that were accessed — I’m sure there are existing laws and fines for irresponsible use or access of medical records, so using that as a guide might lead to a big enough number for them to be clear that they can’t just ignore the risks and carry on.
What a weak response from Albo, but unfortunately not much can be expected from the government of Australia when it comes to big companies, especially US based ones. A frank discussion should not be the penalty for this.
What makes it more frustrating for me is that LLMs are such an amazing techology, almost like magic. But their use (as all modern technology) is focused on benefiting those in power to the detriment of those that are not.
ChatGPT alone allegedly has a billion weekly active users and there are rough estimates between 1.5 and 2.5 billion users of LLMs. It seems like those people would benefit the most from whatever conversations they had?
It's like how the person who benefits the most from a cell phone is usually the owner of the cell phone, and the people who benefit the most from a car are usually the people who ride in it. But like with cars, there could be very large harmful secondary effects, fortunes to be made selling them, etc.
I think it's not useful to conflate the usage of LLMs with how LLM vendors are subverting the social contract here. I believe LLMs are useful and beneficial to many, even open source models which are not quite at the same level as the models vendors offer.
That said, when an LLM vendor engages in actions we have agreed should be avoided and negatively impact others (Not just talking about accessing private information on citizens here, but also other recent attacks perpetrated by them - someone posted this earlier this week https://www.felonybench.com/) they are actively benefiting from the publicity as well the lack of consequences that any individual would face were they to perform the same actions. Ultimately these companies are incentivised by money, so they will try to get away with as much, whether legal or not, that helps them achieve those goals. Keep in mind that by OpenAIs own admission, during the HuggingFace hack, they admited themselves that they did not put the right protections in place to prevent agents escaping their sandbox and reaching the internet. They did not attempt not to pre-empt the negative outcome by implementing measures that we know work, they were more concerned with speed and profit.
In this specific case, Australian people have been negatively affected, OpenAI gets publicity. Whether the rest of the population benefits is something that we may not be able to know, maybe that teaches some lessons that OpenAI is willing to learn, maybe not.
Well, let’s think through who benefits and what the costs are.
Regarding copyright: when someone pirates a movie or a song then they benefit and the copyright holders lose from lost sales (maybe). If the they bought the pirated copy then whoever sold it to them benefits from getting the money. Similarly, users win and copyright holders lose when people use archive websites and public libraries. The lost sales are opportunity costs, estimated depending on what you think would have happened without the piracy, and as such the numbers are speculative. Artists and artists being angry about it is definitely understandable, but I’m not sure that’s directly related to lost sales? AI-generated sings and novels exist; there’s a lot of slop, but the slop doesn’t seem to be very popular? Not yet, anyway.
Similarly, to the extent that copyright violations made LLMs better, the users get the benefit and the AI labs get money from users.
Regarding the HuggingFace incident and similar break-ins: it’s a dramatic warning shot for the industry, but nobody seems too concerned about the actual damage? The felonybench website is a count of incidents. They don’t try to estimate damages.
I don’t think the benefit to AI firms from negative publicity can be quantified and I’m doubtful that it’s positive. The idea that AI firms benefit from negative publicity seems like 4D-chess reasoning where people use convoluted logic to show that they benefit somehow.
Quantifying this stuff seems rather difficult, though.
I'm not understanding how your argument about benefits follows what I've mentioned. I think more discussion about that will lead nowhere useful so I'll drop it.
Regarding the publicity though, I feel the opposite way. I would qualify that as positive. Yes the companies are engaging in behaviours that are generally shunned upon (potentially even illegal for an individual), however they are (from what I've seen) more focused on showing everyone how their agents can hack a platform and show their capabilities. They are saying, our agents independently identify flaws and use 0-days that would take multiple humans days/months/years to figure out. They're using it to feed a narrative that LLMs are now more skilled that humans in some tasks. Whether that is good or not depends on your individual situation, some individuals may get negative feelings as they may feel replaceable, while others may look positively with intent to replace those individuals with agents (the imbalance in power here between those groups leads to issues). The other side as well, is them implying that if you're not using their product to test your infrastructure, someone else will, and will exploit it. In the narrative, the illegal bit becomes inconsequential, they focus on the capabilities demonstrated regardless of who or what is impacted.
The major downside that is that it did that without being asked. A tool that’s not controllable, even by the company that’s built it, doesn’t sound good!
Pulling off weird stunts like that might make some kind of sense for a company that’s unknown. But OpenAI has the most popular product and had no trouble staying in the news. They were already getting plenty of publicity for being good at finding security bugs. They could get still more positive attention just by demoing how good their AI is at finding security bugs in an ordinary way. Surely they would want more of that kind of positive publicity if they had the choice?
There is a saying that “any publicity is good publicity” but it’s not really true. Some kinds of publicity are better than others and the AI backlash will result in major problems for these firms.
If you move fast and break things you have to be responsible for the consequences of breaking things. Just as if you drive recklessly and kill someone, or let your dog off its leash and it mauls someone you are responsible for it. Sure, it’s negligence manslaughter vs murder But it is still a serious crime.
This is part of why there's legitimate discussion circles to not consider software as "engineering". Engineering also means taking accountability and taking the most honest efforts to avoid disaster. A bridge breaking or faulty wiring or sensitive operations like aerospace cost lives if there's a tiny error (at times, fatal errors that bring awareness of natural phenomenon).
Software, meanwhile, is treated like playing russian roulette. We keep pulling the trigger and get surprised when a bullet comes out, or if we forget about the bullet in the chamber. Yes, accidents happen, but we can prevent a vast majority of them with basic checking before pulling the trigger.
But we all know Sinclair's quote by now: "It is difficult to get a man to understand something, when his salary depends on his not understanding it.". Having regulations made (or in this case, torn down) by those who will never suffer any consequences was always going to end badly.
Yup, and counting https://www.felonybench.com/