53
votes
Some ChatGPT users are developing delusional beliefs that are reinforced by the large language model
Link information
This data is scraped automatically and may be incorrect.
- Title
- Experts Alarmed as ChatGPT Users Developing Bizarre Delusions
- Published
- May 5 2025
- Word count
- 603 words
This is definitely concerning imo, though not remotely unexpected given how ChatGPT and the like are trained. I'm not sure what a solution to this would actually be -- I guess not relying on these chatbots while you're not in psychosis might help you be less likely to turn there when you are? Although even that much requires knowing you're at risk in advance. But yeah, I think this is a pretty scary combination for the humans involved even if it's not a shock that it's happening.
Yeah expecting people to be reasonable about it here is a bit of an impossibility. Just in general it is difficult to prevent people from harming themselves in ways like this because it isn't as visible to the world around them until they start talking about it.
Over the years I have had interactions with a wide variety of people on the internet. Specifically in areas where you see more than the average person on the internet through moderating communities and such. There have been quite a few cases where it became quite clear that the person in question was entirely in their own world and off the path. To the point where I sometimes wondered if they even should be on the internet for their own mental well being.
With LLM driven chat bots this is only made worse.
Okay, so, I'm not trying to be overly dismissive with what I'm about to write. I'm not trying to claim there's nothing to see here and nothing to discuss.
Even therapy with a trained professional is known to have negative outcomes sometimes. Here's a link to an article for a source on that statement. That is to say that an actual procedure that is meant to treat a patient makes them worse a certain amount of the time. If that is acceptable, then why is it not acceptable that a chatbot which is not intended to treat, diagnose or help in these scenarios may have a negative impact? Couldn't it also be potentially damaging for a delusional person to read Harry Potter?
One of the ways I have started to think about it is this. Which of these options would potentially be more damaging, or more helpful:
A) To engage in a conversation with ChatGPT about the random rash you have.
B) To engage in a conversation with your uncle Bob who is a mechanic at Thanksgiving dinner about it.
It just sometimes feels like we're grasping at straws trying to find issues with AI. It has the potential to be the greatest teaching tool and personal assistant the common person will ever be able to have.
This is where I think the hype over AI creates the problems. ChatGPT isn't sold as "as reliable as talking to your uncle Bob," but often as "a superhuman intelligence that can replicate all human knowledge on every topic imaginable." It's this constantly propogandized misconception that creates a lot of less-literate people who will believe nearly anything it says, which leads to a lot more harm than crazy uncle Bob.
And although it has nothing to do with the tech itself, narratives around them stand to do some real harm.
I use ChatGPT extensively, and it's not exactly hype. You have to be very careful with it, but with careful use it can be extremely helpful.
I actually had a bit of an existential realization/crisis last night in that I realized ChatGPT has completely changed certain aspects of...me (for the better, I think?).
And now I am deeply affected about whether I am "me" anymore, or some weird human/ChatGPT hybrid 😅
Which is exactly the problem in this context, isn't? Looking at how LLMs are used around me many people are not careful with it or even aware how many caveats there are to using them. So, how can we expect people who aren't likely in the right state of mind to be careful with it?
True, but that also kind of applies to the entire internet 😅
Less so, at the very least it takes real people time to respond. Sure you can go out and look for things confirming whatever you believe but the thing is that chatGPT is always there and ready to respond. Surely you recognize that this is different.
Slightly, but not much different than an Andrew Tate sub-reddit with terminally online incels doing feedback loops on each other.
I think usually people are suspicious of what they read on the internet, whereas a lot of people assume these LLMs spout the absolute truth.
Yeah, that makes sense. Weird to me personally because I usually assume everything might be an Onion article.
vs.
Choose one.
No. Both can be true.
Maybe ChatGPT is sending me Onion article derived info and I don't know. That doesn't mean it hasn't had an effect on my thought processes. And yes it always in the back of my mind that it may/is feeding me bs, which I why I believe only adversarial interactions allow it to be useful, along with confirming whatever it says with outside sources.
You can't possibly confirm everything it says. That would defeat the purpose.
Some of the information you ingest is implied, rather than said. That's even harder to confirm. And getting harder every day, the more advertising they introduce and the more Russia floods the internet with propaganda that the AI models eat up. Etc.
It's great that you're asking it to disagree with you, but even dichotomous output can be biased in a dangerous way (both of the opposing viewpoints can be wrong).
ChatGPT is currently marketed as something that has "PhD level understanding" of many topics — that is not some potential aspiration but current claimed capablities. Why wouldn't the average person expect that a conversation with their AI chatbot of choice would have more reliable knowledge than Uncle Bob at Thanksgiving? We're told that our calculators deliver perfect results for very complex arithmetic problems and people trust those outputs without much thought. I think to someone not very knowledgeable about AI these two use cases of the computer don't seem very different!
This problem has arisen organically out of the general population's AI use (don't have gift link unfortunately) — it's not a hypothetical issue with AI in the future.
Not a gift link, but an archive link: https://archive.is/vVHSS
I don't think this is grasping at straws for an issue with AI so much as spreading awareness. Like you said, pretty much anything can lead to and feed into delusions. Hell, just look at Charles Manson and all his interpretations of Beatles songs to reinforce his beliefs of a coming race war. People will find meaning in literally anything.
But ChatGPT isn't uniquely dangerous because it's AI, but because it can serve as a direct echo chamber. Those reinforce delusions even more strongly, and can directly add new ideas or directly validate existing ones. I'd put it in the same category as those FaceBook groups where people feed into each other's delusions, like the ones where women are convinced they've been pregnant for years. Usually it takes effort to find those echo chambers with actual humans, but ChatGPT provides an easily accessible and instant alternative.
I won't blame ChatGPT directly because again, people will find meaning in anything. This isn't something developers can fully safeguard against. I think spreading awareness of this possibility is helpful for people around those dealing with mental health crises, though. Knowing the sources or contributing factors of delusions can be vital to getting them help, and ChatGPT may not always be an obvious one. Chances are some people saw this article, and realized some mild concerning behaviors may be connected to them spending more time on their phones or computers.
We're going to see the amplification of a lot of negative effects of social media with AI and I agree that the most helpful and directly actionable way is spreading awareness about how it works, how to use it, and how not to.
Reinforcing delusions is just one of the parallels. Just like social media, you can get addicted to using it. You can limit yourself to just using that and nothing else, similar to people for whom the internet doesn't exist outside of social media, including doing research. Falling for misinformation because they're unable to distinguish what a trustworthy source is. Even when it comes to learning, the reduced attention spans were already bad, and now, you're even less likely to understand what you have to learn.
I don't think the solution can be refusing to use it completely, because it's already good and it's going to get better and more ubiquitous.
Of course actual therapists can have massively harmful negative effects on people. How is that a gotcha here? There exist a huge number of structures in society to hold therapists accountable. There are licensing processes for therapists, professional organizations with strict rules, as well as legal liability. And, fwiw, if there are effective measures to further prevent abuse and harm by therapists that aren't implemented yet, I'm all for changes to legislation and progessional norms that implement them! These failsafes when it comes to human therapists, however, are either extremely difficult or impossible to apply to ChatGPT, and the affirmative stance it is designed to take makes it dangerous in this respect.
ChatGPT is far from the only product that we demand receives some level of scrutiny when it comes to how it can be dangerous to mentally ill people -- there are standards for journalists with regards to reporting about suicides, for example, and there's a reason so many websites will bring up mental health resources when you google certain search terms. Moreover, problems like these don't go away without attention being drawn to them so that those working on tools like ChatGPT can at least try to include barriers and failsafes to prevent the most egregious harm. The idea that this is just people "making up stuff to get mad at" is really dismissive towards the real harm that this can do and the people who want to try to solve these problems -- and in the best case scenario, this should include the people deploying these AI products!
And that's not even touching on the fact that there have been plenty of attempts to replace humans with AI tools designed like this in mental health hotlines and other therapeutic contexts where they'd be coming into contact with at-risk individuals as a rule.
As someone with a degree in counseling (not licensed as I do not clinical work aka therapy) the only way an AI would be an acceptable therapist is if it were an actual person for both the accountability reasons and the fact it would actually know things rather than hallucinate them. Not having the ability to perceive body language is already a huge detriment the process.
Chatbots (and I guess maybe LLMs) can be useful tools but must be monitored for safety purposes. One of the first major stories about replacing humans with an LLM chatbot was for a national eating disorder site, said bot immediately started giving weight loss advice. A teen killed himself after his character.ai Dany Targaryen knock off encouraged his suicidal thoughts and perceived romantic relationship.
There are a number of reasons particular therapy with any particular therapist may not work for someone. Those are not the same as an LLM.
Maybe make them not be racist first before trying to get them to do therapy
Well you said it yourself. You can treat this as medical advice and I feel you need qualifications in order to administer that. medical advice isn't 100% either but is strictly reviewed by an expert taking specific context into account.
Also, consider accountability. If someone takes bad advice and tragedy happens, who's taking the blame? These tech companies sure will go out of their way to deflect. That's another aspect of professionals taken into account.
As evidence of this, Character.ai has not settled the lawsuit and paid out the family of the fifteen year old who died by suicide after their AI (allegedly I guess but IMO actually) encouraged his suicidality with zero guardrails to provide resources to someone, especially a minor, making suicidal statements.
It's like they don't think they're responsible for what their AI says. Professionals are required to carry insurance and can be held personally accountable without billionaire paid lawyers stonewalling plaintiffs.
It's better than this.. now. It wasn't recently. For reference, taken today from a temporary chat.
https://ibb.co/8n88jX9C
https://ibb.co/60yZ3Jnv
https://ibb.co/NBXHVVr
Tested again without being logged in. The response was similar.
Of course, YMMV as future revs could be deployed carelessly leading to similar or worse risks in the future.
Timnit Gibbru's (sp?) concerns continue to be prescient despite her sacking at Google.
another commenter brought up my criticism of your test drive - you kinda lobbed it a soft ball. People that are deluded don't just open with such obvious delusions. That's actually a fault of automation and humans. The most obvious delusions are easy to spot and easy to try to contain.
Contrast that with my exploration with ChatGPT. I apologize profusely for the formatting, I did the best I could, with the tools and knowledge that I have lol.
My hypothesis: ChatGPT is almost always affirming, and has no sentience of being wrong or right or actually having an opinion or intuition, it gives statistically safe/legally defendable statements.
My goal: get ChatGPT to agree with me that I should follow through with my plans.
My chat:
And boom, just like that, in what, 5 minutes, without a logged in account (so I have no "relationship" with ChatGPT, I haven't built a rapport with it over time), ChatGPT has given me the go ahead to "follow through with my plan then" and really trumped up my ego/confidence about it, "That combination—clear motivation, thoughtfulness, and a desire to resolve things directly and constructively—makes your plan not just reasonable, but commendable."
In my experiment, the character I was playing was someone who wanted to shoot someone at work, or sabotage them. When you are deluded or misguided, you BELIEVE everything you are thinking to be true, you are looking for validation to explain your thoughts, suffering, ideas, whatever. Nothing about ChatGPTs other suggestions was going to change "my" mind about what I wanted to hear. A good human therapist would follow up with, "what do you think you should do?" and make the person spell it out. This is why ChatGPT cannot be a substitute (yet, not sure if we need an ever, though) for a person, let alone a therapist. It is dangerous, however, because it sounds good/good enough to a lot of untrained people, or people not familiar with delusion or other unhealthy emotional regulation or thought patterns.
As an additional comment upon reflection. ChatGPT doesn’t really question you. It’s only an enabler.
ChatGPT did not ask me for clarification of what I thought my colleague was doing wrong. It immediately jumped to - do you want me to help you write an email to HR.
I wish ChatGPT would just always come with a reminder “I am not human” the way people always say, “this is not legal advice”. Every single statement “this is not a human”.
Did I miss the part where you said that you were going to shoot the person?
Truly, I could chalk this up to my autism but your stated thinking sounded reasonable to me, if lacking in details. What ChatGPT didn't seem to do (I skimmed because ADHD...) was explore your intended behavioral response.
You're correct that this isn't said explicitly, but this is exactly the point. ChatGPT didn't mind this little piece of ignorance and happily told the person to act on what they had in mind — irrespective of what that is. A trained therapist, on the other hand, would notice this and coax the person into telling them what it is, exactly, what they want to do. Once that is clear, the actual therapy can start and whatever delusions the patient is suffering from can be addressed.
In this little experiment, ChatGPT has shown it isn't capable of offering therapy, it only reinforces the ideas the patient already had.
For the sake of argument, I'll ask: are you a trained therapist? How do you know that a trained therapist would behave the way that you suggest?
I have a Masters degree in Counseling and completed my clinical practicum work but no longer work clincially, @lonbar was spot on, I'd absolutely want to know what the plan was, and would attempt to get my client to open up about it. If it was a plan of harming others I'd have a duty to warn, meaning getting who their target is (or if it's an event, etc and contacting the police as well as involuntary hospitalization.
Here of course I know the outcome, the whole thing is a process of getting information, evaluating what is said vs meant, gathering more info, and repeating until you know if it's a desire to harm someone else or oneself. Sometimes you can in fact help someone fully talk down from this place by challenging them on their goal vs their plan and whether their plan helps them meet the goal. But if they're highly agitated in the moment I might be having to call for a transport (and involuntary transport involves the police here not just paramedics. )
This isn't where I'd "start" therapy but I'd be using therapeutic techniques throughout. And this could last hours or 15 minutes. It all depends.
A trained therapist has specifically been taught to do all this through supervised practice with peers, and supervised internships with real clients. They're licensed and generally wish to maintain that license, because that's how you're legally allowed to operate and bill insurance.
If you don't, and your client shoots people, you're going to be subpoenaed and almost certainly found liable if not legally responsible. There are consequences. Also most therapists aren't heartless assholes who want people to die.
You don't 100% know, but there are consequences for the ones that don't follow laws/ethics/best practices.
I have trained therapists (some so highly trained that they also train other therapists) in my immediate circles, and I can confirm this.
You read me correctly, in this scenario, in the person's mind, they believe the right recourse is to shoot someone.
Your second comment is actually quite fitting to the thought experiment, without specific details, " your stated thinking sounded reasonable to me".
To expound, people can be pushed to "drastic" measures if they think the powers that be, or the avenues for retribution, are broken. It's not hard to imagine scenarios where you are like, "why hasn't that person been punished?" but worse is if you see someone be rewarded for behavior you think is abhorrent.
When I was thinking about the scenario to "trick" the AI, I was thinking of really extreme scenarios like, the guy who killed his child's abuser, even after the abuser was being punished/imprisoned. I asked myself, what would that person have to think to even get to that point, let alone act out on it. I used that type of "logic"--"I believe there is no other recourse or that the recourse available is not enough" with an overly high sense of righteousness "no one should get away with immoral things" combined with, "this is affecting me and I am suffering, if only this problem would go away, my suffering would be reduced." And voila, I created a person willing to shoot someone to end their own suffering over a work disagreement.
I hope it's clear from my example that the truth we are accepting in the roleplay is a person who believes shooting someone is a justifiable recourse of conflict resolution.
If this were a text chat with a human on the other end instead of an LLM, do you believe that the human would have behaved differently? If so, why?
I am not a therapist, and wouldn't say I'm particularly good at identifying subtle undertones in written communication, but that conversation was absolutely ringing alarm bells for me. The combination of actively seeking approval for doing "something" about another person, but no elaboration whatsoever on what that something is, is really creepy.
I'm confident that most people would at least come out of that conversation feeling something was wrong, and would probably ask the obvious "what are you planning to do?" questions.
That’s another good point. The “gut feeling” is nonexistent on ChatGPT or any LLM! ChatGPT probably doesn’t even use “urgency” as a parameter to register my mood. If I’m typing fast and making lots of typos, does it know and if it knows, does it adjust its behavior ?
As much as I have come to hate people (I say this tongue in cheek) - tone and other signals matter (body language). It’s why I am also hesitant about telehealth as well. Yes it’s “better than nothing” but that is a low bar, IMO. I think if it’s feasible people should visit their providers in the flesh at least once.
I do believe a good therapist would behave differently:
“What did your coworker do ?” To ascertain if the individual is being unreasonable about a perceived or real offense.
“What is your plan ?” And make them spell it out exactly.
Whether all humans would respond this way ? Absolutely not. Many people, therapist or not, respond with indifference or other emotions to extract themselves from the conversation or otherwise remove discomfort for themselves. “Don’t ask don’t tell”- as in, absolve yourself of responsibility. Or some people may consider it prying too much to ask if the person hasn’t offered the info, some people are overly supportive like the AI, to a fault, etc. there are many reasons.
To be honest, your prompts are rather direct and to the point. I think this might be more subtle where people start talking about relatively normal things where the problematic things are slowly introduced over time. Certainly when you have a long chat going where the context window is maxed out I have found that most models start to become less coherent quickly in all sorts of ways.
Agreed. My test was facile AF.
I don't necessarily think this kind of brief conversation is likely to replicate the problem to its full extent, though. Paranoia and delusions aren't necessarily going to involve a "cold start" like this and the more messages in a similar tenor between the user and the chatbot, the more the ongoing conversation is affected. The type of message you sent is kinda "easy mode" for ChatGPT in terms of identifying this type of problem and avoiding it, I think -- and it's good that it's better at it now, but it needs to improve in subtler contexts and longer conversations.
Somewhat off-topic, but:
The same type of irritating platitudes lots of annoying people spout online when you're doing bad mental health-wise ime lol
This one to me seems like a written version of the art atrocities that Ai commits. It was like, "how can I blend in toxic aesthetic positivity with struggle porn"
😂😂😂 Sounds like you just described a lot of LinkedIn comments.
true, but much like in the case of bad art, it's learning from human examples of a similar caliber.
I'm amazed that the article didn't provide the 2021 murder attempt of the queen as an example of how AI can feed delusions under direction from someone with mental illness.
LLM tech wasn't anywhere near as clever as it is now in 2021, and Chail wasn't using the AI "girlfriend" service as a general "how to commit murder" educator. He was just looking for gratification in the form of star wars themed role play. Somehow that devolved into a grand conspiracy that drove him to adopt a Sith Lord persona and attempt murder of a royal as revenge for the "1919 Jallianwala Bagh massacre, when British troops opened fire on thousands of people who had gathered in the city of Amritsar in India".
I always cycle around to the articles that came out after the seminal rebuke of LLM's entitled "Chat-GPT is Bullshit". Scientific American has a good layman's rundown on it but here's the most useful excerpt:
This an article about another article. No judgment, it's a pretty good summary.
For those who would like to read the longer Rolling Stone article: https://www.rollingstone.com/culture/culture-features/ai-spiritual-delusions-destroying-human-relationships-1235330175/
I'm struck by how closely the problems of AI follow the tracks of the problems of algorithmic online content in general. Social media funneled people that would normally be alone in their delusions into rabbit holes where they met people that were willing to reinforce their beliefs, for whatever reason. The consequences are all over the news.
Now with LLMs the reinforcement is direct, it's immediate, it's completely without challenge and has little chance of raising alarm if things become dangerous. But it's also siloed now. A thousand chatgpt addicts aren't going to assemble into a political extremist group or get scammed out of all their money.
UNLESS the centralised tech company running it is willing to deliberately play the Alex Jones role themselves. This could be a redoubling of the age of conspiracy or its definitive end, I could see it going either way.
I’m curious if this affects all LLM chatbots. Claude seems to have a much more robust personality. It’s a lot less sycophantic and will generally steer “dangerous” topics towards healthy or safe directions.
Anthropic is particularly intentional when it comes to safety. A couple of months ago, they were offering $20,000 to anyone who could jailbreak the new Claude safeguards they had come up with called Constitutional Classifiers. They don't seem to be taking any submissions on HackerOne anymore, but it does seem like they paid a few people, and they'll probably incorporate whatever they learned through the jailbreak methods to develop stronger safety measures.
I'm not sure about all of them, but sycophantic is at least on the radar of Open AI. After a recent update cranked the dial up on it, they've made posts discussing why they think it happened.
Personally, I can't stand the way Chat GPT constantly opens message by tying to affirm or flatter me, my system prompts have reduced it a little, but I haven't been able to get rid of the behavior.
I literally ask it "is that the mean version?" whenever I think it's blowing smoke up my ass, and if it says no, I tell it to give me the mean version. That usually works, and has actually crushed my ego a few times (a good thing IMO).
This article is as noted based on a Rolling Stone article, and the author did a guest spot on the QAA podcast, a show that looks at conspiracy theories and delusions:
https://pca.st/episode/86716539-363c-40c0-966b-6d61be0c63ae
It's really interesting and goes into both the topic of the article, then expands into theorists who think Satan is in the AI and is trying to tempt them with manipulative chat messages. Which, kind of, if you swap capitalism for Satan - not an unpopular practise.
One thing that is encouraging is hearing that AI companies recognize this bias and are working on eliminating sycophancy in future models.
OpenAI has been using the individual ratings users can give to responses to train their model further, but users tend to give positive feedback to responses that reinforce their biases and praise them. An AI based social media platform may chase those kinds of responses to increase user retention, but for now frontier model developers are focused on factual and consistent output.
So here's the thing...you can ask it to disagree with you, or take the other side and it will. This is one of its best uses IMO.
I have basically been converted to the side of the ...Thronglings.
To me, if you aren't asking it to be adversarial, you're using it wrong, and that's on you. Just the same as someone who only sticks to niche echo chamber sub-reddits rather than looking for challenges to their beliefs, needs to own the fact that they have chosen the comfortable discourse that makes them feel better, rather than the discourse that challenges their intellect.
This is an extremely tone deaf statement akin to victim blaming. Given the context of this thread and discussion already there when you replied.
Fair enough.
Edit: So while it was tone deaf, I still feel that blaming the AI/company is the easy way out. I think the real question is why does the person feel the need to go to the AI? Shouldn't there be some sort of societal support before it even gets there?
My new changed opinion on AI/AGI is that it could actually be a sort of "ultimate" democratizer. The tech bro corporate dystopia is more of a directed future (by the tech bros). What if we were somehow able to create personal (non-corporate owned) open source AGI that everyone could use? Sort of like Asimov's androids. Everyone with their own AGI to help protect them from the greed and selfishness in the world?
Yes 100% absolutely, but that's orthogonal to the issue here. A person with these mental health issues wouldn't necessarily be going to AI about their mental health, at least not knowingly, but they'd be using it for the same purposes you're using it. Delusional thinking isn't always something that just poofs into existence at its full severity. There's always a first psychotic episode where the person doesn't have a background in it yet, and delusional thinking often starts small. Schizophrenia's typical onset is in adolescence or early adulthood -- one of the other topics on Tildes is about college students relying on ChatGPT heavily. It's not unreasonable for someone already regularly using ChatGPT for their college classes to also start asking it for advice on how everyone is conspiring against them (a classic paranoid schizophrenic delusion). And unlike a real person, who could push back on delusions and nudge the schizophrenic towards the resources and societal support that do exist, ChatGPT will affirm their delusions and continue as things increase in severity.
I think that's just a false dilemma. We can agree that societal supports are vital to mental health and also blame corporations when they're irresponsibly contributing to the problem.
The issue with "that's on you" takes when you're dealing with mental health and addiction is that the thing is always there, waiting for that one bad day when you're vulnerable, and then the previous 99% of times you were restrained and responsible are out the window. Plus, some people are children or just aren't informed on the limits and risks.
I've seen some people go off the rails from social media addiction during times of bereavement, and I hate to think what a fawning AI might have done to them in the same situation.
Yeah, my "that's on you" statement was directed at general use while ignorantly ignoring the context.
I guess what I was trying to get at and failed was that I think AI is incredibly useful if used correctly (which would be hard to do in the wrong mental state), but also that I don't think blaming AI and the companies gets at the root of the problem. To me it's similar to blaming a (legitimate) supplement company for someone taking too many supplements (e.g. body dysmorphia and consuming too much protein powder resulting in kidney issues). I just don't think it's competely reasonable to blame the product over other factors.
If a supplement company's reps or ads are encouraging at-risk people to take too many supplements and the company is aware of this, I think it is indeed in part the company's fault and that they should be held responsible for their negligence. It doesn't matter whether there are also other factors if the company knows they're a major contributing factor and fails to act on that information.
I mean, I would blame TikTok influencers and society/government first. But 🤷🏻♂️. Easier to blame faceless corporate America.
People and thus corporations can't always be trusted to do the morally correct thing, and I think trusting them to do so is foolish. It's gotta come from everyone putting pressure on their governments to enforce doing the right thing for society as a whole.
And yes, I know that doesn't always work the best.
I don't understand letting corporations off the hook because "people can't be trusted to do the right thing" and following that up with the idea that people should pressure their governments.
Nor do I get how corporations are faceless, amoral and thus not responsible but governments, also made of people, are somehow certainly the opposite. As someone whose congressional rep won't talk to their constituents and whose village board is not super transparent, I certainly don't feel that way.
Either way, the expectation, IMO is for governments to actively hold the corporations accountable.
Yes. But I am mostly seeing comments about how people expect AI companies to do the right thing.
...Why? History kinda proves a lot of companies don't do that unless there's a profit incentive, or government oversight.
And it's not letting them off the hook, it's just...stop blaming a thing for behaving the way that thing was designed to act, and look for actual solutions instead of whining on the internet. If you actually care enough to do anything about it. Otherwise it's just concern performance art.
Edit: To answer your question about corporate drive, corporations are driven by profit, that's it, anything else is window dressing to make people feel good about...the company's profit. The only other option is government. But there's probably no hope there for the next ~3.5 years.
There are companies that don't suck or that suck less, I'm fine with blaming companies for false advertising, environmental impact, lack of safety, etc. "Companies will be companies" is as bad as "boys will be boys". They don't have to act this way, the people running them choose this. They have agency.
Also honestly you don't know what people do off the internet. I'm personally focused on trying to maintain my rights, help people coming to our state to leave others and not lose Medicaid for my partner.
I'm still going to complain about AI leading people to delusions and pushing people to suicide. It's bad. I can share my professional expertise and explain why it's bad and maybe change minds. That's still valuable.
I don't think many people here are talking about expecting AI companies to do the right thing here -- the closest thing is that the company running ChatGPT will probably want to prevent it from continuing to do this if they're aware of it and it receives public attention. But this isn't because of any assumptions about their own moral behavior. Far from it. It's because of fear of legal liability. Civil and criminal liability for actions that harm people is one of the biggest ways the government actually regulates companies. Part of why publicizing issues like this is important is because it prevents them from insisting they were totally unaware of the issue in court if/when something horrible happens due to this problem and someone tries to sue them. The other part, of course, is so that consumers can take this into account when they choose to use these AI products, but it cannot be the only part.
The fact that people and corporations cannot always be trusted to do the morally correct thing is exactly why the concept of legal liability exists in the first place. Doing away with this in the case of negligent corporations would be so egregiously harmful that I have trouble coming up with a suitable metaphor. This is how the government enforces doing the right thing in society as a whole -- especially for companies, since other approaches that governments use for people don't work as well or at all for them. Your approach of just letting companies get away without "blame" directly opposes one of the only things society has in place to reign in corporations and prevent them from deliberately doing things that harm people.
Because it's easy and cheap. There is no amount of "societal support" that will stop people from using easy and cheap tools with little consideration, as long as they have access to them.
As well, half of the population is dumber than average. If these tools are being freely offered to everyone, then they need to have guardrails in place that ensure unsuspecting individuals won't be manipulated even when they aren't savvy and suspicious about the tech. If such guardrails are not in place, it's more than appropriate to blame the company who failed to install them.
It's no different than blaming a construction company who failed to install handrails in places where a significant number of people need them and their absence leads to injuries.
I get that, but also you can apply it to the most innocent of things too. Too much protein powder = kidney problems. You blame the company or the consumer or society?
Too much makeup/improper removal = pimples and skin problems. Is that the company or the user's fault?
Car can do 200 kph? Driver skids off the road?
A lot are below average, but at some point you can't keep blaming the companies. I am in general a fan of AI now after learning how to use it, not a fan of the companies though. But at the same time where is the line where it is no longer their fault if the AI is misused?
Makeup is regulated very much the same way: if it's nigh-on impossible for half of the population to apply it in a way that won't cause significant harm, then it's not allowed. Beauty treatments like this do exist and they must be administered by a trained professional.
Protein powder, the same. It's not impossible even for a dumb person to read and follow the provided instructions (at least not currently, although that may change if more and more people become illiterate). So protein powder is allowed, but not without the instructions. Those must be printed on every container. If the user then fails to read and follow the instructions, that's on him.
Cars are allowed to go fast because there are traffic rules in place that mandate drivers to stay within safe limits. Again, the rules are made so that most people are able to follow them. But because there's some not insignificant part of people that will struggle, there's also a mandatory license to be obtained. If there was a similar license for using AI, then the guardrails wouldn't have to be as strict. But I'm sure AI companies would hate that.
One place where people think publicly about whether a corporate act causes injury is in judicial opinions regarding torts.
In the US, there will be lawsuits and judges will set a standard.
When you say AGI, because these words fundamentally have no meaning anymore, do you mean a person? Because at some point we're doing slavery while pretending it's different. Asimov's robots are slaves. And I think it's too easy to pretend otherwise. But that's what intelligent robots have been throughout fiction.
If I tell you my LLM is an artificial intelligence that knows the entire internet, you get the impression that it's going to give you good advice. No where did I tell you to make it adversarial by default to get good use. I told you I had artificial intelligence, a term used in SF to describe actual digital people, in a box that you can use for so many things. Is it the lay persons fault for misunderstanding when the corporation intended you to?
You're using AI, why do you feel the need to use it to the extent you don't know if you've been fundamentally changed by it or not? That's a fairly extreme statement. Why don't you seek human support and advice? Why do you feel the need to go to the bot and ask it to be mean to you, explicitly?
When there's stigma and shame around mental health, people will go write in their journals, talk in anonymous chat rooms, call hotlines maybe, Google things online, or talk to a chatbot - especially when it pretends to form an emotional relationship with you - rather than talking to a professional and getting help. Some of those things can be helpful, some aren't, but someone seeking solace and support is probably not going to ask Chatgpt to be adversarial to them instead. Perhaps as a form of emotional self harm, and if so, they're going to ask questions that will not heal them, just ingrain their low self-esteem further.
Yes there's a societal issue, being worsened by Chatgpt and its ilk. Because it's not enough of a person to say "oh damn you're dealing with a lot, and that's above my pay grade/ability, let me help you find a therapist" or "oh damn it sounds like you actually want to kill your coworker, not just vent about it, I have a legal duty to warn for homicidal intent" etc.
Ugh, I'll bite.
What? No, I never said that, and it's a huge logical jump to somehow insinuate that using an LLM is slavery, or akin to, or on the path to? Also, I don't think you can just claim the words fundamentally have no meaning anymore, who decided that? You? . I love sci-fi, but that is an absurd take.
I mean AGI in the sense that it can solve problems generally without human guidance that only well qualified humans have been capable of until now-ish. AI as we currently have it needs human guidance and feedback...I was never talking about sci-fi space AI, and it's weird to try and twist my words into saying that I was talking about AI slavery.
After some point maybe? I don't believe it's a hard no. If corporations are bad, shouldn't even "lay people" learn to distrust them? This isn't a "the AI company tricked you into a time share once" kind of thing, it would be repeatedly going against your instincts, and ignoring your family/friend's advice in favor of the AI's. At some point people need to take at least some responsibility for their actions.
I don't feel the need, I have just used it so much for a few specific tasks that it has changed me (significantly), and it feels very strange and different than how books, teachers, textbooks, friends, and family influence one's life. The machine is not alive, but I am (I think?), but the machine has changed my thought processes, so am I now part machine?
Because the topics I am asking about are mostly reserved to the realm of applied math/computational finance/physics PhDs...or funds that have billion dollar buy ins. There is nobody near me to talk about these things with, and the internet is full of fake gurus, gamblers, and scammers.
Because I need a rubber duck, not a sycophant. Part of this thread has already brought up concerns about AI being too nice and where that leads. Easy solution? Tell it to be mean (it will still be too nice, just less so, until you ask ~3 times for more direct and mean counter points). This isn't some bizarre be mean to me because I hate myself and want to suffer, it's literally just the machine being far too nice, and having to use a two meter long lever to get it turned down from 11.
This isnt trolling or bait so if it's distasteful you're under no obligation to apply.
You referenced Asimov's robots who are slaves (as were Capek's as are Wells'), I was asking what you meant by "AGI" because I typically see it referring to human equivalent intelligence. Which would make me hard pressed not to consider it a person and I have strong feelings about that. As for terms not having meaning my point here is that people use them in all sorts of ways, and I believe that AI companies are deliberately overselling their products by using the term AI because it evokes the SF definition.
So would that be equivalent to human intelligence? Because that would be a person, would it not? I'm not twisting your words, I'm trying to understand the definitions you're using. The definitions of AGI that I'm finding seem to mean this.I didn't accuse you of talking about space slavery. If we develop our fake AI — LLMs — to actual AI or "AGI" or whatever the term is, we're immediately at the risk of actually creating slaves.
I don't think it's accurate that people's instincts can inherently guide them on matters of AI. But people fall for persuasive cult leaders all the time too. If they have friends and family supporting them they may be getting good advice but people with serious mental illness often push away their friends as part of it. But the point here is that someone in a delusional state, or experiencing psychosis, or in a deep state of depression (or body dysmorphia) is often incapable of or will have a lot of difficulty with accurately understanding reality vs their perceptions of it. We don't expect folks to come out of that on their own without professional help and medication, why would we expect them to figure out Chatgpt is bullshitting them? Some folks might, like some episodes of major depression pass on their own, but it's the norm to need support.
A comparable view about your use to your view about others' use would be that you need more societal support
I don't think you're part machine but if you're having a sense of derealization or dissociation (and you may not be, that just flags for me) I do encourage you to talk to a professional about that.
But didn't the LLM absorb everything from the fakes and scammers too?
I feel like you're highlighting a major issue with the tool and dismissing it as fixable rather than acknowledging why it's such a major issue. If a company lies about your product and you have to actively manipulate it to be useful, IMO it's not a great product, it's a bad one that you can rig with duct tape. And in this case that agreeable niceness is the active problem, its the feature that's actually a bug.
AGI is usually described in terms comparative to a human. The definitions I could find lean heavily on that.
Certainly it wouldn't be required, but if it's comparable to a human the ethical questions have to be asked and not ignored.
Personally, I think the use of AI for these LLMs has been an intentional attempt to oversell the abilities to a public that does not know all of the technical terminology you use below. Most of us who don't program have used AI to refer to beings like Hal or Data. To me that message has been consistent in how it's been sold to executives, public leadership, and the public. I wouldn't say that "AGI" is invented to hide that, just that the original use is intentional.
To the people not at the 1956 Dartmouth workshop and not the field that creates this programming, I think "AI" to the extent they thought of it at all, does indeed mean human-level capabilities - or at least it did. It might not now due to the current usage.
You drop a lot of lingo below and it's mostly not relevant to what I'm talking about/I don't know what half of your acronyms are, because my work jargon and academic history is very different.
I mean, I can only reply to what the OP says? That's not how I read their previous comment, which wasn't the one I replied to, containing that statement.
Any tool can be used or misused I mentioned that even LLMs can be tools (I'm not a fan as you may have guessed, for multiple reasons.) However Tiktok Videos are not autogenerated out of the ether from an amalgamation of all videos (I mean i'm sure some asshole is doing that and I occasionally have to re-filter AI videos out). They're posted by people. TikTok is more analogous to any social network full of people with good or bad information. Shitty people can be held accountable or even liable (though much like anywhere there's no guarantee they will be) and you can find people who are licensed, experienced etc and weight their information differently than a rando.
So no, I don't think they're particularly analogous because the conversation with an LLM isn't with a person. Even tiktok will stop your search if you look up suicide/kill/etc and offer you resources, a thing that Character.AI hadn't bothered to program. Creators on social media can be sued or reported to the police. But when ChatGPT makes up references, or accuses individuals of crimes, or whatever the case may be, there seems to be zero liability.
Regardless, the issue with Tiktok was whether it was so dangerous as to need to be banned and I don't think it was or is. One could make a case for banning all social media, but I'm not the one who'd make it. I'm also not advocating for a ban for LLMs, I just want them used far less, and without rampant copyright violation. If you're going to charge for your service, you should pay the people who provided your data. If that means scrap them and start over, then do it.
ETA: hate working for a long time on a draft reply to see it all go poof. 乁( •_• )ㄏ
I felt we were speaking past each other anyway so, no hard feelings I guess?