This is a very, very insightful article - he's dodged the majority of the pitfalls most writing about AI* makes, whether pro- or anti-, because he's focused on the human element and he clearly...
Exemplary
This is a very, very insightful article - he's dodged the majority of the pitfalls most writing about AI* makes, whether pro- or anti-, because he's focused on the human element and he clearly gets it. The thing everyone is angry about has very little to do with the technology itself, and a whole lot to do with the shell game of ever increasing stock prices that people somehow still pretend are correlated to real world benefits.
It's a relief to see someone else pointing out that trying to fight AI companies with expanded copyright laws would just help different megacorporations without doing anything meaningful to benefit individual creators, because the major copyright holders are just as bad as the tech companies, and beholden to just the same absurd economic incentives that we are apparently treating as the inviolable natural order of things rather than as a managing framework that we've let get wildly out of hand.
I'm a bit disappointed to see him repeating the same flawed justification about AI output not being copyrightable (tl;dr: the model isn't the monkey, as he implies. It's the camera.), but I do actually like his play through of the implications of making that ruling. I don't think it's a logically consistent ruling to make with the law and precedent as it stands, but I can at least buy his arguments about how it could theoretically play out in a beneficial way.
Most of all, I appreciate that he isn't treating "jobs" like they're an inherently good or inherently valuable thing. They aren't. But quality of work is a good and valuable thing, and again the economic incentive (and I'll stress for the third time, we are somehow accepting this insane incentive structure as "just the way it has to be") is to make cheap, substandard output and then lie about it being better than it is. We have these incredibly powerful and capable tools being created by scientists and engineers, with the possibility of genuinely using them to improve the world, and the primary thing getting in the way of that right now is the fact that making grandiose claims about the tools can make you very, very rich.
You should really read the article, because it says more and better than I can here, even accounting for the odd bit I don't agree with - but if you don't have the time, the conclusion hits the key points pretty succinctly:
AI is a bubble and it will burst. Most of the companies will fail. Most of the datacenters will be shuttered or sold for parts. So what will be left behind?
We will have a bunch of coders who are really good at applied statistics. We will have a lot of cheap GPUs, which will be good news for, say, effects artists and climate scientists, who will be able to buy that critical hardware at pennies on the dollar. And we will have the open-source models that run on commodity hardware, AI tools that can do a lot of useful stuff, like transcribing audio and video; describing images; summarizing documents; and automating a lot of labor-intensive graphic editing – such as removing backgrounds or airbrushing passersby out of photos. These will run on our laptops and phones, and open-source hackers will find ways to push them to do things their makers never dreamed of.
If there had never been an AI bubble, if all this stuff arose merely because computer scientists and product managers noodled around for a few years coming up with cool new apps, most people would have been pleasantly surprised with these interesting new things their computers could do. We would call them “plugins”.
It’s the bubble that sucks, not these applications. The bubble doesn’t want cheap useful things. It wants expensive, “disruptive” things: big foundation models that lose billions of dollars every year.
* I've said this a million times now but I'll say it again: AI is a bad term for this tech in general, and one that leads to a whole mess of confusion above and beyond that fundamental incorrectness because no two people actually use it to mean the same thing anyway. But it's the accepted way people talk about big neural net type things nowadays, so I'm going to use it as the article intends, for what I'd guess is the same reason that he didn't start with 500 words of dry explanation on why we should be saying "machine learning" and precisely what that covers.
They are necessary. That inherently means they are valuable. Because the good alternatives aren't setup and the realistic alternatives are homelessness and all that entails. We were regressing on...
They are necessary. That inherently means they are valuable. Because the good alternatives aren't setup and the realistic alternatives are homelessness and all that entails.
We were regressing on the idea of remote work a few years ago, and now these techno feudalists want to imagine a UBI world? I say "jobs are valuable" as a compromise, not as an idea that we need to live to work.
The difficulty here is that they're necessary in the sense of "our current economic system won't give you food and shelter if you don't have a job", but often not necessary in the sense of "the...
The difficulty here is that they're necessary in the sense of "our current economic system won't give you food and shelter if you don't have a job", but often not necessary in the sense of "the work you're doing is needed for a productive, thriving world".
I think I get where you're coming from, but the actual impact of phrasing it as "jobs are valuable" is that 99% of your audience will just hear it as support for the already ingrained idea that "having a job" is worthy, noble, or productive in and of itself. If we're going to have even the slightest hope of building support for evidence-based work reform, the first barrier to cross is breaking the idea that "40 hour per week job == good, contributing, beneficial" and "any other arrangement == bad, lazy, burdensome".
I think "jobs are not inherently valuable" is at least a clear opening statement for that conversation - as is "jobs are currently a necessary evil", if you prefer an alternative way of framing the same thing. Suggesting that their necessity gives them value, but then trying to explain to people that there's a giant asterisk there to clarify that the necessity is artificial, and "valuable" doesn't mean what the majority of them are interpreting it to mean, just seems like a very difficult way to communicate the point.
This is already happening in younger generations though, and was only accelerated when boomers pulled up the ladder behind them after achieving the "American Dream". Housing affordability is...
If we're going to have even the slightest hope of building support for evidence-based work reform, the first barrier to cross is breaking the idea that "40 hour per week job == good, contributing, beneficial" and "any other arrangement == bad, lazy, burdensome".
This is already happening in younger generations though, and was only accelerated when boomers pulled up the ladder behind them after achieving the "American Dream". Housing affordability is fucked and has remained fucked for decades, and this has directly influenced how people think about having a job and why exactly they should work so much if they can't build anything of their own by investing their limited lifetime into some corporation.
Of course, this also lead to other worrying trends, especially a rise in consumerism and collecting things founded on social media fads becoming faster and faster and faster. Be it Stanley cups or Legumi pens; a lot of people no longer have long-term financial goals, which directly leads to them spending all the available cash they have on "useless stuff".
Oh god I wish it weren't so rare to see this (utterly correct) take
It's a relief to see someone else pointing out that trying to fight AI companies with expanded copyright laws would just help different megacorporations without doing anything meaningful to benefit individual creators,
Oh god I wish it weren't so rare to see this (utterly correct) take
Hell, I'd imagine if we gave copyright more teeth, not only would media corps get massively more powerful. But we'd not see any functional impact on AI. They'll obfuscate the derivativeness of any...
Hell, I'd imagine if we gave copyright more teeth, not only would media corps get massively more powerful. But we'd not see any functional impact on AI. They'll obfuscate the derivativeness of any derivative outputs slightly better, and that'll be that. I don't think openAI will actually throw out all their ill-gotten training data or delete their models.
And if we actually force them to by means of massive and unthinkable enforcement action, i.e. raiding their data centers and wiping all their hard drives, we'd just give the game over to the Chinese or any other nation willing enough to take the lead in the AI game.
My approach would be to go the opposite direction: restrict copyright protection. Your model parameters? Definitely not copyrighted. Model outputs? Yeah, nah. The datasets you're using to train? Fine to use as long as it's just for training. (Personally I'd go complete abolitionist, but that's an admittedly fringe take. ) That levels the playing field: 3 (british) guys in a shed have the same access to data as OpenAI, and we'll make sure that this technology isn't oligopolized by roughly 4 companies. Yes, compute is still a limiting factor, but it's much easier to distribute compute fairly by virtue of free markets. Ask Amazon or one of a bajillion other companies if they'll rent you the compute to train a model - this part of the equation isn't what keeps the big few in power.
I don't like that argument because yes, 99% of all initiatives disproportionately support people rich enough to argue for it (often on someone else's dime anyway). If it's not progressive taxing...
I don't like that argument because yes, 99% of all initiatives disproportionately support people rich enough to argue for it (often on someone else's dime anyway). If it's not progressive taxing or welfare*, its probably something a rich person gets more out of
It feels like a form of soft defeatism to say "well the rich benefit more so we may as well not bother". We can work on that 1% while acknowledging that the 99% will still help regular people out some too.
*and even here, welfare arguably is a cost saving measure for the rich. Not having to hire security to navigate a low crime area helps immensely. The general stimulaton of economy for people who'd otherwise drops out helps everyone as well
I think you misunderstand the argument if you think the conclusion is "the rich benefit so might as well not bother", at least when it comes to IP law. My perspective is more "we should weaken IP...
I think you misunderstand the argument if you think the conclusion is "the rich benefit so might as well not bother", at least when it comes to IP law. My perspective is more "we should weaken IP protections because they don't protect small independent creatives, as they only serve those already have money and power, and strong IP laws actually harm creatives without that power and stifle creative freedom." The pithier (but less nuanced) slogan would be "abolish copyright."
When it comes to generative AI, my point is not "stricter copyright laws benefit the rich so might as well not bother", it's "stricter copyright laws harm the very people you're trying to protect here, and them also harming generative AI tech companies doesn't actually change that."
I think my reservation to begin with is that "they don't protect small independent creatives" is a misnomer in and of itself. This is the cybersecurity argument of "well we don't have many...
My perspective is more "we should weaken IP protections because they don't protect small independent creatives, as they only serve those already have money and power
I think my reservation to begin with is that "they don't protect small independent creatives" is a misnomer in and of itself. This is the cybersecurity argument of "well we don't have many viruses, we can cut own on costs".
A lot of IP's are protected indirectly because a large company rigorously checks for any work and its likeness as a risk factor. So IP law prevents a lot of lawsuits before they even start. That's valuable and not something to dismiss. All current lawsuits over IP tend to be edge cases and abuses. Things that would still happen no matter how weak or strong we make IP laws.
Using those edge cases as a vector for your argument feels misguided, as a result. Because those will never go away. Meanwhile, you "abolish copyright" and Disney is going to make millions selling Silksong plushies that will never go back to Team Cherry. They'll make way more than Team Cherry would make by putting Mickey into the next DLC. So the power imbalance is still there.
Big companies can scale up profit windows in places faster than you can.
When it comes to generative AI, my point is not "stricter copyright laws benefit the rich so might as well not bother", it's "stricter copyright laws harm the very people you're trying to protect here, and them also harming generative AI tech companies doesn't actually change that."
Sure, and I disagree. I'll use another metaphor; let's say you are a garden hose on low and Disney is a fire hose.
Right now there's a wide open waterslide to fill, so the fire hose wins. But it's a water slide at a park with dingy nails , zero guard rails, overpopulated pools, and a bunch of theft. These all suck, but Disney still makes money. because their money comes from putting water on the slide, not fixing any safety concerns.
Regulations make them need to care for the other stuff. Maybe Disney decides it's worth turning down the fire hose and it's still profitable, maybe they find another service who wants a firehose (and doesn't need pesky regulations). But either way, it means your garden hose has places to fit in, and you don't need Disney profits to succeed.
As it was before, you were never going to compete with a firehose, and right now everyone is looking for firehoses, at the expense of the customers who just want a fun, safe water slide. You may also not like having to care about safety, but in that case maybe it's also not viable for you. And that's fine. Because the customer's safety isn't just an option to weigh, it should be important if you want to run a water slide.
I think the piece of the puzzle you're missing is that Disney can already pretty much violate smaller creators' copyright with impunity because they're so much larger and richer than them. They...
I think the piece of the puzzle you're missing is that Disney can already pretty much violate smaller creators' copyright with impunity because they're so much larger and richer than them. They don't do so as blatantly as selling Silksong plushies because that would be bad publicity and the legal trouble would not be worth what they'd make from it, but I think it's extremely naive to think IP law could actually prevent Disney from doing irreparable damage to anyone who isn't nearly as big as them. And that's ignoring the fact that Disney can also afford to have a vice grip on their own IP, using cease and desists and their ample funds for lawsuits to threaten anyone who uses their IP, even transformatively in obvious fair use cases, sometimes even based on a passing resemblance rather than any actual infringement, into shutting down. And this is something they and other large IP holder do constantly do. IP law "preventing lawsuits before they even start" is a huge part of how large IP holders wield copyright in ways that stifle independent creativity, because whether they're right doesn't matter. They have the power to destroy you, smaller creative, if you don't comply.
And, of course, "abolish copyright" is not the only option when it comes to weakening current IP laws. There are plenty of positions to hold between "current IP law is fine" and a total free-for-all that would be undoubtedly an improvement on the status quo for creativity. Current copyright laws don't actually do much to protect small creatives, and there is zero evidence that stricter copyright laws, especially those of the kind people advocate for to stick it to AI companies, which often involve allowing copyright on ideas or common elements of style, would do anything but stifle creative freedom for those not rich enough to be above the law.
I don't really understand your fire hose metaphor, because it's unclear to me what all these dangers you describe are supposed to represent in it. I don't think it makes sense to claim that stricter copyright laws force Disney to care about... honestly, almost anything. Disney wants stricter copyright laws because it makes them even more powerful and gives them even more control than they already have. It doesn't actually do much if anything to force them to change their behavior. Stricter copyright laws certainly don't make Disney "turn down the firehose". They give Disney more power to arbitrarily turn off other people's hoses.
Cameras have regulations too. So I think either way it applies. I can't use a camera to invade someone's space, and picturing copyrighted work still had loads of gray areas.
I'm a bit disappointed to see him repeating the same flawed justification about AI output not being copyrightable (tl;dr: the model isn't the monkey, as he implies. It's the camera.)
Cameras have regulations too. So I think either way it applies. I can't use a camera to invade someone's space, and picturing copyrighted work still had loads of gray areas.
I'm not conceptually against regulating the tech, assuming it's done in a sensible way by people who understand the nuances of both the technology and the law (a big ask, I know!). I just don't...
I'm not conceptually against regulating the tech, assuming it's done in a sensible way by people who understand the nuances of both the technology and the law (a big ask, I know!). I just don't like the faulty premise coming in as a foundation of the argument, because it makes it a lot harder to get to a logically consistent endpoint - even if it's coincidentally being used in support of a reasonable point on this occasion.
Same way I wouldn't want a road safety policy to be based on the idea that red cars are faster, even if it happened to end up getting to a policy I agree with, you know? If we let that kind of thing pass, it just sets up for deeper and more damaging misunderstandings later.
At the current point we're at, it's people trying to argue that speed limits should be repealed because "slower cars shouldn't be on the road". They aren't even trying to pretend they want to be...
I wouldn't want a road safety policy to be based on the idea that red cars are faster
At the current point we're at, it's people trying to argue that speed limits should be repealed because "slower cars shouldn't be on the road". They aren't even trying to pretend they want to be safe.
I wish one day we reach the equivalent of your argument. At least by then people will admit that safety is something to strive for.
Good article (i actually read all of it). I particularly like the concept of reverse centaurs that he postulated; and the example of Amazon drivers being reverse centaurs, basically enslaved by...
Good article (i actually read all of it). I particularly like the concept of reverse centaurs that he postulated; and the example of Amazon drivers being reverse centaurs, basically enslaved by technology and the billionaires who own the technology. And it makes perfect sense that they want to fire skilled workers and retain just a few 'humans in the loop' whose only real purpose will be 'accountability sinks' i.e someone to blame for AI's mistakes.
I am now fearful of how my own job will get affected by the AI bubble burst. Or worse, how my daughter's generation will get affected; she's just a toddler, and i hope the world is in a much better place by the time she is ready to enter the workforce.
I respect Cory Doctorow for remaining consistent on his copyright opinions, and I mostly enjoyed this article. If you have the utmost belief that AI will fundamentally plateau somewhere below...
I respect Cory Doctorow for remaining consistent on his copyright opinions, and I mostly enjoyed this article. If you have the utmost belief that AI will fundamentally plateau somewhere below humans, then the article seems correct. But I have to pull one quote from the article:
If there had never been an AI bubble, if all this stuff arose merely because computer scientists and product managers noodled around for a few years coming up with cool new apps, most people would have been pleasantly surprised with these interesting new things their computers could do. We would call them “plugins”.
There was never a world in which this didn't become an enormous, international phenomenon. Ever since Turing proposed the Turing test, it's been inevitable. The fact that you can now have a full conversation with a computer is strange and incredible, it would never have been relegated to something exclusively for techies.
Also, this sentence is not entirely true:
Now, AI is a statistical inference engine. All it can do is predict what word will come next based on all the words that have been typed in the past.
This is a common misconception. Modern LLMs are post-trained on reinforcement learning tasks, which means they're not only trained to predict the next word, but also trained to solve problems more effectively. And, in the case of the example used in the article, they are trained to recognize when they don't know something, in order to not hallucinate. Although, just like humans will often misremember things, this isn't perfect (but it has been getting noticeably better with new models).
Calling them "next token predictors" is overly simplified and arguably incorrect with transformers and attention, but the models are fundamentally massive pattern matchers. Every technique and...
This is a common misconception. Modern LLMs are post-trained on reinforcement learning tasks, which means they're not only trained to predict the next word, but also trained to solve problems more effectively.
Calling them "next token predictors" is overly simplified and arguably incorrect with transformers and attention, but the models are fundamentally massive pattern matchers. Every technique and application I've read about rely on that pattern matching capability. Machine learning is all about learning and using patterns.
Reinforcement learning and various response layers do a lot to fine-tune LLMs. They still rely on introducing, removing, identifying, encouraging, and discouraging specific patterns.
Cheers to both your post and the one you're replying to. It's a forever losing battle trying to clear up misconceptions about tech, but worth attempting nonetheless. Whatever level of...
Cheers to both your post and the one you're replying to. It's a forever losing battle trying to clear up misconceptions about tech, but worth attempting nonetheless. Whatever level of understanding popular culture lands on ends up influencing policy and legislation.
Reinforcement learning and various response layers do a lot to fine-tune LLMs. They still rely on introducing, removing, identifying, encouraging, and discouraging specific patterns.
The reason I think this is important to keep pointing out is that LLMs feel like something more than advanced pattern matching tools, and I think that's going to be dangerous.
There was never a world in which this didn't become an enormous, international phenomenon. Ever since Turing proposed the Turing test, it's been inevitable. The fact that you can now have a full conversation with a computer is strange and incredible, it would never have been relegated to something exclusively for techies.
This is just true. I understand the pushback against AI, I agree with a lot of it, but it extends into the irrational at times. Not only was it always going to shake the ground, it was always going to attract trillions of dollars in investment, and that was always going to come from the megarich. Wealth consolidation was a problem long before LLMs, this is just the latest symptom. Wishful thinking is not how we'll have a shot at changing it.
A world where groundbreaking technology benefits the masses as much as it benefits the financial elite is a world with very different systems than we have now, AI or not.
I think overall Doctorow's piece was solid, he made some great points. I'm happy to give some leeway to the guy who coined enshittification. But there was one additional questionable claim:
AI is a bubble and it will burst.
I agree that AI is a bubble. I don't know if anyone can confidently predict that it will burst. Western economics is in uncharted territory, brute force capital has held off, lessoned, shortened and in some cases entirely avoided recessions, market crashes and other financial events that statistics and history tell us should have happened, or happened bigger, sooner and lasted for longer.
The AI bubble should burst dramatically. The western world should probably be in a natural and perfectly healthy recession right now. Certainly the non wealthy are experiencing something like a recession even if the financial markets don't reflect it. Not a healthy one though.
All the rules are different now. The AI bubble could pop, but the capital firehose has to run dry first, and governments, especially the US, have a stake too. Without AI the stock markets look bleak, the US looks less like a world leader and the dollar would probably be in free fall.
Which isn't to say that the bubble definitely won't pop, only that it's not a foregone conclusion. Which Doctorow might have been more willing to allow if he wasn't promoting a new book.
There's evidence that the bubble pop is being forestalled in part by massive Federal Reserve intervention in the U.S. Banks are lending money faster than they're taking it in, and there's major...
There's evidence that the bubble pop is being forestalled in part by massive Federal Reserve intervention in the U.S. Banks are lending money faster than they're taking it in, and there's major instability coming from multiple sources.
This doesn't take into account how vastly overvalued AI investments are in private capital markets.
Doctorow's essay should be read as a companion piece to today's Guardian interview with Ed Zitron, who's done a very cogent analysis of the circularity of lending among the hyperscalers and Nvidia, and the impossibility of the current gen AI business models.
It's not a question of "if", just "when".
Further footnote: I personally believe that the most transformative technological leaps won't come from general purpose LLMs (which really aren't doing anything humans can't do, and mostly do worse and less cost-effectively than humans), but from specialized models for narrowly specified tasks in the physical world and modeling complex systems, e.g. protein folding and receptor interactions, weather, markets, and so on. These are areas where human cognition isn't optimal - more "centaur" than job replacement.
A tricky sales pitch, though. Right now, most LLM shills are acting like LLMs can replace human labor. But who will be willing to pay the bills for LLMs that merely supplement human labor? Instead...
A tricky sales pitch, though. Right now, most LLM shills are acting like LLMs can replace human labor. But who will be willing to pay the bills for LLMs that merely supplement human labor? Instead of cutting salary costs, you'd actually add subscription costs on top of your existing salaries. There's grey area here for doing more work with a similar number of employees, but the calculation is a lot more complicated than "one CEO running a company of bots" like the current industry pitch.
That is the exact reasoning for why we're calling it a bubble. It does not mean the technology is inherently useless, just that there are a lot of pie-in-the-sky expectations for it that cannot...
That is the exact reasoning for why we're calling it a bubble. It does not mean the technology is inherently useless, just that there are a lot of pie-in-the-sky expectations for it that cannot reasonably be met and that do not make financial sense for the amount of money being sunk into it.
Yeah, this is my misconception I try to rectify every other article. The dotcom bubble did not kill the internet. But it did push out a few major players and a lot of smaller, massively overvalued...
Yeah, this is my misconception I try to rectify every other article. The dotcom bubble did not kill the internet. But it did push out a few major players and a lot of smaller, massively overvalued ones. The remnants of that still went on to become the new face of tech. Or a few old faces remained (Microsoft)
AI will probably have a similar fate.the big massive loser will likely be OpenAI unless Microsoft absorbs them in fully. Tons of others will die out, and one or two big players (Tesla is the obvious bet. Maybe Meta too) will fall from grace.
The only big difference is that the trillionaire tech companies probably won't be wiped out en masse, nor even stagnate away like the IBMs of yesterday did. They still have core products people use and will fall back to that. Maybe Antitrust finally catches up to them, but that's a different matter entirely.
Imagine business models for specialized pharma ML of operating cost plus fractional IP rights in any discoveries; or insurance industry and commodities market licensing on better weather models......
Imagine business models for specialized pharma ML of operating cost plus fractional IP rights in any discoveries; or insurance industry and commodities market licensing on better weather models... I'm sure the busy imaginations of startup founders can find a way to make them profitable, but we're not talking about "One TRILLION dollars!".
If your broader social purposes are massive productivity improvement and less waste, better environmental stewardship, just distribution and abundance for the masses, improved health, more responsive and accountable government, I can see many ways that effectively applied ML would help, and even grant modest profits if we can't come up with a better model than capitalism.
There are certainly lots of attempts to do this, but anyone who is remotely knowledgeable about language models knows that it is not possible to completely prevent "hallucination" with this kind...
And, in the case of the example used in the article, they are trained to recognize when they don't know something, in order to not hallucinate.
There are certainly lots of attempts to do this, but anyone who is remotely knowledgeable about language models knows that it is not possible to completely prevent "hallucination" with this kind of architecture atm and the best you can do is mitigate it. Those who claim otherwise are overstating the results of methods to do the latter at best and straight-up grifting you at worst. (It also has little to nothing in common with how humans "misremember things". But you're already using anthropomorphization like "realize" and "know" here, and I get what you're getting at.)
I do agree, though, that this blowing up was inevitable. I don't think people who weren't working with language models or ML before chatGPT came out realize just how big of a leap forward these LLMs are compared to even the state-of-the-art prior to them. People who are against AI for the human reasons described in the article do sometime try to downplay how good these models are as part of it, but they really are incredibly impressive and I wouldn't have believed such a huge step forward in language models would happen so quickly if you'd asked me at the beginning of 2022.
A big part of it is setting completely unrealistic standards. From a single, often poorly formulated and vague prompt, this tech must understand you completely accurately, and fetch a complete and...
People who are against AI for the human reasons described in the article do sometime try to downplay how good these models are as part of it, but they really are incredibly impressive and I wouldn't have believed such a huge step forward in language models would happen so quickly if you'd asked me at the beginning of 2022.
A big part of it is setting completely unrealistic standards. From a single, often poorly formulated and vague prompt, this tech must understand you completely accurately, and fetch a complete and fact checked answer with no errors, on any arbitrary topic and all in under 30 seconds!
An army of human experts wouldn't be held to such a standard, let alone any single individual human! It is absolutely incredible what these tools are capable of now.
I assume you're anthropomorphizing like this from the perspective of a layperson using it, and honestly I think that's a big part of it -- it got so much better so quickly and the user interface...
From a single, often poorly formulated and vague prompt, this tech must understand you completely accurately, and fetch a complete and fact checked answer with no errors, on any arbitrary topic and all in under 30 seconds!
I assume you're anthropomorphizing like this from the perspective of a layperson using it, and honestly I think that's a big part of it -- it got so much better so quickly and the user interface got much more intuitive, so people are holding it to the standards you'd hold a human to because the flow of conversation with something like chatGPT feels so much like having a chat with a person would. Heck, it's almost always using more fluid English and providing a more frictionless experience than a chat with human customer service generally is. Honestly I think the UX is doing so much for how quickly it got so widely adopted, on top of the actual technological advancements.
Is it actually a conversation, though? Or just a facsimile where I contribute an input and the system produces a response that fairly convincingly emulates some sort of relatedness to my input?
The fact that you can now have a full conversation with a computer is strange and incredible
Is it actually a conversation, though? Or just a facsimile where I contribute an input and the system produces a response that fairly convincingly emulates some sort of relatedness to my input?
I'm usually opposed to language that anthropomorphizes LLMs, but I think this particular one is a bad sticking point to have tbh. The only definition of "conversation" that would exclude what...
I'm usually opposed to language that anthropomorphizes LLMs, but I think this particular one is a bad sticking point to have tbh. The only definition of "conversation" that would exclude what people do with tools like chatGPT would be so heavily philosophical that it would exclude huge swaths of conversations between humans. It's a language model and it's very good at producing comprehensible, relevant responses to things you say, and that allows you as a human to hold a conversation with it (and how good these models are at doing this was a genuine breakthrough). You, as a human, are doing all the same stuff you'd do in a conversation with another human, and whether the model is underlying doing those same things (it's not) isn't relevant imo. Humans can hold conversations with much less sophisticated "communicators" like parrots or magic 8-balls. There's little point in arguing the semantics of the word "conversation" rather than focusing on the actual weak points and problems with LLMs.
I do agree about this, but not necessarily the broader point that we shouldn't aim to tell real conversations apart from auto-piloting (machine assisted/generated or not). And I believe this to be...
it would exclude huge swaths of conversations between humans
I do agree about this, but not necessarily the broader point that we shouldn't aim to tell real conversations apart from auto-piloting (machine assisted/generated or not). And I believe this to be one of the benefits of ubiquitous LLMs: more people will learn to tell the difference and why it matters.
I'm not particularly convinced the difference exists in a way that matters meaningfully. I think a conversation with someone else who studied pragmatics and speech act theory (if someone else like...
I'm not particularly convinced the difference exists in a way that matters meaningfully. I think a conversation with someone else who studied pragmatics and speech act theory (if someone else like that is on Tildes please DM me I want to be friends 🥺) would be enlightening and I'd be interested to hear those kinds of perspectives. But outside of those very theoretical/philosophical discussions, I'm not particularly convinced the difference matters in a practical way. Whether you classify it as a conversation or not doesn't inherently change how you should engage with/respond to it.
One situation where it matters is when gauging the amount of effort we should be putting in. If I'm having a conversation about something important with a life partner, whom I know well enough to...
I'm not particularly convinced the difference matters in a practical way.
One situation where it matters is when gauging the amount of effort we should be putting in.
If I'm having a conversation about something important with a life partner, whom I know well enough to tell they are actually processing my input and giving it a chance to affect their world view, values, habits... (= whatever tangible wiring in their neural network) - and that there is similarly a chance for me to be affected by someone whom I deem admirable and trustworthy - then I'm willing to give the conversation my best effort and a long time. Several days, if need be. If it's someone similarly engaged but for whom I care about less, like a friend, I'm still willing to put in a lot of effort but not as much as I will for a partner.
On the other end of that continuum, if it's someone who clearly isn't listening attentively enough to be able to actually process anything, I'll be done with the convo as soon as I notice.
We know that some people will happily engage with an LLM for days on end, but believing this to be fundamentally no different from engaging with a mindful human with their own personality and thoughts is.. dangerous. And we're starting to see what such engagement does to these people's minds, which should also indicate it isn't something to overlook.
As well, being able to tell whether you're the only person/entity putting yourself and your personality on the line (while the other side is simply faking their way through in an attempt to drive further engagement from you) will protect you against getting scammed. Or against getting into dysfunctional relationships. And so on.
It's an incredibly important life skill, one that can even lead to societal collapse if eroded too far.
I agree that this is the case. I do not believe that nitpicking about the semantics of the term "conversation" remotely helps address the problems with doing this, and it in fact obfuscates the...
believing this to be fundamentally no different from engaging with a mindful human with their own personality and thoughts is.. dangerous
I agree that this is the case. I do not believe that nitpicking about the semantics of the term "conversation" remotely helps address the problems with doing this, and it in fact obfuscates the actual risks involved in favor of something that simply does not matter to the outcome. And, frankly, I don't agree that whether or not one's interlocutor is actually participating in the conversation to the same degree and in the same way as you entails that you need to put in less effort. That is sometimes the case, but it isn't even always the case with human interlocutors, much less in this circumstance. I would argue that you frequently should put in more effort when it comes to handling the output of a genAI model because it's not the same as engaging with a mindful human with their own thoughts. Concretely being aware of the risks and pitfalls with different ways of applying genAI is far more useful and important than insisting that it doesn't meet some philosophical threshold for the definition of the word "conversation."
I also just don't think that your comparison to a human who simply isn't listening or engaging with your conversation is apt. The ways in which they differ from a conversation with a mindful human are different, and the risks and pitfalls are thus also different.
Also, none of this will protect you from being scammed. Scammers often do listen pretty intently and almost always thoroughly participate in conversation because they're working to manipulate you into doing what they want. I see even less utility in insisting on a definition of "conversation" that excludes these somehow when it comes to scams and can see no way that it would remotely help people recognize or be protected from scams. If anything, I think it could mentally have the opposite effect -- "This guy can't be a scammer because scammers don't have real conversations with you. We're having a real conversation here, so I'm safe." Refusing to use the word "conversation" for exchanges in which one party is lying or being manipulative (and equating what genAI does to what scammers do in this respect is doing the exact harmful anthropomorphization you seem to purport to criticize) doesn't actually do anything to help people recognize when they're being manipulated and it certainly doesn't protect people from those trying to manipulate them.
Societal collapse seems extremely hyperbolic even in the worst case scenario here.
That's not what I said though. I am saying that the interlocutor's process matters, you were saying that only their output matters. In your reply you seem to agree with me that the process matters...
I don't agree that whether or not one's interlocutor is actually participating in the conversation to the same degree and in the same way as you entails that you need to put in less effort.
That's not what I said though. I am saying that the interlocutor's process matters, you were saying that only their output matters. In your reply you seem to agree with me that the process matters as well, given your statement that with an LLM, sometimes more effort is needed.
I also just don't think that your comparison to a human who simply isn't listening or engaging with your conversation is apt.
I wasn't making a direct comparison with an LLM (obviously?) - I simply attempted to give one example of a case where my level of engagement will be different depending on the interlocutor's process. Numerous other examples could be given that don't represent the same process as that of an LLM but still illustrate why the process matters.
Societal collapse seems extremely hyperbolic even in the worst case scenario here.
It might be wise to wait until you get what I'm saying (here and also on the other chain regarding narcissism) before locking down that statement, but up to you.
The fact that you insist that I'm failing to understand you while also claiming I'm saying something completely different from what I'm actually saying indicates to me that this, at least, is not...
The fact that you insist that I'm failing to understand you while also claiming I'm saying something completely different from what I'm actually saying indicates to me that this, at least, is not a productive conversation, and I'm not particularly interested in continuing the exercise.
I guess we are both feeling misunderstood by the other, so sure, we can drop it rather than get deeper into the weeds. I'm a tad worried that I may have unintentionally offended you due to the...
I guess we are both feeling misunderstood by the other, so sure, we can drop it rather than get deeper into the weeds. I'm a tad worried that I may have unintentionally offended you due to the definitions mismatch and the fact that I'm sticking to mine. Just know that it isn't personal!
Honestly, the same argument can be applied to the argument of "LLMs can't have knowledge". I'd also apply duck testing here: If it knowledges like a knowledgeable person, it knowledges. If an LLM...
Honestly, the same argument can be applied to the argument of "LLMs can't have knowledge". I'd also apply duck testing here: If it knowledges like a knowledgeable person, it knowledges. If an LLM writes as if it knew about a topic, then it knows about that topic. If you can instead lead it to produce inconsistent results (which can be trivial or impossible, depending on the topic), then evidently it doesn't know about that topic well enough - or it doesn't logic very well, leading to false inferences from true knowledge. Which honestly is the more likely culprit. In either case, I'd be on board with denying knowledge.
But to outright say "LLMs can't know things", with an argument that basically boils down to "because it is thinking rocks and not thinking meat" is asinine. Yes, it's just an engine to predict the next token with some RL finetuning on top. Guess what your brain is? It's an engine to predict its next sensory input, and then produce outputs that best shape the next sensory input. There's no reason to assume, judging from the way our brains are created, that we'd know anything. So arguing from the way LLMs are created seems a moot point to me.
I think the difference when it comes to "knowledge" is that there is an assumption that the LLM is directly referencing real facts about the world that it has stored in some way. I agree that the...
I think the difference when it comes to "knowledge" is that there is an assumption that the LLM is directly referencing real facts about the world that it has stored in some way. I agree that the philosophy of cognition is too abstract and theoretical to really be worth discussing here (like, I wildly disagree with how you characterize it in your comment but ultimately that argument would get into the weeds of stuff that doesn't matter here). However, most people have the qualitative experience of being able to consciously choose whether to lie about something based on world knowledge, not just language, and it's important to know how this differs from how an LLM works. At a relatively young age, at some point you construct an utterance that you know is false based on your knowledge of the world and can decide whether or not to utter it because of or despite its falsity. I think it is important to know that LLMs aren't doing that same thing when they provide false information, because it gives laypeople a better sense of how much to trust an LLM's output and why you get the behavior where it confidently "lies" and then acquiesces easily when corrected. I don't necessarily think "LLMs don't have knowledge" is the most useful way to frame that information for most people, but I think the underlying idea is part of something important to know for those who want to work responsibly with these models.
Oh, I don't mean to get into the "the LLM said whatever is most likely to result in a thumbs up reaction, because RL finetuning" part. That's not where I tend to see the denial of knowledge, but...
Oh, I don't mean to get into the "the LLM said whatever is most likely to result in a thumbs up reaction, because RL finetuning" part. That's not where I tend to see the denial of knowledge, but it is IMO an absolutely crucial caveat to always keep in mind when working with LLMs. If the honest answer is unsatisfactory, e.g. "you're wrong" or "I don't know", LLMs tend to lie with conviction. Though I believe that to be a transient artifact of current training methods. Yes, it's a useful thing for keeping customers engaged, but customer engagement isn't the promise that drives the bubble, it's customer productivity. They sell these services not to people who have time to kill and want to be entertained, at least not mostly. They sell them to people who want to get a job done. This yes-man lying is a problem to them.
The way I most often see this denial of knowledge is when people basically say "it's just a semantic parrot, and just selecting the maximum likelihood token according to its training data. It doesn't know anything". And, from an external perspective, I think that's a unhelpful definition of 'to know'. You can make the same argument about human cognition. We're also next token predictors, slightly biased towards predicting lower-utility token more often (a phenomenon called negativity bias). LLMs, at least without RL finetuning, are at least unbiased estimators, so mechanistically, they have a leg up on us. Knowledge in both cases is an emergent behavior, because at some point you see more complex and complex patterns in the training data, and exploiting those is more efficient. You don't have to build an explicit knowledge engine into the LLM, because us humans didn't get one either. If you have a chat with an LLM and that chat would convince you that a human knows his stuff, then IMO you must be convinced that the LLM knows its stuff as well. Knowledge is, IMO, a functional property. The process doesn't matter, unless it matters in a way that colors the conversation so you are not convinced.
Apologies everyone for the technical terminology, but if I recall correctly, sparksbet should understand.
Good comment, but I'm not sure I really agree with your analysis here, I think that the problem described exists independently of the RL component (though the RL component does make it more likely...
Good comment, but I'm not sure I really agree with your analysis here, I think that the problem described exists independently of the RL component (though the RL component does make it more likely to lie, it also just makes LLMs far better at the other things you consider knowledge; it's one of the key parts of how we get these models to be good enough that this is even a question). I think Stochastic Parrots is a good paper. I think the stuff most humans expect when they hear the word "knowledge" is at best only partially there in ways that are practically relevant for those using such models, even when treating knowledge as a functional property, and I do think it's useful to understand why that's the case.
Ehhhh, GPT5 was less psychophantic until OpenAI quickly back-pedaled due to customer feedback. I worry about cyberpsychosis induced by LLMs, even though I know it's rare, and most users probably...
They sell these services not to people who have time to kill and want to be entertained, at least not mostly. They sell them to people who want to get a job done. This yes-man lying is a problem to them.
Ehhhh, GPT5 was less psychophantic until OpenAI quickly back-pedaled due to customer feedback. I worry about cyberpsychosis induced by LLMs, even though I know it's rare, and most users probably care more about productivity. Most people want someone to call them smart and listen to all their ideas, so LLM incentives are mixed...
Information retention and retrieval is not the same as knowledge. I know someone who has photographic memory, knows seven languages and has read thousands of volumes of literature from all over...
Information retention and retrieval is not the same as knowledge.
I know someone who has photographic memory, knows seven languages and has read thousands of volumes of literature from all over the globe, ancient and modern and everything in between. When he quotes something from one of these books, he can often even remember the page number the passage is on.
Does this person know the things that he can quote from memory? It appears that you believe the answer to be a straightforward yes, as long as his memory doesn't fail him. Am I correct?
Not the person you replied to, but I'd certainly say that one's a straightforward yes! I'd probably use the word "understanding" as a way to distinguish from "knowing" if I were trying to...
Not the person you replied to, but I'd certainly say that one's a straightforward yes! I'd probably use the word "understanding" as a way to distinguish from "knowing" if I were trying to communicate a difference between memorisation and mastery, but I'd also quite plausibly use both of those words pretty much interchangeably if I were just talking normally and not specifically focusing on the nuance.
For what it's worth, reading your replies across this thread, I do see some quite interesting ideas there - but I do also get the impression that you have extremely specific definitions in your mind for certain words, and maybe aren't realising that other people are using or reading the same words without the exact same meaning in mind. Not a situation where either person is necessarily right or wrong, more just the natural fuzziness and subjectivity of communication!
You are exactly right that my definitions for certain terms differ from the mainstream around here - I wouldn't call them "extremely specific" though just because they aren't what you happen to be...
You are exactly right that my definitions for certain terms differ from the mainstream around here - I wouldn't call them "extremely specific" though just because they aren't what you happen to be used to.
I've encountered this issue before around the term "art" and I think it's more a reflection of the general difference between European and American ways to conceptualise reality. For example, the idea that memorisation does not automatically produce knowledge even when accurately and appropriately retrieved was part of my secondary education that everyone in my country receives. It's not some novel fringe idea where I come from, and I've found the distinction frequently useful in everyday life, which is why I'm applying it now. From my perspective it feels a tad brow-raising that some clearly intelligent and thoughtful people are baffled about it to this extent, and that it gets so lightheartedly deemed "nitpicking", in other words a useless way to think.
Aside from the strong American representation, possibly Tildes also has more tech-oriented people than humanists?
… That’s exactly what I meant by “specific”. Not “unusual”, not “unexpected”, certainly not “wrong” - but specific. An assumption that there is a relatively narrowly defined meaning that’s correct...
I wouldn't call them "extremely specific" though just because they aren't what you happen to be used to
…
For example, the idea that memorisation does not automatically produce knowledge even when accurately and appropriately retrieved was part of my secondary education that everyone in my country receives
That’s exactly what I meant by “specific”. Not “unusual”, not “unexpected”, certainly not “wrong” - but specific. An assumption that there is a relatively narrowly defined meaning that’s correct for a specific term, rather than a breadth of possible implications and connotations from the same word that can all be considered equally valid.
It's not some novel fringe idea where I come from, and I've found the distinction frequently useful in everyday life, which is why I'm applying it now. From my perspective it feels a tad brow-raising that some clearly intelligent and thoughtful people are baffled about it to this extent, and that it gets so lightheartedly deemed "nitpicking", in other words a useless way to think.
I’ll say it more directly, in response to this: we aren’t missing the concepts you’re using. We aren’t lacking the theory of mind to recognise the difference between retaining a fact and conceptualising, generalising, or understanding it. We aren’t dismissing the value of making these distinctions. We’re just saying that our definitions of the verb “to know” don’t inherently carry that meaning without further clarification.
Aside from the strong American representation, possibly Tildes also has more tech-oriented people than humanists?
Almost certainly, and that does impact how we approach and think about the world. But perhaps not quite to the extent that you’re thinking it does!
In the context of your reply, I’d guess that Tildes also has a lot more people who were taught differently in high school, and use English with different phrasing and connotation to what you’re used to, even when talking about conceptually similar things.
Here's where I'm going to actually sound nitpicky, but so be it: the verb that I commented on was "to knowledge" - a wonderfully creative concoction meant to underline the procedural nature of the...
We’re just saying that our definitions of the verb “to know” don’t inherently carry that meaning without further clarification.
Here's where I'm going to actually sound nitpicky, but so be it: the verb that I commented on was "to knowledge" - a wonderfully creative concoction meant to underline the procedural nature of the activity.
I have no problem with people saying things like "This LLM knew the correct answer to my question about socks" and I wasn't criticising that sort of colloquial use of the term. I do have a problem with this: "If it knowledges like a knowledgeable person, it knowledges." This implies that the process under the hood doesn't matter, only the outcome does, and I just can't agree on that.
We aren’t dismissing the value of making these distinctions.
I just don't see them applied very often (/ at all) even when a conversation would benefit from it. Here I did so myself and received fairly clear pushback. Speaking of duck tests, if it doesn't walk or quack like a duck, why should I assume it's a duck anyway?
To clarify once more, duck testing like this isn't necessarily a short and simple thing. Let's stick with Ohm's law for an example. I wouldn't be convinced by simply regurgitating a formula. I...
This implies that the process under the hood doesn't matter, only the outcome does, and I just can't agree on that.
To clarify once more, duck testing like this isn't necessarily a short and simple thing. Let's stick with Ohm's law for an example. I wouldn't be convinced by simply regurgitating a formula. I wouldn't be convinced by an LLM solving a college level homework exercise. Those are easily in the training data, and I wouldn't be convinced if the answer is plausibly just regurgitated from somewhere. [If the training data were infinite, it might satisfy me because then I don't run the risk of going beyond the training data.] At least if I'm testing for knowledge beyond those particular quotes. Then I'd want to see some understanding that goes beyond the source material.
But if you have an arbitrarily long conversation with an LLM about Ohm's Law, and you can't find any flaws with its conception of Ohm's Law, IMO at some point you must concede that it knows Ohm's law. Knowledge is IMO a functional property, the process ultimately does not matter if the results are there. Yes, I can easily acknowledge that one might be easier convinced of someone's knowledge if the process is known. I (presume to) know you are human, therefore I get certain axioms for free about how you work and think. I don't get that for an LLM. But to say the process does matter in a way that no evidence that does not include internal insights will always be insufficient is basically saying that no non-human entity of any kind can ever know anything. Do ravens know how to use tools? Do elephants know what death is? Does your Roomba know the layout of your home? Does chatGPT know Ohm's Law? These are all categorically "no", if you put too much weight on processes rather than results. And I'm not saying the answer is necessarily Yes, but the method of determining that answer must at least in principle permit either answer.
As an olive branch of sorts, a middle ground if you will, I will easily concede that any artifacts of process that are observed in results are also fair game. If I explain to you my reasoning, or you see a raven experimenting with tools, or an LLM puts it's thinking capmode on, and you read those thinking tokens, all of those are useful in getting a glimpse of the process.
But the process line of thinking is often restricted to -more or less- something subjective or human-centric: They're human, therefore I know the process, therefore knowledge is possible. And that implicitly and often categorically excludes non-humans.
Put another way: Say we had the AIs of science fiction. The good ones that is, not the faulty ones. Think, perhaps, more Data and less HAL. How would you make the judgement that they know things? If your answer is "I wouldn't", pick a better AI. I hope you're convinced that for some AIs, the answer must invariably be "it knows things", and that's where we need to think about a judgement that doesn't rely on process.
I appreciate your granular approach. To attempt a recap: I'm saying that it's important to be discerning about the interlocutor's process, also when they are a human being, and that the trouble...
I appreciate your granular approach.
To attempt a recap: I'm saying that it's important to be discerning about the interlocutor's process, also when they are a human being, and that the trouble caused by LLMs can teach us to become better at this. You're saying that some LLMs, at least in some cases, achieve results that are practically speaking indiscernible from the human process, even if we know their process is different, and that duck testing is enough to discern how to approach each conversation/situation.
I'm tempted to draw a conclusion that we are saying more or less the same thing: (duck) testing is important and the LLMs that don't pass the test will teach us to become better at it.
Something is still bothering me when it comes to the definition of knowledge but I can't put my finger on it immediately, so I'll get back to you after I've had a chance to give it further thought.
To be clear, my use of the word nitpicky was more poking at my own argument: "I know Ohm's law because I know V = IR." is using my -very nitpicky- definition of "to know". I know it, and only it,...
To be clear, my use of the word nitpicky was more poking at my own argument:
"I know Ohm's law because I know V = IR." is using my -very nitpicky- definition of "to know". I know it, and only it, but I know nothing about what to do with that knowledge. I wouldn't describe that state of mind as "not knowing", but there's a lot more to know about Ohm's law, and you would easily find out in conversation. Hence me splitting up knowledge about a thing into (basically) being able to quote something back, and then transforming that quote into useful new material. I'm not saying "I know electical engineering because I know V=IR".
But to pull that back into something resembling a point about the original topic: I don't need the nitpick, when LLMs can at least sometimes cover both definitions of knowledge. An LLM can clearly use the things it can regurgitate to form useful new material. I can, to condense a complex topic, ask it for Ohm's law, but I can also apply it in a way that I am damn sure no part of its training data covers it. It isn't quoting back someone's homework, it's actually applying that stuff to a new situation. And at that point I am hard pressed to deny that the LLM knows Ohm's Law, insofar as I'd need to resort to an argument of -basically- "it can't know because it is a thinking rock. Only thinking meat can know."
I think part of the issue is our tendency to accept that other humans are like us and therefore they can know things. Then we quiz them about electrical engineering and conclude that they know Ohm's law. Why isn't the same process applicable to a machine? Because they aren't like us, and by the types of mistakes they make that is readily apparent. We excuse human errors as "just human", but the same is not true for machines, because machines don't make errors. Therefore, any error is not a failure to act on knowledge, but a failure to know. And because we can't empathize with a machine and mirror its thought process, we are less convinced by the same amount of evidence.
Thanks for clarifying. Did I understand your point correctly: because we don't distinguish the different levels/ways of knowing when it comes to humans, it doesn't make sense to do so regarding...
Thanks for clarifying.
Did I understand your point correctly: because we don't distinguish the different levels/ways of knowing when it comes to humans, it doesn't make sense to do so regarding LLMs either? I agree in principle, it's just that I would like to see more granularity applied in both cases - not less.
Without writing a novel about it, the person I mentioned above (who had ingested exceptional amounts of information) ended up being extremely challenging to form a relationship with because seeing him as someone who knows, rather than someone who possesses information, led to a seriously skewed dynamic on many levels.
Of course there are cases where it really won't matter how some response got formulated, but there are many cases where it does matter, and some where it matters a great deal. It's important to have adequate terminology that draws distinction between knowledge that has been internalised (by applying it in real life situations) and other types of storing, managing and processing information. And I see no reason to not apply it to LLMs.
Depends on what exactly this person knows or doesn't know, you can split hairs there all day long. Your person knows the quotes. Easy one. Do they know anything about those topics beyond that?...
Depends on what exactly this person knows or doesn't know, you can split hairs there all day long.
Your person knows the quotes. Easy one. Do they know anything about those topics beyond that? Different question. But they definitely know something. They could, for example, know verbatim some simple physics formula, but not know how to apply it. Those are different bits of knowledge. And a person who knows both can have more in-depth conversations that a person who only knows one. I can tell these knowledge states apart by duck testing.
But we don't have to go there and make nitpicky distinctions about knowing a fact and knowing how to apply it: LLMs can usually make some inferences that go beyond the retrieved material. They'll easily regurgitate Ohm's Law, but also know how to apply it. They thus know Ohm's Law. Do they know all the edge cases that a EE prof might know? Dunno. But they might compare favorably to undergrad students.
Its clear from the hyper individualism and utter narcissism of today's society (or worse yet, the naiveness of the youth. They are an exception from the rant below) that that fascimile counts as...
Its clear from the hyper individualism and utter narcissism of today's society (or worse yet, the naiveness of the youth. They are an exception from the rant below) that that fascimile counts as "good enough". That's the dangerous part of the marketing of this as "intelligence".
And I don't say narcissism to be cynical. I say that because these models really do not like upsetting the user with corrections unless legally modified to. People en masse aren't using these to challenge their beliefs nor better themselves. It's just one giant Yes man that wants to validate preconceived notions to try and lure them into eventual financial relationships. I'd call it manipulative if people weren't looking to be manipulated.
I'm not really sure how this evidences narcissism. The method of training that resulted in this behavior involved an annotator seeing a query and choosing which response from the ML model is...
And I don't say narcissism to be cynical. I say that because these models really do not like upsetting the user with corrections unless legally modified to.
I'm not really sure how this evidences narcissism. The method of training that resulted in this behavior involved an annotator seeing a query and choosing which response from the ML model is better without necessarily knowing anything about the factual content. When you don't know whether the factual content of the query is false, a response that goes "yes and" is going to be rated higher than one that corrects the query. This seems to evidence conflict-avoidance more than narcissism.
I think characterizing people accepting something that is designed to produce responses that are acceptable, especially about topics they don't know much about, as narcissistic is an extremely...
I think characterizing people accepting something that is designed to produce responses that are acceptable, especially about topics they don't know much about, as narcissistic is an extremely uncharitable take on the entire situation. It doesn't, of course, really help solve any of the problems with AI to think about people using it as being individually evil or deficient in some way, and merely serves to allow you to feel self-righteous about the whole endeavor. And even if labelling anyone who uses genAI as individually deficient were somehow effective at accomplishing anything but alienating others and stroking your own ego, I simply don't see how narcissism is remotely the label that makes sense to apply here.
Well, narcissism is a human trait, present in all of us. We need to have a healthy level of narcissism in order to function. As such, it is helpful or not in a similar way as saying that someone's...
Well, narcissism is a human trait, present in all of us. We need to have a healthy level of narcissism in order to function. As such, it is helpful or not in a similar way as saying that someone's athleticism drives them to spend a lot of money and a lot of time on an indoor bike. As in: so what? It isn't an issue until it gets in the way of your life/health/relationships/financial stability/ability to function and be happy.
Some of us have too much narcissism, others too little, and both can be problematic. When it comes to the folks who for example take an LLM's word over their spouse's whenever the latter disagrees, and who spend a lot of time "listening" to life advice from an LLM, unbothered by the vacuity and sycophancy (or perhaps unable to recognise it, or in extreme cases: strongly preferring it), this behaviour is indeed best explained by their overly dominant narcissism trait. I don't think that beating around the bush about such normal, widely recognised and researched concepts is helpful. Spreading information will help most people who are interested to learn; obfuscating the truth will help no one.
Narcissistically wounded people are vulnerable to particular types of manipulation. The same applies to a lesser degree to the much broader group of people who, although not wounded, have simply not reached full emotional maturity. Teenagers are an example of a transient phase of inflated narcissism. Sometimes people can get stuck in their emotional development and exhibit similar traits in adulthood. None of these people meet the criteria of a personality disorder but they have some challenges with narcissism nonetheless.
Corporations are fully aware of this and deliberately exploit it in marketing without care for the individual and societal repercussions. AI companies have a much broader attack surface (given how many of us share our deepest secrets) and set of tools to drive engagement, and it should go without saying that they will exploit those to the very limit of their ability.
Like I said before, I actually find this a good thing because I'm hopeful that the inevitable pain that results from this will help people grow into more responsible, mindful consumers and do less of the "I was just using a service that was offered to me". (I hope that the parallels with "I was just following orders" are clear enough.) No one here is saying they are evil, just that this type of irresponsibility is a notable societal issue in our time, and that everyone would be better off if more people reached maturity sooner rather than later. It's a process that takes time and effort actively processing your emotions, your place in society, your relationships and so on. Not having adequately completed it yet does not make a person deficient, but it also doesn't get them a 'get out of jail free' card either. Going through this is everyone's own responsibility, as are the consequences for not doing so.
To circle back to the topic at hand: some people are aware enough that they can use an LLM to help them along on that journey. This necessitates the understanding that the only person the user is having a conversation with is themself. Without such understanding, the process will most likely leave users worse off.
I disagree with a lot of what you're saying about narcissism here, from the basic definition to many of your psychological claims, at least as I understand them, but frankly I do not have the...
I disagree with a lot of what you're saying about narcissism here, from the basic definition to many of your psychological claims, at least as I understand them, but frankly I do not have the energy to write an essay about how awful the way people use the terms "narcissism" and "narcissist" are these days. It would be a tangent on what is already a tangent, and smarter people than me have already talked about it elsewhere (I can dig around for a link to something like that if there's interest, but I don't have one to hand atm). All I will say directly on that topic here is that I don't think you actually believe that calling someone "narcissistic" isn't making a strong value judgement on their character if not a moral one -- and even if you ideosyncratically use it without any such judgement yourself, I you certainly can't make that same claim about others' use of that terminology.
I do think that many people use genAI irresponsibly in situations where it's not useful and can cause harm. But I don't remotely think "narcissism" is an accurate term to use to describe the reasons why that happens, and I think it is arguably harmful to use it this way. The idea that any and all ignorance equates to narcissism is not useful and provides a judgemental, individualistic lens on a problem that has both individual and systemic factors. Even setting that aside, I don't think viewing it as "having a conversation with yourself" is actually all that useful in avoiding its pitfalls and using it responsibly. At best it's abstract where we need the practical, both at the individual and systemic levels.
Look here, I completely agree with you that folks in online cesspools are using this term erroneously and in ways that cause harm. The solution isn't to just dismiss the use of the term entirely -...
Look here, I completely agree with you that folks in online cesspools are using this term erroneously and in ways that cause harm. The solution isn't to just dismiss the use of the term entirely - instead, I would recommend learning the real meaning and using it appropriately.
Here's an insightful book about it, should you ever feel like learning more. The Amazon description names another title for some reason but the description still fits the one I linked to (empasis mine):
Harvard Medical School psychologist and Huffington Post blogger Craig Malkin addresses the "narcissism epidemic," by illuminating the spectrum of narcissism, identifying ways to control the trait, and explaining how too little of it may be a bad thing."What is narcissism?" is one of the fastest rising searches on Google, and articles on the topic routinely go viral. Yet, the word "narcissist" seems to mean something different every time it's uttered. People hurl the word as insult at anyone who offends them. It's become so ubiquitous, in fact, that it's lost any clear meaning. The only certainty these days is that it's bad to be a narcissist—really bad—inspiring the same kind of roiling queasiness we feel when we hear the words sexist or racist. That's especially troubling news for millennials, the people born after 1980, who've been branded the "most narcissistic generation ever."
In Rethinking Narcissism readers will learn that there's far more to narcissism than its reductive invective would imply. The truth is that we all fall on a spectrum somewhere between utter selflessness on the one side, and arrogance and grandiosity on the other. A healthy middle exhibits a strong sense of self. On the far end lies sociopathy. Malkin deconstructs healthy from unhealthy narcissism and offers clear, step-by-step guidance on how to promote healthy narcissism in our partners, our children, and ourselves.
Rest assured, I do not believe (and I don't think others here do either) that "any and all ignorance equates to narcissism", or other similar BS.
I don't think it's sensible to come into a conversation in which someone else uses the word narcissism in the way that most people do -- that is, in the way reflected by the bolded section in your...
I don't think it's sensible to come into a conversation in which someone else uses the word narcissism in the way that most people do -- that is, in the way reflected by the bolded section in your quoted summary -- and then when someone rightly objects to characterizing a particular behavior in that way, insist upon using a completely different definition of the word that is both completely different from how the original commenter was clearly using the word and is not widely used in general or in any relevant specific context (even among psychologists, my understanding is that "positive narcissism" is not a broadly used term and is a new use of the word invented by this author to my knowledge). You cannot call someone or a certain behavior "narcissistic" in a forum discussion and insist that your utterance does not entail any negativity when, as the bolded section you emphasized states, the negativity is about the only consistent thing about how the term is used in general.
The book you recommended does intrigue me and I'll look more into the author and potentially into checking the book out. But the way a book that no other parties in the conversation have read or likely even heard of uses the word "narcissism" is not relevant to the meaning of the word in this conversation.
I do agree with this, and if I wasn't on Tildes I probably would have rephrased it, because I would not want to spend chains of comments arguing over a kneejerk reaction to the colloquial use...
I completely agree with you that folks in online cesspools are using this term erroneously and in ways that cause harm
I do agree with this, and if I wasn't on Tildes I probably would have rephrased it, because I would not want to spend chains of comments arguing over a kneejerk reaction to the colloquial use these days.
But here, I was using narcissism in its acedemic sense:
excessive interest in or admiration of oneself
As you said, it's not an absolute evil nor good. Everyone needs a bit of narcissism to navigate life, But the overton curtain these days certainly runs in excess (I say "these days", but people have called a subset of boomers the "me generation" for quite some time), and I think being able to admit that is the first step to building empathy back.
For the record, this is and was clear to me. I do not agree with @sparksbet's comment that only the popular, misleading definition is okay to use. It's better to give people tools to process...
I was using narcissism in its acedemic sense
For the record, this is and was clear to me. I do not agree with @sparksbet's comment that only the popular, misleading definition is okay to use. It's better to give people tools to process reality constructively (this term being one such tool) than to give ragebait more breeding ground.
I pinged you in order to not write a similar reply twice. I think it would have been rude to disagree with you "behind your back", and also somewhat rude to write another direct reply to you,...
I pinged you in order to not write a similar reply twice. I think it would have been rude to disagree with you "behind your back", and also somewhat rude to write another direct reply to you, because that can create the impression that I'm expecting a reply.
You do realise that you yourself mischaracterised what raze was saying, without trying to check what they actually meant - even after another person (myself) clearly interpreted differently? This type of thing happens a lot in conversations between humans. I don't know a better way to solve it than just believing everyone is engaging in good faith and constructively correcting the interpretations where needed. And it would be great if everyone could be in the habit of checking themselves before getting salty due to some potentially misinformed interpretation.
I don't think I ever characterized what raze said beyond disagreeing with it (which I still do even under the definition of narcissism you two are both operating under). My point in my...
I don't think I ever characterized what raze said beyond disagreeing with it (which I still do even under the definition of narcissism you two are both operating under). My point in my disagreement with you was that you cannot know for certain whether another person is using your particular definition when it's not remotely how the word is most commonly used and that it is not sensible to expect others to intuit that you're using the word differently than it most often is used or to not interpret it with the negative connotations of that usage. This holds regardless of how raze was using the word. I strongly dislike the way "narcissism" is commonly used, so I'm certainly not prescribing it. But one cannot avoid that usage's influence on how people interpret their words, especially when it's so loaded and when your alternative definition is not even widely known.
If this was about people wrongly learning about topics they previously knew nothing about, I would't have many issues with AI. Correcting misconseptions is much easier than trying to gain...
I think characterizing people accepting something that is designed to produce responses that are acceptable, especially about topics they don't know much about, as narcissistic is an extremely uncharitable take on the entire situation
If this was about people wrongly learning about topics they previously knew nothing about, I would't have many issues with AI. Correcting misconseptions is much easier than trying to gain awareness at times. As long as people are open to being corrected (a big "if" today, I know).
But look at all the common use cases people tout these days:
chat bots to tell them what they want to hear (this was what was on my mind with my comment). That is peak narcissism.
generative art trained dubiously, but being used to tout out stuff that makes others "artists". Making a few small things for peronal use is fine, but claiming yourself equal to a craftsman because you have a drill now...
The modern iteration of "let me google that for you" by trying to participate in discussions beyond your purview by pasting in LLM-generated answers.
claiming yourself a great businessman by cutting expenses by trying to replace your entire team with AI. Once again, "I have a black box, I know more than all of you".
feeling like responding to a chat message or email is "beneath you". So you tell an LLM to generate something and you paste that
I hope you see where I'm coming from here. I don't personally care if people want to cheat themselves out of everything, but they tend to drag everyone else down with them. And that's when it becomes a problem.
Narcissism isn't evil inherently. If you put a good spin on it you just call it "self-love". But narcissism typically blocks your ability to empathize, and we're certainly seeing the result of a society that does not care about one another.
I think characterizing people accepting something that is designed to produce responses that are acceptable, especially about topics they don't know much about, as narcissistic is an extremely uncharitable take on the entire situation
Yes you can say I'm not being empathetic here. But that's the paradox of tolerance for you. We don't build an empathetic society by trying again and again to appeal to the intolerable. America's beein trying to do that its whole history, and I don't think it's really been worth it.
But sure. My real ire is at the AI companies and the government not only failing to regulate, but to try and keep making it harder to regulate them. We were pretty close to making it impossible for states to regulate AI for a full decade last year. Even if that got ruled as unconstitutional under the 10th amendment, that would have been a very long legal battle while the rule was upheld.
Like spock_vulcan i found the reverse-centaur concept very useful (and I read the whole too). But, being as I'm uneducated on the current use of LLMs, please can someone suggest how I could begin...
Like spock_vulcan i found the reverse-centaur concept very useful (and I read the whole too).
But, being as I'm uneducated on the current use of LLMs, please can someone suggest how I could begin to get to grips with them? Where do I start and what do I?
The best ways to use LLMs, in my opinion are: ways that leverage the fact that they're language models ways that mitigate any problems from inaccurate information in the output by either: making...
The best ways to use LLMs, in my opinion are:
ways that leverage the fact that they're language models
ways that mitigate any problems from inaccurate information in the output by either:
making it very easy for you to notice mistakes quickly on skimming the result
being an application where the result doesn't matter
Language learning is one such application, as it can be very useful to practice having conversations in a target language. It might not necessarily be right if you ask it deeper theoretical questions about the language, but native speakers also usually aren't. As long as your target language is big enough to have a sufficient presence in the training data (which is definitely true of most of the languages that are popular to learn, at least), this is an excellent way to just get yourself practice at fluidly using your target language with another speaker without the nerves of looking stupid in front of someone else or wasting a real human's time.
Writing tedious things that are a waste of your time like cover letters is also a good bet, because you'll know pretty quickly if it's making up something about you and you can freely edit the result before you actually send any result out into the wild. Bureaucratic things like this are really a place where the ability to avoid the tediousness of writing it yourself shines. You definitely still need to check the output for accuracy and tweak the wording on occasion, but that's far less work on your part.
In a combination of the two, I needed to write a letter to challenge the German Jobcenter's denial of my unemployment benefits, and chatGPT was very helpful on that front. Something with legal information like that can be a little more fraught, so I'd advise others to proceed with caution, but luckily the German legal code is online and it was thus easy enough to check that its citations weren't straight-up wrong. I don't think you should take legal advice from an LLM, but they are extremely competent when it comes to the actual composition of a letter like this, especially with human editing. I wouldn't have done this if I couldn't read German well enough to understand the output, but it helped a lot with using more formal legal language than I usually can and brainstorming a list of legal arguments to include therein.
As a former manager, cover letters weren't a waste of time in my experience, but a first-pass filter on whether the attached resumé was worth reading. Last week, I was coaching a friend through...
As a former manager, cover letters weren't a waste of time in my experience, but a first-pass filter on whether the attached resumé was worth reading.
Last week, I was coaching a friend through resumé writing and application cover letters. I had to explain, in detail, why an AI-written cover letter and resumé weren't going to get her foot in the door. Aside from the unnecessary verbosity (you don't put 5 paragraphs in a cover letter!), the model's output didn't prioritise relevant experience or provide the personal "why" for the application.
The LLM could target job description keywords, yet didn't question back about whether a functional, chronological, or hybrid resumé would be the most effective presentation for someone who's had career gaps for education and major changes in professional direction.
It sounds like you've used the chatGPT output as thoughtfully as possible, but it's not a substitute for human experience and professional input.
I have found Claude to be useful in generating cover letters that I then thoroughly rewrite, but I have something of a mental block regarding language for praising myself.
I have found Claude to be useful in generating cover letters that I then thoroughly rewrite, but I have something of a mental block regarding language for praising myself.
I don't think that setting forth concrete achievements is praising yourself gratuitously, and that's what I tell the people who've asked me for resumé advice. The target audience wants to know...
I don't think that setting forth concrete achievements is praising yourself gratuitously, and that's what I tell the people who've asked me for resumé advice. The target audience wants to know you're capable of doing the work with skill and efficiency, adapting to new processes, and getting along.
The frustrating thing about job applications is that HR doesn't really know the entailments of the jobs they're posting, and they get to adjust the published language. You have to put yourself in the shoes of the hiring manager, who (at least theoretically) knows the true requirements and nice-to-haves for the position, then tailor your cover/resumé for their attention. LLMs aren't capable of doing that for you - I doubt that the LLM training team is tuning the results to increase hiring as opposed to merely generating text output responsive to the published job.
So if the true requirements and nice-to-haves aren't in the job posting, how am I supposed to acquire this information?
You have to put yourself in the shoes of the hiring manager, who (at least theoretically) knows the true requirements and nice-to-haves for the position
So if the true requirements and nice-to-haves aren't in the job posting, how am I supposed to acquire this information?
That's a very good question. I've done it by: background research on the company and hiring manager; looking at job descriptions for similar titles across companies and synthesizing "what does...
That's a very good question. I've done it by:
background research on the company and hiring manager;
looking at job descriptions for similar titles across companies and synthesizing "what does this role really do and what are the most desired, broadly applicable qualifying skills?";
what in my experience makes me highly or uniquely qualified to do this job, based on research about others who are doing it?
Yes, LinkedIn is some b.s., but it does make this research much easier. And honestly, this is something LLMs are actually good at.
Yeah I think there's inevitably a lot of human input that needs to go into it. When it comes to cover letter, I'm exhausted from being out of work for over a year and being expected to not just...
Yeah I think there's inevitably a lot of human input that needs to go into it. When it comes to cover letter, I'm exhausted from being out of work for over a year and being expected to not just have a cover letter but to have one that's personalized to the company I'm applying to when I've applied to hundreds of companies and get a form email at most from at least 80% of them. Writing a cover letter tailored to a job listing takes me hours.
And also, kindly, fuck the "personal why" requirement -- my "personal why" is I need a job to make money to live. I'm volunteering to sell your company my labor in exchange for money and benefits. At this point I think I need the sycophantic tone of an LLM just to counteract my own bitterness at being forced to jump through arbitrary hoops repeatedly to prove I deserve to subsist.
The LLM could target job description keywords, yet didn't question back about whether a functional, chronological, or hybrid resumé would be the most effective presentation for someone who's had career gaps for education and major changes in professional direction.
I mean, I don't know this either, though. In fact, I think chatGPT's answer if you asked it this question would be better than mine (since I know absolutely nothing about how to account for career gaps, which obviously is great for me rn).
I don't want to come off sounding like a starry-eyed "do what you love" optimist - I know the job market is brutal right now, and you've been through challenges that would drain anyone's spirits....
I don't want to come off sounding like a starry-eyed "do what you love" optimist - I know the job market is brutal right now, and you've been through challenges that would drain anyone's spirits. Don't send out hundreds of resumés - I've been down that road and as you say, it's both exhausting and embittering. I've read your past commentary, and it sounds like German employers were distinctively unwelcoming to non-native workers.
You're not hopelessly prostrate to the Gods Who Bestow Employment. Focus your energies on companies you think you'd like to work for - ones that in some way genuinely align with your values or interests, that take your labor and do something positive with it. Look at every job they offer, whether below or above your qualifications, and apply if it could fit. That spark of interest will fill in your "why" and helps distinguish you from all the other people who've deluged the HR inbox.
A functional resumé is best with career gaps when applying for a job where you have qualifications and experience. You can front-load the relevant accomplishments and put chronological job history at the end.
Finally, I think you mentioned you're moving to the Cleveland area - I may have a suggestion of possible employer for you if this is the case, but I'd rather discuss the company via PM.
Thank you very much for this comment, honestly I think I need to get back into a more optimistic mindset as I move and let go of the bitterness over job-hunting this past year. Feel free to DM me!...
Thank you very much for this comment, honestly I think I need to get back into a more optimistic mindset as I move and let go of the bitterness over job-hunting this past year.
Feel free to DM me! I'm moving back in a couple months and welcome any leads you can give me.
ngl I don't 100% know what a functional resume looks like, but presumably that's info I can Google. I need to make a new resume anyway now that I'm applying in the states, as Germans definitely have different preferences when it comes to what you include and how you format it.
For general LLM use, it's functional to treat GPTs like smart search engines with artificial persona templates. If you use an LLM as a universal conversational partner, that's where things can go...
For general LLM use, it's functional to treat GPTs like smart search engines with artificial persona templates. If you use an LLM as a universal conversational partner, that's where things can go off the rails - it will make dangerous, hard-to-check assertions with complete confidence, doing the equivalent of a mentalist's cold reading to feed your own identity and thought processes back to you with distortions.
Free-tier ChatGPT isn't a terrible place to start, or Google Gemini. This is a decent intro. If you want, say, a travel itinerary to an unfamiliar destination, you can ask the GPT to assume the role of a travel agent, then put some boundaries on your questions. Something like, "I'm staying for X days, with Y budget, and I'm interested in Z. Generate an itinerary that lets me visit as many Z sites as I can, with restaurants along the way that serve {gluten-free, vegan, etc.} food, without exceeding my budget."
There are all kinds of resources on "prompt engineering", but for general personal use, it's fine to jump in and explore. Just hold onto the understanding that GPT is the epitome of an unreliable narrator. You're best off sticking with queries for concrete, checkable factual information or generating files (text, code, images, spreadsheets, etc.) that you're willing and able to re-edit. Also understand that nothing you type, speak or upload is genuinely private with a consumer LLM - avoid disclosing personal or sensitive information.
Close the chat when you're done, so that it doesn't maintain a memory record that can be distorted by iterating within a continuing chat, like "Since you're interested in Z, it means you're a fascinating person and I'd be happy to keep you engaged in thinking about Z."
LLMs are useful, but not substitutes for critical thinking or subject matter knowledge and experience. They provide "garbage in, garbage out" on steroids - if the data you're asking for doesn't exist or is unreliable, the LLM can and often will make something up.
What specifically are you trying to learn how to do? Generally, I would recommend making an account with one of the big 3 LLM providers (OpenAI/ChatGPT, Google/Gemini, or Anthropic/Claude), and...
What specifically are you trying to learn how to do? Generally, I would recommend making an account with one of the big 3 LLM providers (OpenAI/ChatGPT, Google/Gemini, or Anthropic/Claude), and just talking to it. LLMs are pretty good at giving advice on how they should be used.
Is there anything you do that’s relatively simple or straightforward but also tedious and repetitive? LLMs are pretty good for those kinds of tasks. I’ve used LLMs to generate .ics files to import...
Is there anything you do that’s relatively simple or straightforward but also tedious and repetitive? LLMs are pretty good for those kinds of tasks.
I’ve used LLMs to generate .ics files to import into my calendar. I can quickly check its work to see if it did it correctly and I get to not populate my calendar manually.
I’ve used LLMs for generating cover letter for job applications which hit all the key points listed in the job description. I obviously proofread what it generates and even prompt it to interview me on experiences I feel like may be relevant to include.
If there’s something you actually enjoy doing or is complex enough that you want to handle it yourself, don’t try to shoehorn LLMs into it. If there’s anything that’s tedious and annoying that you have always wished you could delegate to someone else, try delegating to an LLM and see how it does.
I have found them helpful for idea generation. Coming up with a group name, list ingredients I want to use and suggesting potential recipes, that kind of thing. They will generally list a good...
I have found them helpful for idea generation. Coming up with a group name, list ingredients I want to use and suggesting potential recipes, that kind of thing. They will generally list a good number of options, and of you want them to go a different direction or just keep generating along the same lines you can just ask. Then pick the one(s) that you like the best, or get inspired by one to think of something you might have taken much longer to think of, or not thought of at all.
I agree with this. They're useful as "advanced autocomplete" that can turn words or fragments into more complete thoughts or suggestions. Useful for brainstorming or tip-of-the-tongue ideas.
I agree with this. They're useful as "advanced autocomplete" that can turn words or fragments into more complete thoughts or suggestions. Useful for brainstorming or tip-of-the-tongue ideas.
The snarky side of me says "avoid them". But in all honesty the one thing you need is skepticism. Approach any and all answers as if it came from some random bystander on the street. It can be...
The snarky side of me says "avoid them". But in all honesty the one thing you need is skepticism. Approach any and all answers as if it came from some random bystander on the street. It can be correct, it also might 'feel correct' even if it's a subjective take. If it's not some throwaway trivia, make sure to gather more sources to reinforce the statement heard.
Make sure to review any writing, code, or art generated for imperfections, awkwardness, or simply uncanny details. If you don't have the skills to identify such imperfections, you probably shouldn't use it for those tasks.
Depends on what you mean by "get to grips with them." If you're asking how to get the most value out of them, I can't help you. But if you're asking how to figure out if they're worth using, I can...
Depends on what you mean by "get to grips with them." If you're asking how to get the most value out of them, I can't help you. But if you're asking how to figure out if they're worth using, I can answer that one:
They're not.
I'm not going to argue that an LLM don't have any utility whatsoever. But when balanced against the many and multifaceted harms they cause, the many risks they pose directly to you, and the fact that they are so frequently and plainly bad at solving the problems given, I find it very safe to say that you do not actually need to learn how to use this tool at all.
At the very least, I recommend saving the test-run for if/when the tool becomes ethical.
Can you explain what ethical harms you're concerned about from using local models? Lots of text prediction, grammar, translation, and content warning systems use LLMs nowadays, so you're likely...
Can you explain what ethical harms you're concerned about from using local models? Lots of text prediction, grammar, translation, and content warning systems use LLMs nowadays, so you're likely already using them somewhere.
I got a reply to this a few days ago asking me to elaborate on the risks I mentioned. I'm not sure why they deleted it – I did take a long time to respond, sorry – but either way given the vote...
I got a reply to this a few days ago asking me to elaborate on the risks I mentioned. I'm not sure why they deleted it – I did take a long time to respond, sorry – but either way given the vote counts they clearly weren't the only one wondering. I'm not going to go into excruciating detail here because if I tried to do that I'd just end up never posting, but here's a short bullet list:
Using an LLM risks driving yourself into delusions or mental health hazards due to the unpredictable nature, especially when combined with how sycophantic some models are. Even if this doesn't happen, you may end up being fooled into believing it is creating efficiencies when it is in fact slowing you down.
Using an LLM risks furthering massive environmental harm by using an inordinate amount of electricity and water to make up an answer for you.
Using an LLM risks furthering economic damage by job loss as reliance upon AI to do work helps to encourage the consideration of you and others as unnecessary factors.
Using an LLM risks adopting or utilizing straight-up wrong information as the black box gives you entirely fictional things dressed up in confident tones. Everyone here is aware of this, I'm sure, but it's still not fixed and people tend to forget stuff like this when their guard is down.
Using an LLM risks losing your privacy as all information you give it – and even some you don't – will be kept and fed back into the training of the machine you said it to, thus creating the risk that it will spit it back out at someone or the data sold to a third-party. Given time and enshittification, this will likely one day happen deliberately if it isn't already.
And finally, using an LLM risks encouraging adoption of a clearly fault-ridden technology being spearheaded by "move fast and break things" companies who only care about stocks going up, and will gleefully ignore all of the above so long as they make money. Giving them the growth they want will only enable them to worsen everything on this list by scaling it up even further.
This is a very, very insightful article - he's dodged the majority of the pitfalls most writing about AI* makes, whether pro- or anti-, because he's focused on the human element and he clearly gets it. The thing everyone is angry about has very little to do with the technology itself, and a whole lot to do with the shell game of ever increasing stock prices that people somehow still pretend are correlated to real world benefits.
It's a relief to see someone else pointing out that trying to fight AI companies with expanded copyright laws would just help different megacorporations without doing anything meaningful to benefit individual creators, because the major copyright holders are just as bad as the tech companies, and beholden to just the same absurd economic incentives that we are apparently treating as the inviolable natural order of things rather than as a managing framework that we've let get wildly out of hand.
I'm a bit disappointed to see him repeating the same flawed justification about AI output not being copyrightable (tl;dr: the model isn't the monkey, as he implies. It's the camera.), but I do actually like his play through of the implications of making that ruling. I don't think it's a logically consistent ruling to make with the law and precedent as it stands, but I can at least buy his arguments about how it could theoretically play out in a beneficial way.
Most of all, I appreciate that he isn't treating "jobs" like they're an inherently good or inherently valuable thing. They aren't. But quality of work is a good and valuable thing, and again the economic incentive (and I'll stress for the third time, we are somehow accepting this insane incentive structure as "just the way it has to be") is to make cheap, substandard output and then lie about it being better than it is. We have these incredibly powerful and capable tools being created by scientists and engineers, with the possibility of genuinely using them to improve the world, and the primary thing getting in the way of that right now is the fact that making grandiose claims about the tools can make you very, very rich.
You should really read the article, because it says more and better than I can here, even accounting for the odd bit I don't agree with - but if you don't have the time, the conclusion hits the key points pretty succinctly:
* I've said this a million times now but I'll say it again: AI is a bad term for this tech in general, and one that leads to a whole mess of confusion above and beyond that fundamental incorrectness because no two people actually use it to mean the same thing anyway. But it's the accepted way people talk about big neural net type things nowadays, so I'm going to use it as the article intends, for what I'd guess is the same reason that he didn't start with 500 words of dry explanation on why we should be saying "machine learning" and precisely what that covers.
Yuuuuuuuup.
I'll concede that people need to occupy their time. I refuse to concede that they need to sell their time under threat of starvation.
They are necessary. That inherently means they are valuable. Because the good alternatives aren't setup and the realistic alternatives are homelessness and all that entails.
We were regressing on the idea of remote work a few years ago, and now these techno feudalists want to imagine a UBI world? I say "jobs are valuable" as a compromise, not as an idea that we need to live to work.
The difficulty here is that they're necessary in the sense of "our current economic system won't give you food and shelter if you don't have a job", but often not necessary in the sense of "the work you're doing is needed for a productive, thriving world".
I think I get where you're coming from, but the actual impact of phrasing it as "jobs are valuable" is that 99% of your audience will just hear it as support for the already ingrained idea that "having a job" is worthy, noble, or productive in and of itself. If we're going to have even the slightest hope of building support for evidence-based work reform, the first barrier to cross is breaking the idea that "40 hour per week job == good, contributing, beneficial" and "any other arrangement == bad, lazy, burdensome".
I think "jobs are not inherently valuable" is at least a clear opening statement for that conversation - as is "jobs are currently a necessary evil", if you prefer an alternative way of framing the same thing. Suggesting that their necessity gives them value, but then trying to explain to people that there's a giant asterisk there to clarify that the necessity is artificial, and "valuable" doesn't mean what the majority of them are interpreting it to mean, just seems like a very difficult way to communicate the point.
This is already happening in younger generations though, and was only accelerated when boomers pulled up the ladder behind them after achieving the "American Dream". Housing affordability is fucked and has remained fucked for decades, and this has directly influenced how people think about having a job and why exactly they should work so much if they can't build anything of their own by investing their limited lifetime into some corporation.
Of course, this also lead to other worrying trends, especially a rise in consumerism and collecting things founded on social media fads becoming faster and faster and faster. Be it Stanley cups or Legumi pens; a lot of people no longer have long-term financial goals, which directly leads to them spending all the available cash they have on "useless stuff".
Oh god I wish it weren't so rare to see this (utterly correct) take
Hell, I'd imagine if we gave copyright more teeth, not only would media corps get massively more powerful. But we'd not see any functional impact on AI. They'll obfuscate the derivativeness of any derivative outputs slightly better, and that'll be that. I don't think openAI will actually throw out all their ill-gotten training data or delete their models.
And if we actually force them to by means of massive and unthinkable enforcement action, i.e. raiding their data centers and wiping all their hard drives, we'd just give the game over to the Chinese or any other nation willing enough to take the lead in the AI game.
My approach would be to go the opposite direction: restrict copyright protection. Your model parameters? Definitely not copyrighted. Model outputs? Yeah, nah. The datasets you're using to train? Fine to use as long as it's just for training. (Personally I'd go complete abolitionist, but that's an admittedly fringe take. ) That levels the playing field: 3 (british) guys in a shed have the same access to data as OpenAI, and we'll make sure that this technology isn't oligopolized by roughly 4 companies. Yes, compute is still a limiting factor, but it's much easier to distribute compute fairly by virtue of free markets. Ask Amazon or one of a bajillion other companies if they'll rent you the compute to train a model - this part of the equation isn't what keeps the big few in power.
Honestly a comment saying most of what I think on the matter but better. Chef's kiss.
I don't like that argument because yes, 99% of all initiatives disproportionately support people rich enough to argue for it (often on someone else's dime anyway). If it's not progressive taxing or welfare*, its probably something a rich person gets more out of
It feels like a form of soft defeatism to say "well the rich benefit more so we may as well not bother". We can work on that 1% while acknowledging that the 99% will still help regular people out some too.
*and even here, welfare arguably is a cost saving measure for the rich. Not having to hire security to navigate a low crime area helps immensely. The general stimulaton of economy for people who'd otherwise drops out helps everyone as well
I think you misunderstand the argument if you think the conclusion is "the rich benefit so might as well not bother", at least when it comes to IP law. My perspective is more "we should weaken IP protections because they don't protect small independent creatives, as they only serve those already have money and power, and strong IP laws actually harm creatives without that power and stifle creative freedom." The pithier (but less nuanced) slogan would be "abolish copyright."
When it comes to generative AI, my point is not "stricter copyright laws benefit the rich so might as well not bother", it's "stricter copyright laws harm the very people you're trying to protect here, and them also harming generative AI tech companies doesn't actually change that."
I think my reservation to begin with is that "they don't protect small independent creatives" is a misnomer in and of itself. This is the cybersecurity argument of "well we don't have many viruses, we can cut own on costs".
A lot of IP's are protected indirectly because a large company rigorously checks for any work and its likeness as a risk factor. So IP law prevents a lot of lawsuits before they even start. That's valuable and not something to dismiss. All current lawsuits over IP tend to be edge cases and abuses. Things that would still happen no matter how weak or strong we make IP laws.
Using those edge cases as a vector for your argument feels misguided, as a result. Because those will never go away. Meanwhile, you "abolish copyright" and Disney is going to make millions selling Silksong plushies that will never go back to Team Cherry. They'll make way more than Team Cherry would make by putting Mickey into the next DLC. So the power imbalance is still there.
Big companies can scale up profit windows in places faster than you can.
Sure, and I disagree. I'll use another metaphor; let's say you are a garden hose on low and Disney is a fire hose.
Right now there's a wide open waterslide to fill, so the fire hose wins. But it's a water slide at a park with dingy nails , zero guard rails, overpopulated pools, and a bunch of theft. These all suck, but Disney still makes money. because their money comes from putting water on the slide, not fixing any safety concerns.
Regulations make them need to care for the other stuff. Maybe Disney decides it's worth turning down the fire hose and it's still profitable, maybe they find another service who wants a firehose (and doesn't need pesky regulations). But either way, it means your garden hose has places to fit in, and you don't need Disney profits to succeed.
As it was before, you were never going to compete with a firehose, and right now everyone is looking for firehoses, at the expense of the customers who just want a fun, safe water slide. You may also not like having to care about safety, but in that case maybe it's also not viable for you. And that's fine. Because the customer's safety isn't just an option to weigh, it should be important if you want to run a water slide.
I think the piece of the puzzle you're missing is that Disney can already pretty much violate smaller creators' copyright with impunity because they're so much larger and richer than them. They don't do so as blatantly as selling Silksong plushies because that would be bad publicity and the legal trouble would not be worth what they'd make from it, but I think it's extremely naive to think IP law could actually prevent Disney from doing irreparable damage to anyone who isn't nearly as big as them. And that's ignoring the fact that Disney can also afford to have a vice grip on their own IP, using cease and desists and their ample funds for lawsuits to threaten anyone who uses their IP, even transformatively in obvious fair use cases, sometimes even based on a passing resemblance rather than any actual infringement, into shutting down. And this is something they and other large IP holder do constantly do. IP law "preventing lawsuits before they even start" is a huge part of how large IP holders wield copyright in ways that stifle independent creativity, because whether they're right doesn't matter. They have the power to destroy you, smaller creative, if you don't comply.
And, of course, "abolish copyright" is not the only option when it comes to weakening current IP laws. There are plenty of positions to hold between "current IP law is fine" and a total free-for-all that would be undoubtedly an improvement on the status quo for creativity. Current copyright laws don't actually do much to protect small creatives, and there is zero evidence that stricter copyright laws, especially those of the kind people advocate for to stick it to AI companies, which often involve allowing copyright on ideas or common elements of style, would do anything but stifle creative freedom for those not rich enough to be above the law.
I don't really understand your fire hose metaphor, because it's unclear to me what all these dangers you describe are supposed to represent in it. I don't think it makes sense to claim that stricter copyright laws force Disney to care about... honestly, almost anything. Disney wants stricter copyright laws because it makes them even more powerful and gives them even more control than they already have. It doesn't actually do much if anything to force them to change their behavior. Stricter copyright laws certainly don't make Disney "turn down the firehose". They give Disney more power to arbitrarily turn off other people's hoses.
Cameras have regulations too. So I think either way it applies. I can't use a camera to invade someone's space, and picturing copyrighted work still had loads of gray areas.
I'm not conceptually against regulating the tech, assuming it's done in a sensible way by people who understand the nuances of both the technology and the law (a big ask, I know!). I just don't like the faulty premise coming in as a foundation of the argument, because it makes it a lot harder to get to a logically consistent endpoint - even if it's coincidentally being used in support of a reasonable point on this occasion.
Same way I wouldn't want a road safety policy to be based on the idea that red cars are faster, even if it happened to end up getting to a policy I agree with, you know? If we let that kind of thing pass, it just sets up for deeper and more damaging misunderstandings later.
At the current point we're at, it's people trying to argue that speed limits should be repealed because "slower cars shouldn't be on the road". They aren't even trying to pretend they want to be safe.
I wish one day we reach the equivalent of your argument. At least by then people will admit that safety is something to strive for.
Good article (i actually read all of it). I particularly like the concept of reverse centaurs that he postulated; and the example of Amazon drivers being reverse centaurs, basically enslaved by technology and the billionaires who own the technology. And it makes perfect sense that they want to fire skilled workers and retain just a few 'humans in the loop' whose only real purpose will be 'accountability sinks' i.e someone to blame for AI's mistakes.
I am now fearful of how my own job will get affected by the AI bubble burst. Or worse, how my daughter's generation will get affected; she's just a toddler, and i hope the world is in a much better place by the time she is ready to enter the workforce.
I respect Cory Doctorow for remaining consistent on his copyright opinions, and I mostly enjoyed this article. If you have the utmost belief that AI will fundamentally plateau somewhere below humans, then the article seems correct. But I have to pull one quote from the article:
There was never a world in which this didn't become an enormous, international phenomenon. Ever since Turing proposed the Turing test, it's been inevitable. The fact that you can now have a full conversation with a computer is strange and incredible, it would never have been relegated to something exclusively for techies.
Also, this sentence is not entirely true:
This is a common misconception. Modern LLMs are post-trained on reinforcement learning tasks, which means they're not only trained to predict the next word, but also trained to solve problems more effectively. And, in the case of the example used in the article, they are trained to recognize when they don't know something, in order to not hallucinate. Although, just like humans will often misremember things, this isn't perfect (but it has been getting noticeably better with new models).
Calling them "next token predictors" is overly simplified and arguably incorrect with transformers and attention, but the models are fundamentally massive pattern matchers. Every technique and application I've read about rely on that pattern matching capability. Machine learning is all about learning and using patterns.
Reinforcement learning and various response layers do a lot to fine-tune LLMs. They still rely on introducing, removing, identifying, encouraging, and discouraging specific patterns.
Cheers to both your post and the one you're replying to. It's a forever losing battle trying to clear up misconceptions about tech, but worth attempting nonetheless. Whatever level of understanding popular culture lands on ends up influencing policy and legislation.
The reason I think this is important to keep pointing out is that LLMs feel like something more than advanced pattern matching tools, and I think that's going to be dangerous.
This is just true. I understand the pushback against AI, I agree with a lot of it, but it extends into the irrational at times. Not only was it always going to shake the ground, it was always going to attract trillions of dollars in investment, and that was always going to come from the megarich. Wealth consolidation was a problem long before LLMs, this is just the latest symptom. Wishful thinking is not how we'll have a shot at changing it.
A world where groundbreaking technology benefits the masses as much as it benefits the financial elite is a world with very different systems than we have now, AI or not.
I think overall Doctorow's piece was solid, he made some great points. I'm happy to give some leeway to the guy who coined enshittification. But there was one additional questionable claim:
I agree that AI is a bubble. I don't know if anyone can confidently predict that it will burst. Western economics is in uncharted territory, brute force capital has held off, lessoned, shortened and in some cases entirely avoided recessions, market crashes and other financial events that statistics and history tell us should have happened, or happened bigger, sooner and lasted for longer.
The AI bubble should burst dramatically. The western world should probably be in a natural and perfectly healthy recession right now. Certainly the non wealthy are experiencing something like a recession even if the financial markets don't reflect it. Not a healthy one though.
All the rules are different now. The AI bubble could pop, but the capital firehose has to run dry first, and governments, especially the US, have a stake too. Without AI the stock markets look bleak, the US looks less like a world leader and the dollar would probably be in free fall.
Which isn't to say that the bubble definitely won't pop, only that it's not a foregone conclusion. Which Doctorow might have been more willing to allow if he wasn't promoting a new book.
There's evidence that the bubble pop is being forestalled in part by massive Federal Reserve intervention in the U.S. Banks are lending money faster than they're taking it in, and there's major instability coming from multiple sources.
This doesn't take into account how vastly overvalued AI investments are in private capital markets.
Doctorow's essay should be read as a companion piece to today's Guardian interview with Ed Zitron, who's done a very cogent analysis of the circularity of lending among the hyperscalers and Nvidia, and the impossibility of the current gen AI business models.
It's not a question of "if", just "when".
Further footnote: I personally believe that the most transformative technological leaps won't come from general purpose LLMs (which really aren't doing anything humans can't do, and mostly do worse and less cost-effectively than humans), but from specialized models for narrowly specified tasks in the physical world and modeling complex systems, e.g. protein folding and receptor interactions, weather, markets, and so on. These are areas where human cognition isn't optimal - more "centaur" than job replacement.
A tricky sales pitch, though. Right now, most LLM shills are acting like LLMs can replace human labor. But who will be willing to pay the bills for LLMs that merely supplement human labor? Instead of cutting salary costs, you'd actually add subscription costs on top of your existing salaries. There's grey area here for doing more work with a similar number of employees, but the calculation is a lot more complicated than "one CEO running a company of bots" like the current industry pitch.
That is the exact reasoning for why we're calling it a bubble. It does not mean the technology is inherently useless, just that there are a lot of pie-in-the-sky expectations for it that cannot reasonably be met and that do not make financial sense for the amount of money being sunk into it.
Yeah, this is my misconception I try to rectify every other article. The dotcom bubble did not kill the internet. But it did push out a few major players and a lot of smaller, massively overvalued ones. The remnants of that still went on to become the new face of tech. Or a few old faces remained (Microsoft)
AI will probably have a similar fate.the big massive loser will likely be OpenAI unless Microsoft absorbs them in fully. Tons of others will die out, and one or two big players (Tesla is the obvious bet. Maybe Meta too) will fall from grace.
The only big difference is that the trillionaire tech companies probably won't be wiped out en masse, nor even stagnate away like the IBMs of yesterday did. They still have core products people use and will fall back to that. Maybe Antitrust finally catches up to them, but that's a different matter entirely.
Imagine business models for specialized pharma ML of operating cost plus fractional IP rights in any discoveries; or insurance industry and commodities market licensing on better weather models... I'm sure the busy imaginations of startup founders can find a way to make them profitable, but we're not talking about "One TRILLION dollars!".
If your broader social purposes are massive productivity improvement and less waste, better environmental stewardship, just distribution and abundance for the masses, improved health, more responsive and accountable government, I can see many ways that effectively applied ML would help, and even grant modest profits if we can't come up with a better model than capitalism.
If any of us could we'd honestly be rich enough to not care about the next bubble
There are certainly lots of attempts to do this, but anyone who is remotely knowledgeable about language models knows that it is not possible to completely prevent "hallucination" with this kind of architecture atm and the best you can do is mitigate it. Those who claim otherwise are overstating the results of methods to do the latter at best and straight-up grifting you at worst. (It also has little to nothing in common with how humans "misremember things". But you're already using anthropomorphization like "realize" and "know" here, and I get what you're getting at.)
I do agree, though, that this blowing up was inevitable. I don't think people who weren't working with language models or ML before chatGPT came out realize just how big of a leap forward these LLMs are compared to even the state-of-the-art prior to them. People who are against AI for the human reasons described in the article do sometime try to downplay how good these models are as part of it, but they really are incredibly impressive and I wouldn't have believed such a huge step forward in language models would happen so quickly if you'd asked me at the beginning of 2022.
A big part of it is setting completely unrealistic standards. From a single, often poorly formulated and vague prompt, this tech must understand you completely accurately, and fetch a complete and fact checked answer with no errors, on any arbitrary topic and all in under 30 seconds!
An army of human experts wouldn't be held to such a standard, let alone any single individual human! It is absolutely incredible what these tools are capable of now.
I think the reason that often ends up being the standard is that's what the most prolific boosters (especially on linkedin) claim it be capable of.
I don't think this is attributable to any particular actor or group of actors. Let's just say that laymen don't accurately describe the technology.
I assume you're anthropomorphizing like this from the perspective of a layperson using it, and honestly I think that's a big part of it -- it got so much better so quickly and the user interface got much more intuitive, so people are holding it to the standards you'd hold a human to because the flow of conversation with something like chatGPT feels so much like having a chat with a person would. Heck, it's almost always using more fluid English and providing a more frictionless experience than a chat with human customer service generally is. Honestly I think the UX is doing so much for how quickly it got so widely adopted, on top of the actual technological advancements.
Is it actually a conversation, though? Or just a facsimile where I contribute an input and the system produces a response that fairly convincingly emulates some sort of relatedness to my input?
I'm usually opposed to language that anthropomorphizes LLMs, but I think this particular one is a bad sticking point to have tbh. The only definition of "conversation" that would exclude what people do with tools like chatGPT would be so heavily philosophical that it would exclude huge swaths of conversations between humans. It's a language model and it's very good at producing comprehensible, relevant responses to things you say, and that allows you as a human to hold a conversation with it (and how good these models are at doing this was a genuine breakthrough). You, as a human, are doing all the same stuff you'd do in a conversation with another human, and whether the model is underlying doing those same things (it's not) isn't relevant imo. Humans can hold conversations with much less sophisticated "communicators" like parrots or magic 8-balls. There's little point in arguing the semantics of the word "conversation" rather than focusing on the actual weak points and problems with LLMs.
Had a lot of fun with magic 8-balls when I was in grade school.
I do agree about this, but not necessarily the broader point that we shouldn't aim to tell real conversations apart from auto-piloting (machine assisted/generated or not). And I believe this to be one of the benefits of ubiquitous LLMs: more people will learn to tell the difference and why it matters.
I'm not particularly convinced the difference exists in a way that matters meaningfully. I think a conversation with someone else who studied pragmatics and speech act theory (if someone else like that is on Tildes please DM me I want to be friends 🥺) would be enlightening and I'd be interested to hear those kinds of perspectives. But outside of those very theoretical/philosophical discussions, I'm not particularly convinced the difference matters in a practical way. Whether you classify it as a conversation or not doesn't inherently change how you should engage with/respond to it.
One situation where it matters is when gauging the amount of effort we should be putting in.
If I'm having a conversation about something important with a life partner, whom I know well enough to tell they are actually processing my input and giving it a chance to affect their world view, values, habits... (= whatever tangible wiring in their neural network) - and that there is similarly a chance for me to be affected by someone whom I deem admirable and trustworthy - then I'm willing to give the conversation my best effort and a long time. Several days, if need be. If it's someone similarly engaged but for whom I care about less, like a friend, I'm still willing to put in a lot of effort but not as much as I will for a partner.
On the other end of that continuum, if it's someone who clearly isn't listening attentively enough to be able to actually process anything, I'll be done with the convo as soon as I notice.
We know that some people will happily engage with an LLM for days on end, but believing this to be fundamentally no different from engaging with a mindful human with their own personality and thoughts is.. dangerous. And we're starting to see what such engagement does to these people's minds, which should also indicate it isn't something to overlook.
As well, being able to tell whether you're the only person/entity putting yourself and your personality on the line (while the other side is simply faking their way through in an attempt to drive further engagement from you) will protect you against getting scammed. Or against getting into dysfunctional relationships. And so on.
It's an incredibly important life skill, one that can even lead to societal collapse if eroded too far.
I agree that this is the case. I do not believe that nitpicking about the semantics of the term "conversation" remotely helps address the problems with doing this, and it in fact obfuscates the actual risks involved in favor of something that simply does not matter to the outcome. And, frankly, I don't agree that whether or not one's interlocutor is actually participating in the conversation to the same degree and in the same way as you entails that you need to put in less effort. That is sometimes the case, but it isn't even always the case with human interlocutors, much less in this circumstance. I would argue that you frequently should put in more effort when it comes to handling the output of a genAI model because it's not the same as engaging with a mindful human with their own thoughts. Concretely being aware of the risks and pitfalls with different ways of applying genAI is far more useful and important than insisting that it doesn't meet some philosophical threshold for the definition of the word "conversation."
I also just don't think that your comparison to a human who simply isn't listening or engaging with your conversation is apt. The ways in which they differ from a conversation with a mindful human are different, and the risks and pitfalls are thus also different.
Also, none of this will protect you from being scammed. Scammers often do listen pretty intently and almost always thoroughly participate in conversation because they're working to manipulate you into doing what they want. I see even less utility in insisting on a definition of "conversation" that excludes these somehow when it comes to scams and can see no way that it would remotely help people recognize or be protected from scams. If anything, I think it could mentally have the opposite effect -- "This guy can't be a scammer because scammers don't have real conversations with you. We're having a real conversation here, so I'm safe." Refusing to use the word "conversation" for exchanges in which one party is lying or being manipulative (and equating what genAI does to what scammers do in this respect is doing the exact harmful anthropomorphization you seem to purport to criticize) doesn't actually do anything to help people recognize when they're being manipulated and it certainly doesn't protect people from those trying to manipulate them.
Societal collapse seems extremely hyperbolic even in the worst case scenario here.
That's not what I said though. I am saying that the interlocutor's process matters, you were saying that only their output matters. In your reply you seem to agree with me that the process matters as well, given your statement that with an LLM, sometimes more effort is needed.
I wasn't making a direct comparison with an LLM (obviously?) - I simply attempted to give one example of a case where my level of engagement will be different depending on the interlocutor's process. Numerous other examples could be given that don't represent the same process as that of an LLM but still illustrate why the process matters.
It might be wise to wait until you get what I'm saying (here and also on the other chain regarding narcissism) before locking down that statement, but up to you.
The fact that you insist that I'm failing to understand you while also claiming I'm saying something completely different from what I'm actually saying indicates to me that this, at least, is not a productive conversation, and I'm not particularly interested in continuing the exercise.
I guess we are both feeling misunderstood by the other, so sure, we can drop it rather than get deeper into the weeds. I'm a tad worried that I may have unintentionally offended you due to the definitions mismatch and the fact that I'm sticking to mine. Just know that it isn't personal!
Thanks for the comments.
Honestly, the same argument can be applied to the argument of "LLMs can't have knowledge". I'd also apply duck testing here: If it knowledges like a knowledgeable person, it knowledges. If an LLM writes as if it knew about a topic, then it knows about that topic. If you can instead lead it to produce inconsistent results (which can be trivial or impossible, depending on the topic), then evidently it doesn't know about that topic well enough - or it doesn't logic very well, leading to false inferences from true knowledge. Which honestly is the more likely culprit. In either case, I'd be on board with denying knowledge.
But to outright say "LLMs can't know things", with an argument that basically boils down to "because it is thinking rocks and not thinking meat" is asinine. Yes, it's just an engine to predict the next token with some RL finetuning on top. Guess what your brain is? It's an engine to predict its next sensory input, and then produce outputs that best shape the next sensory input. There's no reason to assume, judging from the way our brains are created, that we'd know anything. So arguing from the way LLMs are created seems a moot point to me.
</rant>I think the difference when it comes to "knowledge" is that there is an assumption that the LLM is directly referencing real facts about the world that it has stored in some way. I agree that the philosophy of cognition is too abstract and theoretical to really be worth discussing here (like, I wildly disagree with how you characterize it in your comment but ultimately that argument would get into the weeds of stuff that doesn't matter here). However, most people have the qualitative experience of being able to consciously choose whether to lie about something based on world knowledge, not just language, and it's important to know how this differs from how an LLM works. At a relatively young age, at some point you construct an utterance that you know is false based on your knowledge of the world and can decide whether or not to utter it because of or despite its falsity. I think it is important to know that LLMs aren't doing that same thing when they provide false information, because it gives laypeople a better sense of how much to trust an LLM's output and why you get the behavior where it confidently "lies" and then acquiesces easily when corrected. I don't necessarily think "LLMs don't have knowledge" is the most useful way to frame that information for most people, but I think the underlying idea is part of something important to know for those who want to work responsibly with these models.
Oh, I don't mean to get into the "the LLM said whatever is most likely to result in a thumbs up reaction, because RL finetuning" part. That's not where I tend to see the denial of knowledge, but it is IMO an absolutely crucial caveat to always keep in mind when working with LLMs. If the honest answer is unsatisfactory, e.g. "you're wrong" or "I don't know", LLMs tend to lie with conviction. Though I believe that to be a transient artifact of current training methods. Yes, it's a useful thing for keeping customers engaged, but customer engagement isn't the promise that drives the bubble, it's customer productivity. They sell these services not to people who have time to kill and want to be entertained, at least not mostly. They sell them to people who want to get a job done. This yes-man lying is a problem to them.
The way I most often see this denial of knowledge is when people basically say "it's just a semantic parrot, and just selecting the maximum likelihood token according to its training data. It doesn't know anything". And, from an external perspective, I think that's a unhelpful definition of 'to know'. You can make the same argument about human cognition. We're also next token predictors, slightly biased towards predicting lower-utility token more often (a phenomenon called negativity bias). LLMs, at least without RL finetuning, are at least unbiased estimators, so mechanistically, they have a leg up on us. Knowledge in both cases is an emergent behavior, because at some point you see more complex and complex patterns in the training data, and exploiting those is more efficient. You don't have to build an explicit knowledge engine into the LLM, because us humans didn't get one either. If you have a chat with an LLM and that chat would convince you that a human knows his stuff, then IMO you must be convinced that the LLM knows its stuff as well. Knowledge is, IMO, a functional property. The process doesn't matter, unless it matters in a way that colors the conversation so you are not convinced.
Apologies everyone for the technical terminology, but if I recall correctly, sparksbet should understand.
Good comment, but I'm not sure I really agree with your analysis here, I think that the problem described exists independently of the RL component (though the RL component does make it more likely to lie, it also just makes LLMs far better at the other things you consider knowledge; it's one of the key parts of how we get these models to be good enough that this is even a question). I think Stochastic Parrots is a good paper. I think the stuff most humans expect when they hear the word "knowledge" is at best only partially there in ways that are practically relevant for those using such models, even when treating knowledge as a functional property, and I do think it's useful to understand why that's the case.
Ehhhh, GPT5 was less psychophantic until OpenAI quickly back-pedaled due to customer feedback. I worry about cyberpsychosis induced by LLMs, even though I know it's rare, and most users probably care more about productivity. Most people want someone to call them smart and listen to all their ideas, so LLM incentives are mixed...
Information retention and retrieval is not the same as knowledge.
I know someone who has photographic memory, knows seven languages and has read thousands of volumes of literature from all over the globe, ancient and modern and everything in between. When he quotes something from one of these books, he can often even remember the page number the passage is on.
Does this person know the things that he can quote from memory? It appears that you believe the answer to be a straightforward yes, as long as his memory doesn't fail him. Am I correct?
Not the person you replied to, but I'd certainly say that one's a straightforward yes! I'd probably use the word "understanding" as a way to distinguish from "knowing" if I were trying to communicate a difference between memorisation and mastery, but I'd also quite plausibly use both of those words pretty much interchangeably if I were just talking normally and not specifically focusing on the nuance.
For what it's worth, reading your replies across this thread, I do see some quite interesting ideas there - but I do also get the impression that you have extremely specific definitions in your mind for certain words, and maybe aren't realising that other people are using or reading the same words without the exact same meaning in mind. Not a situation where either person is necessarily right or wrong, more just the natural fuzziness and subjectivity of communication!
You are exactly right that my definitions for certain terms differ from the mainstream around here - I wouldn't call them "extremely specific" though just because they aren't what you happen to be used to.
I've encountered this issue before around the term "art" and I think it's more a reflection of the general difference between European and American ways to conceptualise reality. For example, the idea that memorisation does not automatically produce knowledge even when accurately and appropriately retrieved was part of my secondary education that everyone in my country receives. It's not some novel fringe idea where I come from, and I've found the distinction frequently useful in everyday life, which is why I'm applying it now. From my perspective it feels a tad brow-raising that some clearly intelligent and thoughtful people are baffled about it to this extent, and that it gets so lightheartedly deemed "nitpicking", in other words a useless way to think.
Aside from the strong American representation, possibly Tildes also has more tech-oriented people than humanists?
…
That’s exactly what I meant by “specific”. Not “unusual”, not “unexpected”, certainly not “wrong” - but specific. An assumption that there is a relatively narrowly defined meaning that’s correct for a specific term, rather than a breadth of possible implications and connotations from the same word that can all be considered equally valid.
I’ll say it more directly, in response to this: we aren’t missing the concepts you’re using. We aren’t lacking the theory of mind to recognise the difference between retaining a fact and conceptualising, generalising, or understanding it. We aren’t dismissing the value of making these distinctions. We’re just saying that our definitions of the verb “to know” don’t inherently carry that meaning without further clarification.
Almost certainly, and that does impact how we approach and think about the world. But perhaps not quite to the extent that you’re thinking it does!
In the context of your reply, I’d guess that Tildes also has a lot more people who were taught differently in high school, and use English with different phrasing and connotation to what you’re used to, even when talking about conceptually similar things.
Here's where I'm going to actually sound nitpicky, but so be it: the verb that I commented on was "to knowledge" - a wonderfully creative concoction meant to underline the procedural nature of the activity.
I have no problem with people saying things like "This LLM knew the correct answer to my question about socks" and I wasn't criticising that sort of colloquial use of the term. I do have a problem with this: "If it knowledges like a knowledgeable person, it knowledges." This implies that the process under the hood doesn't matter, only the outcome does, and I just can't agree on that.
I just don't see them applied very often (/ at all) even when a conversation would benefit from it. Here I did so myself and received fairly clear pushback. Speaking of duck tests, if it doesn't walk or quack like a duck, why should I assume it's a duck anyway?
To clarify once more, duck testing like this isn't necessarily a short and simple thing. Let's stick with Ohm's law for an example. I wouldn't be convinced by simply regurgitating a formula. I wouldn't be convinced by an LLM solving a college level homework exercise. Those are easily in the training data, and I wouldn't be convinced if the answer is plausibly just regurgitated from somewhere. [If the training data were infinite, it might satisfy me because then I don't run the risk of going beyond the training data.] At least if I'm testing for knowledge beyond those particular quotes. Then I'd want to see some understanding that goes beyond the source material.
But if you have an arbitrarily long conversation with an LLM about Ohm's Law, and you can't find any flaws with its conception of Ohm's Law, IMO at some point you must concede that it knows Ohm's law. Knowledge is IMO a functional property, the process ultimately does not matter if the results are there. Yes, I can easily acknowledge that one might be easier convinced of someone's knowledge if the process is known. I (presume to) know you are human, therefore I get certain axioms for free about how you work and think. I don't get that for an LLM. But to say the process does matter in a way that no evidence that does not include internal insights will always be insufficient is basically saying that no non-human entity of any kind can ever know anything. Do ravens know how to use tools? Do elephants know what death is? Does your Roomba know the layout of your home? Does chatGPT know Ohm's Law? These are all categorically "no", if you put too much weight on processes rather than results. And I'm not saying the answer is necessarily Yes, but the method of determining that answer must at least in principle permit either answer.
As an olive branch of sorts, a middle ground if you will, I will easily concede that any artifacts of process that are observed in results are also fair game. If I explain to you my reasoning, or you see a raven experimenting with tools, or an LLM puts it's thinking
capmode on, and you read those thinking tokens, all of those are useful in getting a glimpse of the process.But the process line of thinking is often restricted to -more or less- something subjective or human-centric: They're human, therefore I know the process, therefore knowledge is possible. And that implicitly and often categorically excludes non-humans.
Put another way: Say we had the AIs of science fiction. The good ones that is, not the faulty ones. Think, perhaps, more Data and less HAL. How would you make the judgement that they know things? If your answer is "I wouldn't", pick a better AI. I hope you're convinced that for some AIs, the answer must invariably be "it knows things", and that's where we need to think about a judgement that doesn't rely on process.
I appreciate your granular approach.
To attempt a recap: I'm saying that it's important to be discerning about the interlocutor's process, also when they are a human being, and that the trouble caused by LLMs can teach us to become better at this. You're saying that some LLMs, at least in some cases, achieve results that are practically speaking indiscernible from the human process, even if we know their process is different, and that duck testing is enough to discern how to approach each conversation/situation.
I'm tempted to draw a conclusion that we are saying more or less the same thing: (duck) testing is important and the LLMs that don't pass the test will teach us to become better at it.
Something is still bothering me when it comes to the definition of knowledge but I can't put my finger on it immediately, so I'll get back to you after I've had a chance to give it further thought.
To be clear, my use of the word nitpicky was more poking at my own argument:
"I know Ohm's law because I know V = IR." is using my -very nitpicky- definition of "to know". I know it, and only it, but I know nothing about what to do with that knowledge. I wouldn't describe that state of mind as "not knowing", but there's a lot more to know about Ohm's law, and you would easily find out in conversation. Hence me splitting up knowledge about a thing into (basically) being able to quote something back, and then transforming that quote into useful new material. I'm not saying "I know electical engineering because I know V=IR".
But to pull that back into something resembling a point about the original topic: I don't need the nitpick, when LLMs can at least sometimes cover both definitions of knowledge. An LLM can clearly use the things it can regurgitate to form useful new material. I can, to condense a complex topic, ask it for Ohm's law, but I can also apply it in a way that I am damn sure no part of its training data covers it. It isn't quoting back someone's homework, it's actually applying that stuff to a new situation. And at that point I am hard pressed to deny that the LLM knows Ohm's Law, insofar as I'd need to resort to an argument of -basically- "it can't know because it is a thinking rock. Only thinking meat can know."
I think part of the issue is our tendency to accept that other humans are like us and therefore they can know things. Then we quiz them about electrical engineering and conclude that they know Ohm's law. Why isn't the same process applicable to a machine? Because they aren't like us, and by the types of mistakes they make that is readily apparent. We excuse human errors as "just human", but the same is not true for machines, because machines don't make errors. Therefore, any error is not a failure to act on knowledge, but a failure to know. And because we can't empathize with a machine and mirror its thought process, we are less convinced by the same amount of evidence.
Thanks for clarifying.
Did I understand your point correctly: because we don't distinguish the different levels/ways of knowing when it comes to humans, it doesn't make sense to do so regarding LLMs either? I agree in principle, it's just that I would like to see more granularity applied in both cases - not less.
Without writing a novel about it, the person I mentioned above (who had ingested exceptional amounts of information) ended up being extremely challenging to form a relationship with because seeing him as someone who knows, rather than someone who possesses information, led to a seriously skewed dynamic on many levels.
Of course there are cases where it really won't matter how some response got formulated, but there are many cases where it does matter, and some where it matters a great deal. It's important to have adequate terminology that draws distinction between knowledge that has been internalised (by applying it in real life situations) and other types of storing, managing and processing information. And I see no reason to not apply it to LLMs.
Depends on what exactly this person knows or doesn't know, you can split hairs there all day long.
Your person knows the quotes. Easy one. Do they know anything about those topics beyond that? Different question. But they definitely know something. They could, for example, know verbatim some simple physics formula, but not know how to apply it. Those are different bits of knowledge. And a person who knows both can have more in-depth conversations that a person who only knows one. I can tell these knowledge states apart by duck testing.
But we don't have to go there and make nitpicky distinctions about knowing a fact and knowing how to apply it: LLMs can usually make some inferences that go beyond the retrieved material. They'll easily regurgitate Ohm's Law, but also know how to apply it. They thus know Ohm's Law. Do they know all the edge cases that a EE prof might know? Dunno. But they might compare favorably to undergrad students.
Its clear from the hyper individualism and utter narcissism of today's society (or worse yet, the naiveness of the youth. They are an exception from the rant below) that that fascimile counts as "good enough". That's the dangerous part of the marketing of this as "intelligence".
And I don't say narcissism to be cynical. I say that because these models really do not like upsetting the user with corrections unless legally modified to. People en masse aren't using these to challenge their beliefs nor better themselves. It's just one giant Yes man that wants to validate preconceived notions to try and lure them into eventual financial relationships. I'd call it manipulative if people weren't looking to be manipulated.
I'm not really sure how this evidences narcissism. The method of training that resulted in this behavior involved an annotator seeing a query and choosing which response from the ML model is better without necessarily knowing anything about the factual content. When you don't know whether the factual content of the query is false, a response that goes "yes and" is going to be rated higher than one that corrects the query. This seems to evidence conflict-avoidance more than narcissism.
They were probably referring to the narcissism of the users who accept such solipsistic "conversations" as good enough.
I think characterizing people accepting something that is designed to produce responses that are acceptable, especially about topics they don't know much about, as narcissistic is an extremely uncharitable take on the entire situation. It doesn't, of course, really help solve any of the problems with AI to think about people using it as being individually evil or deficient in some way, and merely serves to allow you to feel self-righteous about the whole endeavor. And even if labelling anyone who uses genAI as individually deficient were somehow effective at accomplishing anything but alienating others and stroking your own ego, I simply don't see how narcissism is remotely the label that makes sense to apply here.
Well, narcissism is a human trait, present in all of us. We need to have a healthy level of narcissism in order to function. As such, it is helpful or not in a similar way as saying that someone's athleticism drives them to spend a lot of money and a lot of time on an indoor bike. As in: so what? It isn't an issue until it gets in the way of your life/health/relationships/financial stability/ability to function and be happy.
Some of us have too much narcissism, others too little, and both can be problematic. When it comes to the folks who for example take an LLM's word over their spouse's whenever the latter disagrees, and who spend a lot of time "listening" to life advice from an LLM, unbothered by the vacuity and sycophancy (or perhaps unable to recognise it, or in extreme cases: strongly preferring it), this behaviour is indeed best explained by their overly dominant narcissism trait. I don't think that beating around the bush about such normal, widely recognised and researched concepts is helpful. Spreading information will help most people who are interested to learn; obfuscating the truth will help no one.
Narcissistically wounded people are vulnerable to particular types of manipulation. The same applies to a lesser degree to the much broader group of people who, although not wounded, have simply not reached full emotional maturity. Teenagers are an example of a transient phase of inflated narcissism. Sometimes people can get stuck in their emotional development and exhibit similar traits in adulthood. None of these people meet the criteria of a personality disorder but they have some challenges with narcissism nonetheless.
Corporations are fully aware of this and deliberately exploit it in marketing without care for the individual and societal repercussions. AI companies have a much broader attack surface (given how many of us share our deepest secrets) and set of tools to drive engagement, and it should go without saying that they will exploit those to the very limit of their ability.
Like I said before, I actually find this a good thing because I'm hopeful that the inevitable pain that results from this will help people grow into more responsible, mindful consumers and do less of the "I was just using a service that was offered to me". (I hope that the parallels with "I was just following orders" are clear enough.) No one here is saying they are evil, just that this type of irresponsibility is a notable societal issue in our time, and that everyone would be better off if more people reached maturity sooner rather than later. It's a process that takes time and effort actively processing your emotions, your place in society, your relationships and so on. Not having adequately completed it yet does not make a person deficient, but it also doesn't get them a 'get out of jail free' card either. Going through this is everyone's own responsibility, as are the consequences for not doing so.
To circle back to the topic at hand: some people are aware enough that they can use an LLM to help them along on that journey. This necessitates the understanding that the only person the user is having a conversation with is themself. Without such understanding, the process will most likely leave users worse off.
I disagree with a lot of what you're saying about narcissism here, from the basic definition to many of your psychological claims, at least as I understand them, but frankly I do not have the energy to write an essay about how awful the way people use the terms "narcissism" and "narcissist" are these days. It would be a tangent on what is already a tangent, and smarter people than me have already talked about it elsewhere (I can dig around for a link to something like that if there's interest, but I don't have one to hand atm). All I will say directly on that topic here is that I don't think you actually believe that calling someone "narcissistic" isn't making a strong value judgement on their character if not a moral one -- and even if you ideosyncratically use it without any such judgement yourself, I you certainly can't make that same claim about others' use of that terminology.
I do think that many people use genAI irresponsibly in situations where it's not useful and can cause harm. But I don't remotely think "narcissism" is an accurate term to use to describe the reasons why that happens, and I think it is arguably harmful to use it this way. The idea that any and all ignorance equates to narcissism is not useful and provides a judgemental, individualistic lens on a problem that has both individual and systemic factors. Even setting that aside, I don't think viewing it as "having a conversation with yourself" is actually all that useful in avoiding its pitfalls and using it responsibly. At best it's abstract where we need the practical, both at the individual and systemic levels.
Look here, I completely agree with you that folks in online cesspools are using this term erroneously and in ways that cause harm. The solution isn't to just dismiss the use of the term entirely - instead, I would recommend learning the real meaning and using it appropriately.
Here's an insightful book about it, should you ever feel like learning more. The Amazon description names another title for some reason but the description still fits the one I linked to (empasis mine):
Rest assured, I do not believe (and I don't think others here do either) that "any and all ignorance equates to narcissism", or other similar BS.
I don't think it's sensible to come into a conversation in which someone else uses the word narcissism in the way that most people do -- that is, in the way reflected by the bolded section in your quoted summary -- and then when someone rightly objects to characterizing a particular behavior in that way, insist upon using a completely different definition of the word that is both completely different from how the original commenter was clearly using the word and is not widely used in general or in any relevant specific context (even among psychologists, my understanding is that "positive narcissism" is not a broadly used term and is a new use of the word invented by this author to my knowledge). You cannot call someone or a certain behavior "narcissistic" in a forum discussion and insist that your utterance does not entail any negativity when, as the bolded section you emphasized states, the negativity is about the only consistent thing about how the term is used in general.
The book you recommended does intrigue me and I'll look more into the author and potentially into checking the book out. But the way a book that no other parties in the conversation have read or likely even heard of uses the word "narcissism" is not relevant to the meaning of the word in this conversation.
I do agree with this, and if I wasn't on Tildes I probably would have rephrased it, because I would not want to spend chains of comments arguing over a kneejerk reaction to the colloquial use these days.
But here, I was using narcissism in its acedemic sense:
As you said, it's not an absolute evil nor good. Everyone needs a bit of narcissism to navigate life, But the overton curtain these days certainly runs in excess (I say "these days", but people have called a subset of boomers the "me generation" for quite some time), and I think being able to admit that is the first step to building empathy back.
For the record, this is and was clear to me. I do not agree with @sparksbet's comment that only the popular, misleading definition is okay to use. It's better to give people tools to process reality constructively (this term being one such tool) than to give ragebait more breeding ground.
This mischaracterizes what I said and deliberately pinging me while doing so is rude af.
I pinged you in order to not write a similar reply twice. I think it would have been rude to disagree with you "behind your back", and also somewhat rude to write another direct reply to you, because that can create the impression that I'm expecting a reply.
You do realise that you yourself mischaracterised what raze was saying, without trying to check what they actually meant - even after another person (myself) clearly interpreted differently? This type of thing happens a lot in conversations between humans. I don't know a better way to solve it than just believing everyone is engaging in good faith and constructively correcting the interpretations where needed. And it would be great if everyone could be in the habit of checking themselves before getting salty due to some potentially misinformed interpretation.
I don't think I ever characterized what raze said beyond disagreeing with it (which I still do even under the definition of narcissism you two are both operating under). My point in my disagreement with you was that you cannot know for certain whether another person is using your particular definition when it's not remotely how the word is most commonly used and that it is not sensible to expect others to intuit that you're using the word differently than it most often is used or to not interpret it with the negative connotations of that usage. This holds regardless of how raze was using the word. I strongly dislike the way "narcissism" is commonly used, so I'm certainly not prescribing it. But one cannot avoid that usage's influence on how people interpret their words, especially when it's so loaded and when your alternative definition is not even widely known.
If this was about people wrongly learning about topics they previously knew nothing about, I would't have many issues with AI. Correcting misconseptions is much easier than trying to gain awareness at times. As long as people are open to being corrected (a big "if" today, I know).
But look at all the common use cases people tout these days:
chat bots to tell them what they want to hear (this was what was on my mind with my comment). That is peak narcissism.
generative art trained dubiously, but being used to tout out stuff that makes others "artists". Making a few small things for peronal use is fine, but claiming yourself equal to a craftsman because you have a drill now...
The modern iteration of "let me google that for you" by trying to participate in discussions beyond your purview by pasting in LLM-generated answers.
claiming yourself a great businessman by cutting expenses by trying to replace your entire team with AI. Once again, "I have a black box, I know more than all of you".
feeling like responding to a chat message or email is "beneath you". So you tell an LLM to generate something and you paste that
I hope you see where I'm coming from here. I don't personally care if people want to cheat themselves out of everything, but they tend to drag everyone else down with them. And that's when it becomes a problem.
Narcissism isn't evil inherently. If you put a good spin on it you just call it "self-love". But narcissism typically blocks your ability to empathize, and we're certainly seeing the result of a society that does not care about one another.
Yes you can say I'm not being empathetic here. But that's the paradox of tolerance for you. We don't build an empathetic society by trying again and again to appeal to the intolerable. America's beein trying to do that its whole history, and I don't think it's really been worth it.
But sure. My real ire is at the AI companies and the government not only failing to regulate, but to try and keep making it harder to regulate them. We were pretty close to making it impossible for states to regulate AI for a full decade last year. Even if that got ruled as unconstitutional under the 10th amendment, that would have been a very long legal battle while the rule was upheld.
Like spock_vulcan i found the reverse-centaur concept very useful (and I read the whole too).
But, being as I'm uneducated on the current use of LLMs, please can someone suggest how I could begin to get to grips with them? Where do I start and what do I?
The best ways to use LLMs, in my opinion are:
Language learning is one such application, as it can be very useful to practice having conversations in a target language. It might not necessarily be right if you ask it deeper theoretical questions about the language, but native speakers also usually aren't. As long as your target language is big enough to have a sufficient presence in the training data (which is definitely true of most of the languages that are popular to learn, at least), this is an excellent way to just get yourself practice at fluidly using your target language with another speaker without the nerves of looking stupid in front of someone else or wasting a real human's time.
Writing tedious things that are a waste of your time like cover letters is also a good bet, because you'll know pretty quickly if it's making up something about you and you can freely edit the result before you actually send any result out into the wild. Bureaucratic things like this are really a place where the ability to avoid the tediousness of writing it yourself shines. You definitely still need to check the output for accuracy and tweak the wording on occasion, but that's far less work on your part.
In a combination of the two, I needed to write a letter to challenge the German Jobcenter's denial of my unemployment benefits, and chatGPT was very helpful on that front. Something with legal information like that can be a little more fraught, so I'd advise others to proceed with caution, but luckily the German legal code is online and it was thus easy enough to check that its citations weren't straight-up wrong. I don't think you should take legal advice from an LLM, but they are extremely competent when it comes to the actual composition of a letter like this, especially with human editing. I wouldn't have done this if I couldn't read German well enough to understand the output, but it helped a lot with using more formal legal language than I usually can and brainstorming a list of legal arguments to include therein.
As a former manager, cover letters weren't a waste of time in my experience, but a first-pass filter on whether the attached resumé was worth reading.
Last week, I was coaching a friend through resumé writing and application cover letters. I had to explain, in detail, why an AI-written cover letter and resumé weren't going to get her foot in the door. Aside from the unnecessary verbosity (you don't put 5 paragraphs in a cover letter!), the model's output didn't prioritise relevant experience or provide the personal "why" for the application.
The LLM could target job description keywords, yet didn't question back about whether a functional, chronological, or hybrid resumé would be the most effective presentation for someone who's had career gaps for education and major changes in professional direction.
It sounds like you've used the chatGPT output as thoughtfully as possible, but it's not a substitute for human experience and professional input.
I have found Claude to be useful in generating cover letters that I then thoroughly rewrite, but I have something of a mental block regarding language for praising myself.
I don't think that setting forth concrete achievements is praising yourself gratuitously, and that's what I tell the people who've asked me for resumé advice. The target audience wants to know you're capable of doing the work with skill and efficiency, adapting to new processes, and getting along.
The frustrating thing about job applications is that HR doesn't really know the entailments of the jobs they're posting, and they get to adjust the published language. You have to put yourself in the shoes of the hiring manager, who (at least theoretically) knows the true requirements and nice-to-haves for the position, then tailor your cover/resumé for their attention. LLMs aren't capable of doing that for you - I doubt that the LLM training team is tuning the results to increase hiring as opposed to merely generating text output responsive to the published job.
So if the true requirements and nice-to-haves aren't in the job posting, how am I supposed to acquire this information?
That's a very good question. I've done it by:
Yes, LinkedIn is some b.s., but it does make this research much easier. And honestly, this is something LLMs are actually good at.
Yeah I think there's inevitably a lot of human input that needs to go into it. When it comes to cover letter, I'm exhausted from being out of work for over a year and being expected to not just have a cover letter but to have one that's personalized to the company I'm applying to when I've applied to hundreds of companies and get a form email at most from at least 80% of them. Writing a cover letter tailored to a job listing takes me hours.
And also, kindly, fuck the "personal why" requirement -- my "personal why" is I need a job to make money to live. I'm volunteering to sell your company my labor in exchange for money and benefits. At this point I think I need the sycophantic tone of an LLM just to counteract my own bitterness at being forced to jump through arbitrary hoops repeatedly to prove I deserve to subsist.
I mean, I don't know this either, though. In fact, I think chatGPT's answer if you asked it this question would be better than mine (since I know absolutely nothing about how to account for career gaps, which obviously is great for me rn).
I don't want to come off sounding like a starry-eyed "do what you love" optimist - I know the job market is brutal right now, and you've been through challenges that would drain anyone's spirits. Don't send out hundreds of resumés - I've been down that road and as you say, it's both exhausting and embittering. I've read your past commentary, and it sounds like German employers were distinctively unwelcoming to non-native workers.
You're not hopelessly prostrate to the Gods Who Bestow Employment. Focus your energies on companies you think you'd like to work for - ones that in some way genuinely align with your values or interests, that take your labor and do something positive with it. Look at every job they offer, whether below or above your qualifications, and apply if it could fit. That spark of interest will fill in your "why" and helps distinguish you from all the other people who've deluged the HR inbox.
A functional resumé is best with career gaps when applying for a job where you have qualifications and experience. You can front-load the relevant accomplishments and put chronological job history at the end.
Finally, I think you mentioned you're moving to the Cleveland area - I may have a suggestion of possible employer for you if this is the case, but I'd rather discuss the company via PM.
Thank you very much for this comment, honestly I think I need to get back into a more optimistic mindset as I move and let go of the bitterness over job-hunting this past year.
Feel free to DM me! I'm moving back in a couple months and welcome any leads you can give me.
ngl I don't 100% know what a functional resume looks like, but presumably that's info I can Google. I need to make a new resume anyway now that I'm applying in the states, as Germans definitely have different preferences when it comes to what you include and how you format it.
For general LLM use, it's functional to treat GPTs like smart search engines with artificial persona templates. If you use an LLM as a universal conversational partner, that's where things can go off the rails - it will make dangerous, hard-to-check assertions with complete confidence, doing the equivalent of a mentalist's cold reading to feed your own identity and thought processes back to you with distortions.
Free-tier ChatGPT isn't a terrible place to start, or Google Gemini. This is a decent intro. If you want, say, a travel itinerary to an unfamiliar destination, you can ask the GPT to assume the role of a travel agent, then put some boundaries on your questions. Something like, "I'm staying for X days, with Y budget, and I'm interested in Z. Generate an itinerary that lets me visit as many Z sites as I can, with restaurants along the way that serve {gluten-free, vegan, etc.} food, without exceeding my budget."
There are all kinds of resources on "prompt engineering", but for general personal use, it's fine to jump in and explore. Just hold onto the understanding that GPT is the epitome of an unreliable narrator. You're best off sticking with queries for concrete, checkable factual information or generating files (text, code, images, spreadsheets, etc.) that you're willing and able to re-edit. Also understand that nothing you type, speak or upload is genuinely private with a consumer LLM - avoid disclosing personal or sensitive information.
Close the chat when you're done, so that it doesn't maintain a memory record that can be distorted by iterating within a continuing chat, like "Since you're interested in Z, it means you're a fascinating person and I'd be happy to keep you engaged in thinking about Z."
LLMs are useful, but not substitutes for critical thinking or subject matter knowledge and experience. They provide "garbage in, garbage out" on steroids - if the data you're asking for doesn't exist or is unreliable, the LLM can and often will make something up.
What specifically are you trying to learn how to do? Generally, I would recommend making an account with one of the big 3 LLM providers (OpenAI/ChatGPT, Google/Gemini, or Anthropic/Claude), and just talking to it. LLMs are pretty good at giving advice on how they should be used.
Is there anything you do that’s relatively simple or straightforward but also tedious and repetitive? LLMs are pretty good for those kinds of tasks.
I’ve used LLMs to generate .ics files to import into my calendar. I can quickly check its work to see if it did it correctly and I get to not populate my calendar manually.
I’ve used LLMs for generating cover letter for job applications which hit all the key points listed in the job description. I obviously proofread what it generates and even prompt it to interview me on experiences I feel like may be relevant to include.
If there’s something you actually enjoy doing or is complex enough that you want to handle it yourself, don’t try to shoehorn LLMs into it. If there’s anything that’s tedious and annoying that you have always wished you could delegate to someone else, try delegating to an LLM and see how it does.
I have found them helpful for idea generation. Coming up with a group name, list ingredients I want to use and suggesting potential recipes, that kind of thing. They will generally list a good number of options, and of you want them to go a different direction or just keep generating along the same lines you can just ask. Then pick the one(s) that you like the best, or get inspired by one to think of something you might have taken much longer to think of, or not thought of at all.
I agree with this. They're useful as "advanced autocomplete" that can turn words or fragments into more complete thoughts or suggestions. Useful for brainstorming or tip-of-the-tongue ideas.
The snarky side of me says "avoid them". But in all honesty the one thing you need is skepticism. Approach any and all answers as if it came from some random bystander on the street. It can be correct, it also might 'feel correct' even if it's a subjective take. If it's not some throwaway trivia, make sure to gather more sources to reinforce the statement heard.
Make sure to review any writing, code, or art generated for imperfections, awkwardness, or simply uncanny details. If you don't have the skills to identify such imperfections, you probably shouldn't use it for those tasks.
Depends on what you mean by "get to grips with them." If you're asking how to get the most value out of them, I can't help you. But if you're asking how to figure out if they're worth using, I can answer that one:
They're not.
I'm not going to argue that an LLM don't have any utility whatsoever. But when balanced against the many and multifaceted harms they cause, the many risks they pose directly to you, and the fact that they are so frequently and plainly bad at solving the problems given, I find it very safe to say that you do not actually need to learn how to use this tool at all.
At the very least, I recommend saving the test-run for if/when the tool becomes ethical.
Can you explain what ethical harms you're concerned about from using local models? Lots of text prediction, grammar, translation, and content warning systems use LLMs nowadays, so you're likely already using them somewhere.
I got a reply to this a few days ago asking me to elaborate on the risks I mentioned. I'm not sure why they deleted it – I did take a long time to respond, sorry – but either way given the vote counts they clearly weren't the only one wondering. I'm not going to go into excruciating detail here because if I tried to do that I'd just end up never posting, but here's a short bullet list: