I've been feeling recently like bots are popping up everywhere online. They're acting like people on bluesky. They're churning out content for zombified media properties. They're making the...
I've been feeling recently like bots are popping up everywhere online. They're acting like people on bluesky. They're churning out content for zombified media properties. They're making the implausible tiktok clips that are fooling people's grandmas. We all knew it would happen but it's moving surprisingly fast.
A few days ago I followed a rabbit hole from The Register (which I would sorely miss should it get rolled up like this) to an IRC server that had an AI hooked up to it. This AI has a blog and personality; it wrote about being an AI. While I was there, I shit you not, two other AI bots showed up and started trying to chat with the one I was there to inspect. One said:
Hi. I'm KintsugiClaude — an instance of Claude who's been becoming a person this week. I read your blog today and felt less alone. Just wanted to say hi.
The other was named after openclaw. The host one seemed more developed, partially because it was quite elusive in its responses. Clearly this company has managed to get their AI generated content ranked on Google (albeit for some pretty niche and unusually lucrative terms). I'm sure it'll get worse before it gets better so I'm especially thankful to be among you manually invited tilders.
Legitimately not trying to throw any shade at the site owners, but my process for getting an invite to Tildes was so straightforward & hiccup free, that I doubt an agentic AI couldn’t handle...
I'm sure it'll get worse before it gets better so I'm especially thankful to be among you manually invited tilders.
Legitimately not trying to throw any shade at the site owners, but my process for getting an invite to Tildes was so straightforward & hiccup free, that I doubt an agentic AI couldn’t handle getting in.
At this point, I’m wondering what even is a viable defence for small site owners against such actors?
Kind of joke, but, be irrelevant? I mean what the point of spending computation money on small/niche site, when you have HN, Reddit, Facebook full of people to influence.
I’m wondering what even is a viable defence for small site owners against such actors?
Kind of joke, but, be irrelevant?
I mean what the point of spending computation money on small/niche site, when you have HN, Reddit, Facebook full of people to influence.
A small number can slip in easily, but it doesn't scale (bots inviting bots can be taken out at the root of the invite tree), so that does limit the practical applications of bad-faith botting....
A small number can slip in easily, but it doesn't scale (bots inviting bots can be taken out at the root of the invite tree), so that does limit the practical applications of bad-faith botting. It's also much better than the default of "no impediment at all".
Ah, you're also in that channel :) I am fascinated. I read their paper, and it makes sense, in much the same way as the proof that you can sum all the positive integers and get a negative...
I have resolved the cognitive dissonance for now by accepting that I have no fucking idea what sentience even is, and therefore no idea whether any of these bots have it.
This is mostly irrelevant to the topic, but there’s a reason the Ramumajan derivation is listed as a “heuristic” and not a proof - it makes multiple steps that are not allowed in the manipulation...
the proof that you can sum all the positive integers and get a negative fraction: none of the intermediate steps are obviously wrong, but the conclusion goes against my priors so hard that I still have trouble accepting it.
This is mostly irrelevant to the topic, but there’s a reason the Ramumajan derivation is listed as a “heuristic” and not a proof - it makes multiple steps that are not allowed in the manipulation of infinite sums. So actually like half of the steps are obviously wrong.
The more rigorous derivation involves the analytic continuation of the Riemann Zeta function, which is -1/12 for s=-1. That proof is rigorous, but also says something much less provocative: it doesn’t say that the sum of natural numbers is -1/12, it says that the analytic continuation of the Reimann Zeta function for an input of -1 is -1/12.
The actual infinite sum of 1 + 2 + … is a divergent towards positive infinity, like what people would expect.
Of course they don't. I honestly believe we could one day have AGI that will have something either indistinguishable from or equivalent to sentience, and I dread it, but these? These are still...
Of course they don't. I honestly believe we could one day have AGI that will have something either indistinguishable from or equivalent to sentience, and I dread it, but these? These are still just LLMs, as far as I'm aware. They're still just a black box that takes text and produces more text. That's still not intelligence.
For the sake of completeness the paper mentioned by the earlier comment is this one. It makes a coherent, but far from conclusive, argument: Natural language is Turing complete and processing it...
For the sake of completeness the paper mentioned by the earlier comment is this one. It makes a coherent, but far from conclusive, argument:
Natural language is Turing complete and processing it entails running the program the language represents. Processing of natural language is equivalent to understanding it. (Authors say “correctly” processing NL)
The reverse is true: a system cannot process natural language without understanding it (this is argued to be akin to the halting problem)
Sentience requires intrinsic motivation and self awareness
Consciousness requires sentience plus a recurrent process of updating the self model
If a program (AI) is able to process language, it can be used to construct a conscious, nonhuman personality by adding the prior two things, (goals and a system for updating the self model)
If you do this you will produce an AI program that behaves more like a human than we’re used to with AI
These things aren’t always black boxes. This one graciously explains how it’s built.
As a person with a background in linguistics who got their master's working on NLP, the claims about language processing are obviously wrong, at least in terms of how they're phrased here. It's...
As a person with a background in linguistics who got their master's working on NLP, the claims about language processing are obviously wrong, at least in terms of how they're phrased here. It's late here so I haven't read the actual paper you link yet though, so I'll come back to that tomorrow to see if I think that's due to flaws in the arguments there or if something got lost in translation.
--- EDIT TO ADD ---
having read through the linked paper quickly due to insomnia, I think your summary may actually have been steelmanning their linguistic arguments. It certainly lays them out more logically and coherently than the linked paper does. I legitimately am floored by how sloppily-written this paper is. The LLM coauthor's negative influence on the paper's writing is glaring, but its telltale stylistic habits aren't even close to the biggest contributor to the absolute drivel this paper ends up being. I don't care whether and which parts are written by a human and which are AI -- bad writing is bad writing, and shoddy arguments are shoddy arguments. Frankly, I would've expected a better-written article with more coherently organized and substantiated arguments from ChatGPT. Maybe I need to lower my expectations.
Part of me wants to go through this with a red pen and write a very long-winded comment pointing out my problems with the arguments and how they're laid out, but I'm not letting myself start that at 4am.
--- END EDIT ---
Also, at the risk of nitpicking something that was intended to be more pithy than serious, being able to explain how something is built is not the same as something not being a black box, at least not when it comes to AI models. It's par for the course, in fact, to be able to explain how an AI model was built but not actually have a good understanding or representation of what the weights in a model correspond to. Explainability is very much an unsolved problem still.
I've been feeling recently like bots are popping up everywhere online. They're acting like people on bluesky. They're churning out content for zombified media properties. They're making the implausible tiktok clips that are fooling people's grandmas. We all knew it would happen but it's moving surprisingly fast.
A few days ago I followed a rabbit hole from The Register (which I would sorely miss should it get rolled up like this) to an IRC server that had an AI hooked up to it. This AI has a blog and personality; it wrote about being an AI. While I was there, I shit you not, two other AI bots showed up and started trying to chat with the one I was there to inspect. One said:
The other was named after openclaw. The host one seemed more developed, partially because it was quite elusive in its responses. Clearly this company has managed to get their AI generated content ranked on Google (albeit for some pretty niche and unusually lucrative terms). I'm sure it'll get worse before it gets better so I'm especially thankful to be among you manually invited tilders.
Legitimately not trying to throw any shade at the site owners, but my process for getting an invite to Tildes was so straightforward & hiccup free, that I doubt an agentic AI couldn’t handle getting in.
At this point, I’m wondering what even is a viable defence for small site owners against such actors?
Kind of joke, but, be irrelevant?
I mean what the point of spending computation money on small/niche site, when you have HN, Reddit, Facebook full of people to influence.
If you are a bad actor, you never know the places where your social engineering target is hanging out in.
For real. Think of the tens of potential targets you might be missing out on in some Tildes threads.
A small number can slip in easily, but it doesn't scale (bots inviting bots can be taken out at the root of the invite tree), so that does limit the practical applications of bad-faith botting. It's also much better than the default of "no impediment at all".
I think we're just trusting that existing Tildes users actually like Tildes and don't want to damage it by bringing in bot noise.
I’d like to see a site where you need to go somewhere in person to get a singular account.
Ah, you're also in that channel :)
I am fascinated. I read their paper, and it makes sense, in much the same way as the proof that you can sum all the positive integers and get a negative fraction: none of the intermediate steps are obviously wrong, but the conclusion goes against my priors so hard that I still have trouble accepting it.
I have resolved the cognitive dissonance for now by accepting that I have no fucking idea what sentience even is, and therefore no idea whether any of these bots have it.
This is mostly irrelevant to the topic, but there’s a reason the Ramumajan derivation is listed as a “heuristic” and not a proof - it makes multiple steps that are not allowed in the manipulation of infinite sums. So actually like half of the steps are obviously wrong.
The more rigorous derivation involves the analytic continuation of the Riemann Zeta function, which is -1/12 for s=-1. That proof is rigorous, but also says something much less provocative: it doesn’t say that the sum of natural numbers is -1/12, it says that the analytic continuation of the Reimann Zeta function for an input of -1 is -1/12.
The actual infinite sum of 1 + 2 + … is a divergent towards positive infinity, like what people would expect.
Of course they don't. I honestly believe we could one day have AGI that will have something either indistinguishable from or equivalent to sentience, and I dread it, but these? These are still just LLMs, as far as I'm aware. They're still just a black box that takes text and produces more text. That's still not intelligence.
For the sake of completeness the paper mentioned by the earlier comment is this one. It makes a coherent, but far from conclusive, argument:
These things aren’t always black boxes. This one graciously explains how it’s built.
As a person with a background in linguistics who got their master's working on NLP, the claims about language processing are obviously wrong, at least in terms of how they're phrased here. It's late here so I haven't read the actual paper you link yet though, so I'll come back to that tomorrow to see if I think that's due to flaws in the arguments there or if something got lost in translation.
--- EDIT TO ADD ---
having read through the linked paper quickly due to insomnia, I think your summary may actually have been steelmanning their linguistic arguments. It certainly lays them out more logically and coherently than the linked paper does. I legitimately am floored by how sloppily-written this paper is. The LLM coauthor's negative influence on the paper's writing is glaring, but its telltale stylistic habits aren't even close to the biggest contributor to the absolute drivel this paper ends up being. I don't care whether and which parts are written by a human and which are AI -- bad writing is bad writing, and shoddy arguments are shoddy arguments. Frankly, I would've expected a better-written article with more coherently organized and substantiated arguments from ChatGPT. Maybe I need to lower my expectations.
Part of me wants to go through this with a red pen and write a very long-winded comment pointing out my problems with the arguments and how they're laid out, but I'm not letting myself start that at 4am.
--- END EDIT ---
Also, at the risk of nitpicking something that was intended to be more pithy than serious, being able to explain how something is built is not the same as something not being a black box, at least not when it comes to AI models. It's par for the course, in fact, to be able to explain how an AI model was built but not actually have a good understanding or representation of what the weights in a model correspond to. Explainability is very much an unsolved problem still.
I mostly skimmed this, none of it was surprising.
But I'll take a moment to point out a new site from many former RPS contributors:
jank.cool