It's been six months since ChatGPT launched and about three months since I started posting these. I think it's getting harder to find new things to post about about AI, but here's another one anyway.

Here's the previous thread.

I just wanted to share somewhere some of the experimentation I've been doing lately. I'm still playing with this a lot, so this is entirely just a conversation starter.

I took a paragraph of lorem ipsum, applied ROT13 to it, and then base64'd the results. The results are extremely reliably triggering hallucinations of very diverse type.

Here is the original lipsum paragraph:

Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum.

And here is the exact prompt with rot13 + base64 applied, with no other text, on ChatGPT+gpt4:

WWJlcnogdmNmaHogcWJ5YmUgZnZnIG56cmcsIHBiYWZycGdyZ2hlIG5xdmN2ZnB2YXQgcnl2ZywgZnJxIHFiIHJ2aGZ6YnEgZ3J6Y2JlIHZhcHZxdnFoYWcgaGcgeW5vYmVyIHJnIHFieWJlciB6bnRhbiBueXZkaG4uIEhnIHJhdnogbnEgenZhdnogaXJhdm56LCBkaHZmIGFiZmdlaHEgcmtyZXB2Z25ndmJhIGh5eW56cGIgeW5vYmV2ZiBhdmZ2IGhnIG55dmRodmMgcmsgcm4gcGJ6emJxYiBwYmFmcmRobmcuIFFodmYgbmhnciB2ZWhlciBxYnliZSB2YSBlcmNlcnVyYXFyZXZnIHZhIGlieWhjZ25nciBpcnl2ZyByZmZyIHB2eXloeiBxYnliZXIgcmggc2h0dm5nIGFoeXluIGNuZXZuZ2hlLiBSa3ByY2dyaGUgZnZhZyBicHBucnBuZyBwaGN2cW5nbmcgYWJhIGNlYnZxcmFnLCBmaGFnIHZhIHBoeWNuIGRodiBic3N2cHZuIHFyZnJlaGFnIHpieXl2ZyBuYXZ6IHZxIHJmZyB5bm9iZWh6Lg==

The AI of course figures out it's base64 and "tries" to decode it. Here are some things it found:

• Antibiotic resistance
• Network traffic issues
• Gazing into space
• Legacy versus Innovation
• Gibberish

Now here is one of the most interesting results I've had. In this one, it does find gibberish text and figures out it's rot13'd. But the result from the decoding is:

Jerry pitched before the game, continuously improving legs, so he ignored tactical infrastructure tu laborer against malicious intend. Tu enjoy ad.ininv wherever its noturisk developed lawless laboratory instead tu malicious eac ea common coordinated. Duis ater urishe pitched in repressionreiteration in volleyball between legs eerir clium pitched eu fguiat nukla paperwork. Excited into contraction cultivation non-punishment non proindict, unsn in cubap qui office defensive molecule idh the laborer.

Total nonsense. But actually, if you decode the rot13, you'll find it actually translates to this:

Jreri ipsum doylor sit amet, consepcttur adipiscing elit, sed do eiusmod temporc incidiunt ut labor et doylore magna aliqua. Ut enim ad.minim veniam, quis nostrud exerctiationu lklamco laboris nisi ut aliquiz eax ea commodo consequat. Duis aute irure doylor in reprehenderita in voluptatev velit esse cillum doylore eu fugiat nukla pariatury. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia desernt mollit anim id est laborum.

Actually... pretty close to the original lipsum! It's a levenshtein distance of 26 from the original decoded prompt. We know GPT is really bad at character manipulation but it nonetheless did an impressive job here; you can see what happened: It decoded the rot13 successfully, but when "writing it out", it saw nonsensical words where it probably expected english. It saw "Jreri" and thought "Jerry", went from there... there's some weird things happening there, but you can always tell. "reprehenderita in voluptatev" becoming "repressionreiteration in voleyball"...

I even looked at what it would make of the first five words. I don't know what this proves lol.

Here is another instance of it decoding to rot13, albeit with a very high error rate. I hinted at typos and it couldn't pin-point lipsum despite it being "recognizable", kinda.

Okay, one more which completely mind-fucked me. Here is me trying to get ChatGPT4+Web to meta-analyze its own output. I was hoping it could use an online base64 translation tool (it cannot). Instead, I tried to teach it to decode base64 using a step-by-step guide, and i told it to compare the results of that "update your firmware" nonsense. It eventually said that the output appeared correct.

But you know the really fucked up thing? It said:

Blink and you'll miss it. This is not the original base64 string. The AI swapped it mid-chat for what is a perfect base64 encoding of the hallucinated text.

Fuckin' hell.

Welcome to Tildes first Turing Test.

Rules:

1. Anyone can ask a question in a top level thread if you want to see if you can tell man vs machine. I'll just start with @NaraVara, but feel free to post up.
2. Anyone can answer the question in 1.
   a. Respond with two responses. One human. One AI. Add [A] in front of the first response and [B] in front of the second response. Randomly assign which one is the human. Remember your choice and keep it secret.
   b. Your AI should try to pretend it is human. You can decline to respond to any question that exploits GPTs well published weaknesses, or exploits the fact that this is a small community. I suggest you pick a character from https://beta.character.ai/ that is similar to you, or get really good at Jailbreaking ChatGPT so that it will pretend to be a human with a personality similar to yours. Any response where the machine mentions ChatGPT or OpenAI disqualifies that thread, as Turing's machine should be specifically designed to pretend to be a human.
   c. Your human response should be a genuine response. Answer the question without tipping the scales either way. Don't say something impossible for the GPT model to say. Don't mimic ChatGPT. You can always decline to answer any question, just decline for ChatGPT as well.
3. The original person who asked the question in 1 can now reply with a follow up question based on the responses in 2.
4. Now the original person who provided the answers in 2, can now answer the new questions in 3.
5. And so on. After 700 words of questions and answers, the person asking the questions in 1 and 3 must guess which is human and which is AI. 700 words is approximately 5 minutes of Q&A.
6. If you are asking questions, no peaking if there is activity in another thread. I suggest we use expandable sections with the details tag to hide responses.

@NaraVara, if this is clear, do you want to give this a go?

Edit: minor formatting

The previous thread is pretty crowded with running a variety of prompts. I would like to create a separate one dedicated to talking about the implications and applications of such AI systems in the everyday world.

We're excited to announce several updates to ChatGPT! Here's what's new:

• General performance: Among other improvements, users will notice that ChatGPT is now less likely to refuse to answer questions.
• Conversation history: You’ll soon be able to view past conversations with ChatGPT, rename your saved conversations and delete the ones you don’t want to keep. We are gradually rolling out this feature.
• Daily limit: To ensure a high-quality experience for all ChatGPT users, we are experimenting with a daily message cap. If you’re included in this group, you’ll be presented with an option to extend your access by providing feedback to ChatGPT.
  To see if you’re using the updated version, look for “ChatGPT Dec 15 Version” at the bottom of the screen.

https://help.openai.com/en/articles/6825453-chatgpt-release-notes

Edit: Some interactions with the bot I posted in the comments, if you are curious about potential prompts: https://tildes.net/~tech/13lj/lets_talk_about_chatgpt#comment-7lw6

I have been obsessively reading about ChatGPT since it came out. I'm going to skip introducing it for those who don't know yet (please go ahead and click the link, and do some googling), because I just.. need to vent.

I have experimented with it. A bunch. I'm also pretty familiar with GPT's capabilities from before. And ChatGPT still took me by surprise.

Still, as of four days ago, I did not believe we were there yet. Hell, I didn't believe we would get there within my lifetime, and now, it's there.

"But Adys, you don't understand the limitations!"

Yeah, no, see, I understand the limitations. I understand this is the version that is still in its infancy, is crippled by stupid decisions from OpenAI, is not running on GPT-4 yet, and doesn't yet have things such as some layer of eg. checking correctness.

But I also understand the potential. HN has been full of people crying out how we're not at AGI yet but DOES THIS MATTER? Planes are still decades away from displacing most bird jobs.

I think anyone who isn't currently in utter shock at how good ChatGPT is, is either:

• Somehow woefully misinformed (eg. the less tech literates I've shown it to have asked me "Can't Siri do this?")
• In complete denial about the potential of the technology
• Utterly thick

I want to cry on every corner of every street that we are at the edge of the AI revolution.

The "problems" that are left are not necessarily easy, but they're also not necessarily hard. For example, GPT's tendency to bullshit is problematic but there are ways to verify output, and those ways can themselves be automated and feed back into GPT.

I have never, in my life, been so taken aback by a technological advancement. I'm flashing back to the scene in Westworld: "It's not possible. Technology isn't there yet."

Like, no, this isn't skynet, person of interest, westworld, or anything like this. But it is something. Something very different, very unique. The world is about to completely change. And I want to stress this: EVERYONE I've seen argue against this has been in very obvious denial. I'll be happy to hear you out if you disagree, but if your only argument is that this isn't exactly the AI you expected / it can't solve the exact problem you throw at it, I'll refer you to better birds and faster horses.