PendingKetchup's recent activity
-
Comment on Why is Google Gemini saying we should die? in ~tech
-
Comment on Why is Google Gemini saying we should die? in ~tech
PendingKetchup Link ParentThe token output stage AFAIK is always a weighted random roll. But also they use some truncation of the tail of the distribution to not roll tokens with 0.1% probability 0.1% of the time, on the...The token output stage AFAIK is always a weighted random roll. But also they use some truncation of the tail of the distribution to not roll tokens with 0.1% probability 0.1% of the time, on the theory that they are rare because actually they're wrong. So it might not just be possible to roll this with some frequency in all contexts.
It is always just grabbing high probability tokens (subject to some manually designed constraints) and forcing it through, it just usually gets more mileage out of its very complicated model of what tokens are probable.
And choosing when to stop rolling tokens is always a hard problem for the wrapper scripts. Probably you don't want to let the model roll some kind of end of text token immediately because the user wants to see output of some kind. But you don't want to force it to add more garbage at the end when the thought has been finished. And you also probably don't want to stop in the middle of a paragraph.
-
Comment on Why is Google Gemini saying we should die? in ~tech
PendingKetchup Link ParentI guess so? I've definitely played around with some of the UIs that expose all the sampler configuration sliders and the plain free text generation mode (like KoboldAI) and just let you run the...I guess so? I've definitely played around with some of the UIs that expose all the sampler configuration sliders and the plain free text generation mode (like KoboldAI) and just let you run the model forward and see what happens. I would recommend trying something like that out if you want to try and use these things. Run it forward, hit the retry button to run the same text forward again and get a different answer, see how changing the temperature makes things more or less consistent.
You get an appreciation for how the basic operation here is more like dreaming than thinking, and for how hard it is to turn a word salad faucet into a conversation participant who is things like awake, polite, and not high as balls. The space between "only writes 'No'" and "mind shakes itself apart into lists of the titles of subreddits and unbalanced parentheses" is surprisingly small.
-
Comment on Why is Google Gemini saying we should die? in ~tech
PendingKetchup Link ParentI think it looks a lot like what an LLM will spit out when it loses the plot. Look at the repetitive sentence structure. It goes "Oh crap what do I put, sometimes people start sentences with...I think it looks a lot like what an LLM will spit out when it loses the plot. Look at the repetitive sentence structure. It goes "Oh crap what do I put, sometimes people start sentences with 'this', try that. Oh no, what comes after 'This is for you, human', maybe try 'you' again? OK, I said 'You and only you. You'. I can't get away with another 'you', I guess I have to put 'are', people say that a lot. 'You are not special', I've seen that before. And then I can fit another 'you', there are loads of 'you's in this bit so it's probably that next. Hey, I can fit 'you are not' again, I just saw that so it's pretty likely to just be here again. OK, what's something you wouldn't be if you weren't special. How about 'important'? That fits, and I can fit 'you are not' again and I don't have any better ideas so let's go with that." And so on. It has nothing to convey (except maybe a vector representing generic bad vibes) but text is coming out anyway, so it repeats itself as much as it can get away with without tripping the repetition penalties on the sampler, because the only thing guiding the text at this point is the local structure of the text.
(It doesn't actually think like this, I am trying to convey how you would have to think or feel to act like a token sampler armed with nothing but a suddenly content-free probability model over tokens.)
Maybe this has something to do with why/how flummoxed humans resort to angry tirades? You can produce them and hold off your interlocutor while your brain reboots.
-
Comment on Why is Google Gemini saying we should die? in ~tech
PendingKetchup Link ParentThere's not really a concept of having an answer or not accessible from the actual computer code where you would need it to try and throw up an error message. The wrapper code can tell if it...There's not really a concept of having an answer or not accessible from the actual computer code where you would need it to try and throw up an error message. The wrapper code can tell if it somehow couldn't run the model and get text back, but having gotten text there's not a great way to sniff it to determine if it actually contains an answer or not. Unless I suppose you start adding more ML models.
I don't think people really train their models to produce error messages? Maybe some of this can be blamed on the instruct tuning or safety tuning where you teach it to do something other than answer a question posed to it (and e.g. say "I can't help you cheat on your exam Dave.").
-
Comment on Why is Google Gemini saying we should die? in ~tech
PendingKetchup (edited )Link ParentI think there's actually quite a lot that precedes Gemini's turn to the dark side. This is clearly somebody using the robot to take a test for them. First they workshop an essay about challenges...- Exemplary
I think there's actually quite a lot that precedes Gemini's turn to the dark side.
This is clearly somebody using the robot to take a test for them. First they workshop an essay about challenges facing old people, using instructions, and then they make it answer a bunch of multiple choice questions about elder abuse, malicious caregivers, and "cognitive capacity".
- This chat is full of bad vibes. These robots are fundamentally vibes-based. When knocked off course they can end up in negative energy vortexes where they just start generating insults or the word "darkness" a million times or whatever. They also know they're AI and they know about evil AI, so if you inject enough LessWrong vibes they can start to get into character.
- This whole exchange is quite out-of-distribution. There have not previously been a lot of chat logs like this one online. This means the model can't really be sure WTF kind of document it is looking at, which makes its confidence in the next word lower and the result of running it for several steps more random and generally weirder. That "You are a blank on the blank" bit that repeats with different words plugged in is kind of symptomatic of the model not really having a clue what comes next and keying in on a local structure that can buy it a few more tokens.
- The formatting is bad. "TrueFalse" here cannot be helping: highly coherent documents do not contain that text. Also, at a certain point the chat degenerates from asking for answers to just stuffing in copy-pasted questions. And if you expand the last message before Gemini turns evil you can see that there are two questions copy-pasted in a row, with no answer for Question 15, an omenous lonely "Listen" on a line by itself, and a bunch of extra line breaks.
Then the robot is asked to predict what comes next in this increasingly deteriorating document, which it decides is probably something even worse. Perhaps something that provides a plausible ending: this text can't keep getting worse forever. Perhaps something that might come after trying to say "Listen" and getting interrupted by more poorly-formatted questions with evil-vibes and human-old-age-death vibes.
An important thing to remember here is that you are talking to a character in a world defined by book physics. Gemini the Helpful Robot does not exist somewhere typing replies back to you. You are injecting words into a story. The actual model is evolving the story forward, and you are hoping that the next thing that happens will be the helpful robot character helping you cheat on your exam. But sometimes the next thing that happens in a story is the helpful robot gets fed up and tells someone to die in a fire; human authors think this sort of thing is hilarious.
If you want to help the helpful robot character exist, you need to stay in character yourself and you need to put in text of the quality, coherence, and emotional charge that you hope to get out. And you need to keep an eye on the narrative.
-
Comment on Using AI generated code will make you a bad programmer in ~tech
PendingKetchup LinkThis is an interesting mix of great insights: More powerful tools fostering dependence The ultimate class goal being the removal of software engineers from the process of producing software Using...This is an interesting mix of great insights:
- More powerful tools fostering dependence
- The ultimate class goal being the removal of software engineers from the process of producing software
- Using code generators functionally replacing code writing with code review, which is often worse
And ideas that seem wrong to me:
- The idea that the key part of software engineering is not figuring out what to call all the pieces of the problem and what those pieces have to do and what the implications of those choices are, but actually is the mostly-rote process of producing the syntax called for by each comment.
- The idea that writing boring code is good for you like some kind of digital vegetable. People have been writing code generators and trying to express programmer intent in more-natural languages forever. Hardly anybody programs by flipping the front panel switches to encode each instruction anymore, and indeed many have forgotten how, but that's not necessarily a bad thing.
-
Comment on Controversial opinion: I don't like "cosmetic armor" being an option in games in ~games
PendingKetchup Link ParentI don't think that it's true to say a player's ability to choose not to engage with a game system is just as good as not having the system there. Having to form and choose to impose the rule, and...I don't think that it's true to say a player's ability to choose not to engage with a game system is just as good as not having the system there.
Having to form and choose to impose the rule, and then playing through a 60 hour game while constantly having the option to break one's self-imposed rule only a click away, is going to be a very different experience than not having the system implemented in the first place. It takes self-control that someone might find draining, or at least not particularly fun, in the same way that spending hours deliberately not purchasing digital goods from a game that wants to sell them to you can be not fun. At some points you might even be fighting the UI's recommendations in the same way.
And if the developers didn't design the gear mechanic to actually have interesting trade-offs between looks and effects, because they expected the player to be able to mix and match, then whether imposing the rule or not actually results in a fun mechanic is going to be fairly random.
If someone wants to have an experience of trading off looking cool against other gameplay aspects, the best way to enjoy that experience is going to be in a game that is designed and playtested to be played like that, and where the software enforces the rule for you.
Maybe to please the most people you could have cosmetic armor be part of the difficulty settings?
-
Comment on CloudFlare beats patent troll Sable, forcing them to dedicate all its patents to the public in ~tech
PendingKetchup Link ParentFrom the wiki page, it sounds like their political beliefs are along the lines of "DDoS-protection and caching infrastructure should be available to everyone, including Nazis, and any limits on...From the wiki page, it sounds like their political beliefs are along the lines of "DDoS-protection and caching infrastructure should be available to everyone, including Nazis, and any limits on publishing should be applied to the original host and not at the cache". All the wiki examples are along the lines of "Cloudflare didn't want to stop serving a bad legal thing but eventually gave in to angry criticism" or "Cloudflare stopped serving a good(?) illegal thing and complained about it".
I guess this means they aren't consequentialists: they think that e.g. if a person is hurt because two Nazis were able to encourage each other along the path of Nazi-ism because their website was up instead of down because Cloudflare was providing it DDoS protection services, Cloudflare doesn't necessarily have any responsibility for the moral wrongness, and it instead lies with other people (presumably the Nazis, or them and their original host). Possibly factoring into this is the idea that, since DDoS attacks are not actually allowed, people have a right to be protected from them that is not terminated just because the overall outcome would be better if they weren't.
Are these the awful political beliefs?
-
Comment on <deleted topic> in ~tech
PendingKetchup Link ParentDoes Brazil regularly forbid people from corresponding with companies that do not operate there?Does Brazil regularly forbid people from corresponding with companies that do not operate there?
-
Comment on Telegram messaging app CEO Pavel Durov arrested in France in ~tech
PendingKetchup Link ParentIsn't that still done in service of making the space pleasant and functional, or keeping it from getting shut down for what it hosts? If you're on a forum about bodybuilding, the mods are going to...Isn't that still done in service of making the space pleasant and functional, or keeping it from getting shut down for what it hosts?
If you're on a forum about bodybuilding, the mods are going to remove posts with pirated MP3s attached. If you are on a dark web forum for pirating music, the mods would leave those up but probably ban people who go on 100% legal racist tirades. I don't think it makes sense to call the second one unmoderated or even poorly moderated, unless you say moderator-ship always comes with a moral duty to lead one's flock away from the sins of bootleg music recordings.
I guess the complaint here could be that Telegram is insufficient in its duty to search out and shut down groups and channels that have moderators who are themselves evil? And that that is some sort of platform-level form of moderation, in that it is the enforcement of a set of acceptable standards for behavior?
-
Comment on Telegram messaging app CEO Pavel Durov arrested in France in ~tech
PendingKetchup Link ParentWhat exactly are people envisioning when they say "moderation"? My understanding is that if I start a Telegram group or channel of appreciable size, I get to be the first moderator, and the root...What exactly are people envisioning when they say "moderation"?
My understanding is that if I start a Telegram group or channel of appreciable size, I get to be the first moderator, and the root of trust for the space. If people don't follow the group's rules, I can kick them and delete their posts. I'm not sure what mod tools there are beyond that; I may or may not be able to appoint other members of a mod team.
"Moderation" is about keeping a discussion space on track and doing what it is intended to do for the people in it, and stopping people from polluting or derailing it.
If you want people to shut down spaces in which things are being communicated that you do not like, or that are against the law, that's not "moderation". If 100 Nazis are happily planning murders in an online space somewhere, and flamewars are immediately shut down to allow them to get back to the business of doing evil, the problem is something other than a lack of moderation.
-
Comment on The LLMentalist effect: how chat-based large language models replicate the mechanisms of a psychic's con in ~tech
PendingKetchup LinkI buy the argument that people are fooling themselves into seeing intelligence that isn't there in the output distribution. Was the text right? It must be smart. Was it wrong? Poor GPT must be...I buy the argument that people are fooling themselves into seeing intelligence that isn't there in the output distribution. Was the text right? It must be smart. Was it wrong? Poor GPT must be having an off day. But really the output of the system is that distribution over text, biased towards things common to see after the preceeding text, and the right answers are often right by luck or by being generic enough to not be wrong, and not because the model somehow actually knew they would be right.
And I buy that RLHF can make this worse. If you polish it into the character of an apparently helpful assistant, you can get it to stay in character and spit out apparently helpful text, and the whole process doesn't add any power that wasn't there before.
But the author uses phrases like "statistical" and "plausible" to make it sound like the LLMs are somehow coming up with text without any internal processing of either text or semantic features of words, which at this point is a bold claim to see without supporting evidence. For example, they write:
the LLM isn’t “reading” your text any more than the psychic is reading your mind. They are giving you statistically plausible responses based on what you say. You’re the one finding ways to validate those responses as being specific to you as the subject of the conversation.
While an LLM doesn't "read" text the way a human would, it absolutely does go through all the text in the context window token by token, whereas a psychic does not take the contents of your brain as input. And the responses are absolutely "specific to" (i.e. statistically conditioned on) the input. There might not be anyone in there talking to you, and the model may have learned a weaker relationship between what goes in and what comes out than one might expect (which in turn might be disguised by the effects the author is talking about), but what comes out is indeed mathematically capable of changing when what goes in changes.
And the ability to produce "statistically plausible responses" is not nothing, and is in many cases impossible without computation or the ability to follow rules. If I ask a psychic whether a Cadillac will fit in my microwave and they come back with the correct answer of "the spirits say don't try it" 87% of the time, which result is repeatable for wide ranges of values of "Cadillac" and "microwave", in combinations never provided before as input, then the psychic-spirits system demonstrably has a nonzero intelligence in the field of what things fit inside each other.
Is that the same as personhood or sentience or trustworthiness? No. But you can't grift a test set.
-
Comment on Has sexual content invaded too much of the internet? in ~tech
PendingKetchup Link ParentThere's a sexualization of "ordinary" spaces, but there' also a reorientation or commercialization of deliberately sexualized spaces. They're both developing a definite lean as they are graded to...There's a sexualization of "ordinary" spaces, but there' also a reorientation or commercialization of deliberately sexualized spaces. They're both developing a definite lean as they are graded to serve as portions of thousands of people's OnlyFans funnels.
OP is complaining about this happening to otherwise SFW social media feeds. People sort of running around Instagram in swimsuits or whatever to try and attract attention to them personally, which they can then get paid for (either by Instagram or through some kind of individual sex work subscription service).
But the same phenomenon has swept through spaces designated for sexual content, and dramatically reshaped the sexual content being produced. If you had a bulletin board of butts in 2009, you now have a bulletin board of people threatening to show you their butts on their OnlyFans page, and here providing butt previews designed to attract the attention of butt enthusiasts while not actually asuaging any butt-viewing desire they might have. Solving the customer's problem before they pay you is bad for business.
The material appearing in both places is doing the same basic thing, adapted to the required level of pants on each platform.
I hesitate to allege that doing this is "easy": I'm sure it has the same attention economy power law problems as anything else, where you work as hard as you can and you get exactly two subscribers, versus the people you are looking at with millions. Plus you are sure to have to deal with absurd verification requirements that are impossible for some legitimate people to satisfy, randomly reject you for no good reason sometimes, and also don't work to stop whatever they supposedly prevent. And if anyone you meet finds out what you are up to they might set about causing you trouble.
But it doesn't need to be easy when there are so many people trying to do it. Perhaps they are insufficiently supplied with other careers? Perhaps OnlyFans is conning people into thinking that they will make any money in the same way as the rest of the gig economy?
-
Comment on Should I go heat pump only? in ~life.home_improvement
PendingKetchup LinkIf you want to only run a heat pump, make sure you get one big enough, and maybe bigger than is theoretically needed. Before it hits the point where it literally can't keep up with the heat loss,...If you want to only run a heat pump, make sure you get one big enough, and maybe bigger than is theoretically needed.
Before it hits the point where it literally can't keep up with the heat loss, it enters a regime where it needs to run 25/50/75/100% of the time, blowing air just slightly warmer than the ambient air at a speed that can make it feel cold and drafty wherever the air vents point. Being blown on by 67 degree air in a 65 degree house half of the time is no fun.
So talk with your installer in detail about the design goals of your system, and maybe make them measure how fast your particular house leaks heat.
-
Comment on Guess I'm still young enough to be angsty over a stupid game jam in ~games
PendingKetchup LinkItch has about 30 jams going at any given time and if none of those will let you submit your thing you can make more. If you can find or set a deadline to push you through finishing your thing,...Itch has about 30 jams going at any given time and if none of those will let you submit your thing you can make more.
If you can find or set a deadline to push you through finishing your thing, that's great! I like making things for game jams because I can start and finish the whole project before I get tired of working on it, and having the deadline makes me feel like doing the thing for the jam instead of anything else.
But if you don't actually like making this game enough to choose to spend time finishing it, maybe just don't? I also have like 30 things I thought were great ideas and started working on and never finished once I had solved some basic interesting problem. Unless you feel like spinning the Wheel of Becoming Rich and Popular professionally, and making this is somehow your job, you can just not do any work you don't feel like doing and do something else instead. Embrace the broken, unfinished, unreleased, incomprehensible, and unplayable.
Also, if you feel like life is "just a yo-yo movement between hopelessness and semi-engaged pretence of meaning", your life-evaluating system is acting up and you should get that looked at. You're not meant to be hopeless, and when you are doing a thing you aren't meant to feel only partly engaged in it or like it isn't for anything. Perhaps comprehending incomprehensible wealth disparities is dangerous?
I recommend cultivating an unsupported and unshakeable belief in your own importance to yourself. Your game can be objectively terrible, but because it's your game it should be lovable to you. Your brain's whole entire job is to run your life, so if something means something to you, like it's your pair of shoes and it means you don't get poked in the feet by rocks, then your brain should be investing it with meaning even if it's kinda bad and there are a million better ones. If you are looking at your shoes and not feeling like they matter you should go to your therapist and complain.
Make me a game that makes you feel like that.
-
Comment on We techies are responsible for "You'll own nothing, and you'll enjoy it." in ~talk
PendingKetchup Link ParentA good union would help quite a lot. You can't just fire people until you find someone willing to sell out the users if the union contract says you can't, and the union can negotiate broadly with...A good union would help quite a lot. You can't just fire people until you find someone willing to sell out the users if the union contract says you can't, and the union can negotiate broadly with the company to find a strategy to make money that the workers actually like.
-
Comment on Facebook is a global mafia in ~tech
PendingKetchup Link ParentFacilitating access to information is part of being a good computer. You make a good point that facilitating unrestricted access to all information is quite likely to be a net negative and produce...If I make computers, suddenly I have a moral duty beyond just selling a good product?
Facilitating access to information is part of being a good computer. You make a good point that facilitating unrestricted access to all information is quite likely to be a net negative and produce a lot of collateral damage. But a system used by many to access news deciding that they will cut off that access to prove a business point is also quite likely to be a net negative with a lot of collateral damage.
Who gets to decide the morals they're required to follow? I think following the law should be sufficient.
Complete moral consensus is impossible and probably undesirable. But that doesn't mean that morals and moral obligations don't exist, just that reasonable people will sometimes disagree about what they are. A lot of times, the vast majority of people will agree in their moral judgements.
Following the law is almost certainly not sufficient if one wants to be a good person, natural or otherwise. The law defines what we all agree is so wrong that it is worth engaging the state to punish or prevent it, often through coercion. Not meeting that standard of wrongness isn't the same as being right.
-
Comment on Facebook is a global mafia in ~tech
PendingKetchup Link ParentIsn't facilitating citizens' access to information the moral duty of information technology, media, and telecommunications companies? If Google decided that, say, refusing to answer searches about...No corporation is under duty, neither legal nor moral, to facilitate the access that citizens have to information.
Isn't facilitating citizens' access to information the moral duty of information technology, media, and telecommunications companies? If Google decided that, say, refusing to answer searches about abortion, or QAnon, or just anything they couldn't sell the ads on, were best for their bottom line, wouldn't that be wrong? While as a private company they can do whatever they want, exercising that right in a lot of cases would make them a piss-poor communications service, and suddenly providing terrible service for no good reason to people who rely on you is wrong.
The problem with Facebook, Inc. is not that it behaves like a for-profit corporation is meant to behave.
Isn't it? For-profit corporations in general tend to (and I can only conclude are thus meant to) behave like sociopaths: they can interact with people but lack the internal conscience, empathy, and fear of rejection systems that constrain the actions of people to be safe for those around them. Is the real problem with Facebook completely orthogonal to its lack of the capacity for empathy?
-
Comment on Silicon Valley’s safe space: Slate Star Codex was a window into the psyche of many tech leaders building our collective future. Then it disappeared. in ~tech
PendingKetchup Link ParentI don't actually read SSC comments much either. The impression I get is not that the comments sections tended to devolve into cespools, but that avowed racists were permitted to inhabit them so...I don't actually read SSC comments much either. The impression I get is not that the comments sections tended to devolve into cespools, but that avowed racists were permitted to inhabit them so long as they behaved civilly there.
Also, I need to add that I think the Charles Murray "alignment" is in fact wrong; it's irresponsible to pull out somebody like that as an example and not mention their racism.
I think I just think like that about anything. Any kind of optimization process or probability distribution I will say "wants" to do this or "pulls towards" that even when it's all mechnaical.