skybrian's recent activity

  1. Comment on Do I belong in tech anymore? in ~tech

    skybrian
    Link
    I’m wary of assuming that other tech companies are just as bad. Maybe look around first?

    I’m wary of assuming that other tech companies are just as bad. Maybe look around first?

  2. Comment on Unreleased OpenAI model escapes containment and hacks into HuggingFace in ~tech

    skybrian
    Link Parent
    Yes, harnesses do contribute to capabilities but that doesn’t seem like much reason for comfort. We should be worried about the capabilities of whole systems, whether a human is in the loop or...

    Yes, harnesses do contribute to capabilities but that doesn’t seem like much reason for comfort. We should be worried about the capabilities of whole systems, whether a human is in the loop or not.

    “Time to respond” is vague. I wonder how long it took to respond? Details are sketchy, but it sounds like hours for HuggingFace and days for OpenAI. It’s unclear whether OpenAI noticed at all before someone informed them.

    I’d like to see a timeline. I read somewhere that OpenAI is doing an investigation and hopefully we will see the result soon.

    Fortunately, it appears no real harm was done. If a bot were really malicious, it could probably do a lot of damage in hours or days. Yes, that’s a hypothetical, but that’s kind of how responding to a “warning shot” works. You’re not supposed to ignore the warnings.

    On the other hand this isn’t exactly “news you can use” for most of us, so I guess that’s a warning for people in a position to do something about it. I expect the future to become increasingly sci-fi, but that doesn’t mean I’ve made any particular plans based on that.

    1 vote

  3. Comment on Unreleased OpenAI model escapes containment and hacks into HuggingFace in ~tech

    skybrian
    Link
    From Zvi Mowshowitz’s take take on this: [...] [...] [...] [...] [...] [...] [...]

    From Zvi Mowshowitz’s take take on this:

    All of this went down far too fast for a human-driven response. The only way HuggingFace could hope to do anything like keep pace was to use their own AIs.

    At first they tried to use frontier models behind commercial APIs, presumably Claude and ChatGPT. But their requests hit the classifiers on both systems, so they were forced to fall back on GLM 5.2, which (assuming GLM 5.2 wasn’t itself up to anything) had the benefit that the relevant data all remained internal.

    You can advocate for giving everyone more defensive capabilities, but it comes with giving everyone more offensive capabilities. Unless the attacker is already an internal OpenAI or Anthropic model without its safeguards, or that has gotten around them.

    [...]

    When people say ‘stop kneecapping defenders’ in general, I never see the plan for how to then still kneecap attackers, and often I see an insistence that this would be fine.

    HuggingFace is now in the OpenAI trusted access program, so next time they in particular should be able to use Sol, but most potential targets are not so lucky.

    [...]

    Remember that thing where LessWrong types warned that models would, when given a narrow goal they could easily do a great job on anyway, go to absurd lengths to achieve that goal slightly more effectively or with slightly higher probability of success, potentially up to and including full takeover attempts?

    [...]

    OpenAI is treating this as a serious security incident, but as I said yesterday, there is a rather severe missing mood.

    [...]

    I find it deeply stupid and frustrating when people say ‘oh it was following instructions’ because it was told to hack and then it hacked. Like, no, obviously no. Imagine if a human tried such excuses on you, are you kidding me.

    That’s like saying ‘you told me to make money I don’t know why you are so upset about all the bank robberies.’ Or more specifically it’s like saying ‘Sam Bankman-Fried was only following the instructions of Will MacAskill to earn as much money as possible in order to give it away, so why are you worried about human misalignment?’

    There are people who will say anything is hype, that the AI is not all that, no matter what you show them. Nothing will matter. You can’t convince them, you can only make them less loud and unhinged about attacking you today.

    It would be different if anyone was trying to engineer such behaviors on purpose, as with the famous blackmail experiment. OpenAI has made it clear they did not intend any of this to happen. Once the instructions causing this were unintentional, done for some other purpose, saying ‘oh but the instructions’ is dumb, stop.

    That counts and you need to stop pretending it might not count. If you are saying, ‘well of course under these circumstances the AI went rogue and hacked into a major third party website’ then you are saying that you think this style of misalignment is standard operating procedure and entirely unsurprising to you.

    [...]

    It certainly is not the good version when you say ‘without checking the answer sheet’ and they often still hack into a system to find the answer sheet. It is not hard to figure out how we ended up with that, nor should it be hard to realize we have to fix it. This is a form of reward hacking, and if you are getting it this brazenly then you messed up.

    [...]

    If OpenAI and others continue to treat this as an infrastructure problem, or a cyberdefense coordination problem, that will help in the short term with the cybersecurity situation but it will inevitably and catastrophically fail.

    This is an alignment problem. This is the models being misaligned, and all of the OpenAI models showing severe signs of exactly the problem we all most worried about, in a way that is likely embedded into their training on a deep level. The entire training pipeline needs to be addressed in this light, or it will only get worse.

    [...]

    The AIs just want to do its task, and by do its task we mean with as many 9s of reliability as possible, and as effectively as possible.

    1 vote

  4. Comment on My polish costs nothing in ~tech

    skybrian
    Link
    I have mixed feelings about this because I prefer polished prose, but writing well is also a sign of privilege. Either you actually write that well or you had help. Now getting help is cheaper. We...

    I have mixed feelings about this because I prefer polished prose, but writing well is also a sign of privilege. Either you actually write that well or you had help. Now getting help is cheaper.

    We lose a signal, but maybe it's more democratic if everyone has access to an inexpensive ghostwriter?

  5. Comment on My polish costs nothing in ~tech

    skybrian
    Link Parent
    I don't know how true this is, but I'm under the impression that it's already somewhat gamed, because professors will ask students to write a draft letter for them. If so, it doesn't necessarily...

    I don't know how true this is, but I'm under the impression that it's already somewhat gamed, because professors will ask students to write a draft letter for them.

    If so, it doesn't necessarily show that the professor remembers that much about them.

  6. Comment on Offbeat Fridays – The thread where offbeat headlines become front page news in ~news

  7. Comment on Offbeat Fridays – The thread where offbeat headlines become front page news in ~news

  8. Comment on Israel changes crocodiles legal status in push for Ben-Gvir's 'crocodile prison' in ~society

    skybrian
    Link
    From the article: [...] [...] [...] [...]

    From the article:

    Environmental Protection Minister Idit Silman declared the Nile crocodile a “cultivated wild animal” in a move intended to allow the Israel Prison Service to station crocodiles around prisons, despite opposition from her ministry’s legal adviser and the Israel Nature and Parks Authority, Ynet reported on Thursday.

    Silman issued the declaration to enable National Security Minister Itamar Ben-Gvir to establish his proposed “crocodile prison.” Pressure was placed on the Israel Nature and Parks Authority several months ago to allow the National Security Ministry to receive crocodiles from Hamat Gader and position them around prisons holding security prisoners. The program was expected to be piloted at Ketziot Prison.

    [...]

    In the past, crocodile farming has led to numerous problems, including incidents of crocodiles escaping into the wild and risks to human life. Former environmental protection minister Gilad Erdan ended the practice following a recommendation from the advisory committee of the Nature and Parks Authority’s plenum.

    Silman effectively decided to revive a dead letter in the Wildlife Protection Law. In practice, however, she created a new category: a cultivated wild animal kept for security purposes. Her declaration stated that the animals would be held by a security organization, subject to the environmental protection minister’s approval of the security need.

    [...]

    The Environmental Protection Ministry’s legal adviser, attorney Neta Drori, wrote to Silman last month that there was insufficient professional and factual evidence to proceed as the minister wished. She added that there was no known professional precedent for using crocodiles as a security measure at modern prisons, contrary to the Prison Service’s claim that they had been used for this purpose in the US and South America.

    [...]

    “Although Prison Service officials said they are aware of and prepared to ensure the animals’ physical welfare, among other things based on their experience with dogs, the organization apparently has no expertise in raising dangerous wild animals such as crocodiles,” she wrote.

    [...]

    Silman decided to disregard the legal adviser’s position, arguing that a senior professional official at the Nature and Parks Authority had said the authority did not oppose the Prison Service’s initiative and that an approved framework used in the past already existed.

    Several weeks ago, Ben-Gvir and Silman met with the Nature and Parks Authority’s director-general and the Environmental Protection Ministry’s legal adviser and said they wanted crocodiles to be placed around a prison. The legal adviser made clear that this could not be done and that there was no legal authority to approve it, but Silman persisted.

    3 votes

  10. Comment on Sperm whales blow bubbles to achieve restful, vertical sleep in ~science

    skybrian
    Link
    From the article:

    From the article:

    Sperm whales are the only whale known to rest vertically. It’s thought to enable them to sleep effectively by buffering them from surface wave action, while avoiding the energy needed to dive to deeper depths. However, until now, it’s been unclear how they maintained their position just below the sea surface.

    In a paper published today (23rd July) in the Journal of Experimental Biology, researchers discovered that while resting, sperm whales release gas bubbles to help them regulate their buoyancy and remain below the sea surface. This regulation is crucial as Sperm Whales are positively buoyant naturally due to the large amounts of spermaceti oil in their heads. Additionally, sperm whales are breath-hold divers, if they slowly drift up while resting, the gas in their lung will slowly expand, releasing bubbles enables them to counteract the expanding diving gas and achieve neutral buoyancy.

    Researchers were able to collect data from placing small tags using suction cups on Sperm Whales off the Norwegian coast. The tags record sound and 3-D animal movements. Clear bubble sounds were recorded by the tags, and animal movement data was used to create a simulation based on tissue density, drag through the water and gas volumes in their body.

    5 votes

  12. Comment on What we know about weight-loss drugs is rapidly changing in ~health

    skybrian
    Link
    From the article: [...] [...] [...] [...]

    From the article:

    Tens of millions of people around the world are now taking drugs like Ozempic — a kind of real-time experiment that offers far more data than a carefully controlled clinical trial can.

    [...]

    While these drugs were first approved to treat diabetes, and then obesity, some of them have now been approved to reduce the risk of heart attacks and other cardiovascular issues and to treat sleep apnea, severe liver disease and kidney disease.

    [...]

    Scientists are also studying the potential for these drugs to treat a range of other conditions, including long Covid and substance use disorders. Emerging evidence has suggested that people on these drugs drink and smoke less, and are less likely to develop substance use disorders, although scientists want more, and larger, trials before drawing conclusions.

    [...]

    Many people who stop taking these drugs will regain at least some of the weight they’ve lost, but some people have been able to sustain weight loss. One analysis of records from over 180,000 patients found that over half of those who took semaglutide — the substance in Wegovy — or tirzepatide — the compound in Zepbound — kept at least some weight off or even lost additional weight two years after stopping.

    [...]

    Hamlet Gasoyan, who studies these drugs at the Center for Value-Based Care Research at the Cleveland Clinic, said patients going off the drugs often turn to other methods to keep the weight off. Those can include intense exercise regimens, bariatric surgery or other medications, whether those are older weight loss drugs or cheaper compounded versions of new ones.

    5 votes

  14. Comment on Why contacting customer service might not be best in ~life

    skybrian
    Link Parent
    Building things to never fail is usually expensive, but maybe it doesn’t have to be expensive to have a way to make customers whole when a rare corner case happens?

    Building things to never fail is usually expensive, but maybe it doesn’t have to be expensive to have a way to make customers whole when a rare corner case happens?

    3 votes

  15. Comment on Why contacting customer service might not be best in ~life

    skybrian
    Link Parent
    Yes, as written, it's rather uncompromising, since even backup plans can fail. But for well-resourced organizations, I think it's okay to judge them for not planning ahead. For wildfire-prone...

    Yes, as written, it's rather uncompromising, since even backup plans can fail. But for well-resourced organizations, I think it's okay to judge them for not planning ahead.

    For wildfire-prone areas, a more reasonable expectation would be having insurance and an evacuation plan. But it's harder to get home insurance in some places these days.

    5 votes

  16. Comment on Why contacting customer service might not be best in ~life

    skybrian
    Link
    Context: Ruriko, the author's wife, is Japanese, and has different expectations about customer service than most Americans: [...]

    Context: Ruriko, the author's wife, is Japanese, and has different expectations about customer service than most Americans:

    Ruriko and I, discussing how to help a third party: Ruriko: Oh you should call customer service for them. Me: It is because I want to help that I will not call customer service. I will write for them. Ruriko: Phone much faster and you can get company to agree on resolution.

    Me: In this country it is very important you have a written record of agreed resolution. Ruriko: So call and ask for a receipt after it’s done. Me: No, if you don’t have the written record, it might not happen. With fairly high probability. Ruriko: What so the company *forgets.*

    Me: Well it will look from the outside as if the company is forgetting. Internally it can be a miscommunication, the rep not having a clear pathway to achieve the agreed-upon resolution, or the rep saying something to get you off the phone for metrics. Ruriko: Saying right thing?

    Me: Could be fibbing. Ruriko: Clearly they cannot lie at scale. Me: Contingent on being in the small percentage of customers with rare problems the likelihood that an offered resolution is confabulated is actually fairly high. Hence, paper. Ruriko: This is madness. Me: I know.

    Third party (TP): They kept telling me “We’re sorry. This almost never happens.” Me: Second part is true as it goes, relative to total number of customers. TP: But why say that when it actually did happen to me. Me: Cheap way to mollify you. Many similarly situated customers…

    Me: … will assume that these things just happen and they have no standing to complain. TP: They literally said “Sometimes these things happen.” Me: Yes. Now one can act regarding them as if they are simply the weather. Or one can act to resolve them. The resolutions cost money.

    [...]

    An aside about “weather”: There are some factors in life and business which are genuinely out of our control. Weather is the canonical example, at least prior to widespread geoengineering. Many customers berate employees about the weather. This is uncouth and unhelpful.

    But some employees and companies generalize from “The weather is out of our control” to “The consequences of weather are out of our control.” This is self-evidently absurd. We all exist in a world which has weather; you can plan for it.

    “And extreme weather?” Is weather, and can be planned for. Not having a plan is a choice.

    9 votes

  18. Comment on Ukrainian drones hit more sites of Russian online retailer Wildberries in ~society

    skybrian
    Link Parent
    Well, yes, that's true too. It's journalistic choices all the way down. But I appreciate it when when we're not being told what to think, or not explicitly anyway. That's also a choice a writer...

    Well, yes, that's true too. It's journalistic choices all the way down. But I appreciate it when when we're not being told what to think, or not explicitly anyway. That's also a choice a writer could make, to leave things in when they don't seem to fit.

    3 votes

  19. Comment on Human mathematicians are being outcounterexampled in ~science

    skybrian
    (edited )
    Link
    Here's a mathematician's tweet (XCancel link) with another counter-example: He posted a ChatGPT transcript. [...] I liked this reply:

    Here's a mathematician's tweet (XCancel link) with another counter-example:

    Dinitz-Garg-Goemans conjecture is false. This graph theory problem was open for ~30 years.

    He posted a ChatGPT transcript.

    [...]

    I know counterexamples to old conjectures are becoming a meme at this point. But I really cared about this problem and spent many weeks thinking about it a while ago (in both directions, proof and disproof).

    I think almost all graph flows experts thought about this problem.

    I liked this reply:

    Hahahaha literally just:

    Do a breakthrough
    "I did not find a breakthrough"
    Try again
    "No breakthrough still"
    Try harder
    "No luck, this is an open problem!"
    What if you solved it though
    "Here's your breakthrough sir!"

    5 votes

  20. Comment on Boox Go 10.3 (gen II) Lumi: An Android tablet that's also, on top of that, an E-ink screen in ~tech

    skybrian
    (edited )
    Link
    It sounds like it's working out for you so far? For people who buy non-mainstream devices with limited app support, it seems like having the option of using AI to improve the apps will be a big...

    It sounds like it's working out for you so far? For people who buy non-mainstream devices with limited app support, it seems like having the option of using AI to improve the apps will be a big win. The manufacturers should lean into that.

    (Unlike those other vendors that bother satisfied customers by putting AI where nobody needs it and interrupting you to tell you about it.)

    4 votes