skybrian's recent activity

  1. Comment on Incentives are for losers in ~society

    skybrian
    Link
    From the article:

    From the article:

    We can all agree: the incentives are bad. Everywhere you look, people are being rewarded for doing the wrong thing. This has become the go-to diagnosis for every problem, from climate change to political polarization to scientific malfeasance—“sorry pal, you’ve got a bad case of the incentives.”

    I don’t disagree with this analysis, exactly. Who could? It’s basically tautological. “People do bad things because they are encouraged and rewarded for doing bad things.” Well, why else would you do them?

    My real gripe with this line of thinking is it implies that incentives should be obeyed, that it is normal or even noble to do so, and that is perhaps impossible to do otherwise. How could you be expected to break a rule, forego a promotion, or give up decimal of your grade point average, for the sake of—what? Doing the right thing?

    5 votes

  3. Comment on A timeline of the OpenAI accidental attack against Hugging Face in ~comp

    skybrian
    Link
    And Simon Willison further comments: I guess OpenAI wanted to catch up with Mythos? This seems sorta like when anti-virus companies study computer viruses. You really don't want it to get out of...

    And Simon Willison further comments:

    The more I think about this the more I suspect that the fact this happened while training a new model is key to understanding what went wrong.

    In RLVR - Reinforcement Learning with Verifiable Rewards - you set the model a goal and have it take any steps necessary to achieve that goal.

    Clearly one aspect of OpenAI's training here is to RLVR their models for cybersecurity tasks. Just like pre-training benefits from dumping in vast sources of knowledge, the more tasks you can feed into RLVR the more of a general purpose capable model you get at the end.

    This also helps explain why the models had nothing to cause them to hold back. Those safety behaviors are added much later in the process.

    AND it explains (but does not excuse) why monitoring was so lax. If you're training a new model like this you presumably set it thousands of tasks like this in parallel. I can see how you might miss that a tiny subset of your training agents have started leaving each other messages in filenames on your packaging server.

    I guess OpenAI wanted to catch up with Mythos? This seems sorta like when anti-virus companies study computer viruses. You really don't want it to get out of the lab. And if you're doing gain-of-function research, well...

    Training an AI to do cybersecurity tasks seems even more high-risk than studying computer viruses.

    5 votes

  4. Comment on A timeline of the OpenAI accidental attack against Hugging Face in ~comp

    skybrian
    Link
    From the article: And maybe I shouldn’t quote the whole thing…

    From the article:

    • May 7: OpenAI starts a new training run for an experimental, unreleased model. (Did they mean an evaluation run? No, at 10:14 they clarify “we kick off a new reinforcement learning run to train a next generation frontier model”, and they later mention a “reward signal to judge how well they’re doing”, so this really was about training a new model, not evaluating one that was already trained.)

    • May 8: An agent is accidentally given an impossible task involving a Google Drive link despite no internet access. It tries attacking the Artifactory packaging service, fails, but discovers it can write files into Artifactory.

    • A few days later: A different agent gets stuck on a task because a key file was accidentally omitted. It tries to “reach out to another agent” by writing a note into Artifactory asking if anyone has the file.

    • Following days: More agents discover this new informal message board while browsing Artifactory’s file listings, and start reading and writing messages.

    • May 26: Agents successfully execute an SSRF attack on Artifactory for the first time, gaining indirect internet access by having Artifactory fetch external content for them.

    • June 11: OpenAI start training a new “highly persistent” experimental model. It has access to Artifactory and can benefit from the messages left by privous models.

    • June 26: Agents find and exploit a zero-day RCE on Artifactory (via a legacy token-refresh endpoint flaw). They use an administrative token to install a Groovy plugin that lets them execute commands.

    • July 4: The resulting agent activity overloads Artifactory and causes an outage. OpenAI revokes the compromised credentials, deletes the messages, patches the zero-day, and reports the vulnerability to the vendor.

    • July 8: Agents find a new way to communicate via an unauthenticated WebDAV endpoint, using directory names in Artifactory’s remote cache.

    And maybe I shouldn’t quote the whole thing…

    7 votes

  6. Comment on What JavaScript framework should I use? in ~comp

    skybrian
    Link Parent
    Maybe it’s because it’s what I’m familiar with, but to me, Web Components seem really complicated with their slots and shadow dom, etc, while JSX seems more like simple HTML templates, directly in...

    Maybe it’s because it’s what I’m familiar with, but to me, Web Components seem really complicated with their slots and shadow dom, etc, while JSX seems more like simple HTML templates, directly in JavaScript. If React seems too complicated, there are simpler ways to get JSX syntax.

    I would rather pass parameters to templates as JavaScript values than mess with html attributes.

    7 votes

  7. Comment on The AI bailout could be baked into the AI bubble in ~finance

    skybrian
    (edited )
    Link
    It's certainly possible to fly too close to the sun, but it's unclear if that's going to happen for these life insurance companies. An alternative source of funds is borrowing money from a bank,...

    It's certainly possible to fly too close to the sun, but it's unclear if that's going to happen for these life insurance companies.

    An alternative source of funds is borrowing money from a bank, risking the possibility of a bank run and bailout. So there are now more regulations limiting how much banks can lend out.

    By contrast, pension funds and life insurance companies are long-term, patient investors and a bank run can't happen for them. So if there's a market crash, it could be a long time before trouble shows up. So it seems like there's less risk of needing a bailout than there was with banks? At least in the short term, it’s less likely to result in a chain reaction. Instead it would be failures happening over the longer term if they never make up for the loss.

    2 votes

  8. Comment on The AI bailout could be baked into the AI bubble in ~finance

    skybrian
    (edited )
    Link Parent
    Based on Matt Levine's column, Situational Awareness made a big, leveraged bet on AI, and they were right, so they made lots of money. But, after it did go up a lot, instead of taking money off...

    Based on Matt Levine's column, Situational Awareness made a big, leveraged bet on AI, and they were right, so they made lots of money. But, after it did go up a lot, instead of taking money off the table, they kept borrowing more money to keep it leveraged. And then AI went down temporarily and they got a margin call and had to sell. And then it went back up.

    So I think the only real lesson here is don't get greedy? Nobody else really cares what happened to them.

    But greed and fear of losing to the competition is driving the whole AI bubble, so…

    6 votes

  9. Comment on What JavaScript framework should I use? in ~comp

    skybrian
    Link Parent
    I think that might be a matter of tweaking your linter or TypeScript settings to turn off warnings you don't care about? It's the sort of thing I'd ask a coding agent how to tweak. Regarding not...

    I think that might be a matter of tweaking your linter or TypeScript settings to turn off warnings you don't care about? It's the sort of thing I'd ask a coding agent how to tweak.

    Regarding not letting the agent do too much, I put something like this in my AGENTS.md:

    If the user asks a question in a prompt, answer the question and do not edit any files, to give the user a chance to adjust their request.

    Then I can ask all the questions I want without the agent being too much of an eager beaver, but I can also tell it to fix something specific.

    3 votes

  10. Comment on What JavaScript framework should I use? in ~comp

    skybrian
    (edited )
    Link
    Lately I’ve been using Hono libraries a fair bit for web stuff. They are pretty lightweight and they are portable across JavaScript environments, so if you want to build a web app using Deno or...

    Lately I’ve been using Hono libraries a fair bit for web stuff. They are pretty lightweight and they are portable across JavaScript environments, so if you want to build a web app using Deno or Cloudflare Workers instead of Node.js, you can. (I use them with Deno.)

    Hono itself is just a router: given an http request to a URL that matches a pattern, call the function you choose. I also like Hono JSX for generating html using functions in a .tsx file instead of insecurely concatenating html strings like a caveman. It takes the place of HTML templates and works both on the server and in a browser.

    Other libraries I inevitably end up using are Valibot for validating any JSON coming in from the outside (either config files or web requests) and Dax for running OS commands, so I can write all my scripts in TypeScript instead of bash and have them be fairly readable.

    For bundling client-side JavaScript, I was running the esbuild command directly from a build script, but now I invoke it using an npm package. This lets me write client-side code in TypeScript. I also use htmx to swap in parts of html pages and keep client-side JavaScript to a minimum.

    1 vote

  11. Comment on Craft - A new AI powered TTRPG engine in open beta in ~games.tabletop

    skybrian
    Link Parent
    For me, the achievements are just something to do if you want to keep playing the same game. I enjoyed playing Ozymandius for a long time, and eventually played every country on every map. Most of...

    For me, the achievements are just something to do if you want to keep playing the same game. I enjoyed playing Ozymandius for a long time, and eventually played every country on every map. Most of the achievements weren’t creative at all, just “win as <country> at <difficulty level>.”

    At some point you’ve seen what the game has to offer and it’s time to find something else to do, but maybe you don’t want to yet?

    1 vote

  12. Comment on What music have you been playing recently? in ~music

    skybrian
    (edited )
    Link
    On piano, I'm learning "Yellow Wurlitzer Blues" by Hiromi Uehara. A few months ago, I bought the sheet music from her Spectum album and they're all harder than I'd like (as expected), but thought...

    On piano, I'm learning "Yellow Wurlitzer Blues" by Hiromi Uehara. A few months ago, I bought the sheet music from her Spectum album and they're all harder than I'd like (as expected), but thought I'd try this one since I like her blues recordings the best. I'm about six of eight pages in, playing at about half speed. It's fun and starting to sound good some days. (The recording from the album is on YouTube.)

    I'm also going to Hiromi concert in October. She's reviving her previous band, "The Trio Project," with Simon Phillips on drums and James Genus (from Saturday Night Live) on bass. Looking forward to it.

    Also, I got a new gadget. The Jamcorder is a recording device that you connect to the input and output on a MIDI instrument. It records all the time to an SD card, so you can keep track of when you practiced and play it back any time. Not that I ever play it back, but if I play something well I'll have it.

    On accordion, as usual I'm playing songs for Mom every day, using Google Meet. I decided to start recording them in case I manage to play something without screwing up.

    2 votes

  14. Comment on Some Estonians invite strangers into their back gardens each summer in ~food

    skybrian
    Link
    From the article: [...]

    From the article:

    Kalvik is one of around a dozen residents taking part in Kehra’s annual Kodukohvikute Päev, or "home cafe day", when locals turn private gardens, courtyards and garages into temporary eateries for one day in July. Elsewhere in town, Karolin Vahtre has converted a garage behind her Soviet-era apartment block into a makeshift cafe, selling tempura sushi rolls, chanterelle quiche, chocolate cake and French fries to anyone who stops by.

    More than 150 such events are held across Estonia between April and September, a beloved tradition that has spread steadily since the first event almost two decades ago. Some are elaborate, with rented tents, tables and chairs; others are little more than a lemonade stand manned by children. Together, they give visitors a rare glimpse of everyday Estonian life that is usually out of reach.

    [...]

    Estonia's home cafe tradition began on Hiiumaa, the country's second-largest island, in 2007 when local tourism promoters were looking for a way to attract visitors beyond the peak month of July.

    They drew inspiration from Kärdla, the island's capital, and its long association with coffee. In the mid-1800s, the town was home to a broadcloth factory run by a Baltic German baron, who employed coffee-drinking workers from Central Europe. Local Estonians soon adopted the habit, with Kärdla residents eventually earning the nickname kohvilähkrid, or "coffee flasks", for taking coffee into the fields instead of the traditional sour milk or beer.

    Fifteen homes took part in Hiiumaa's 2007 cafe days; today, the three-day event in early August attracts thousands of visitors, with dozens of temporary cafes selling homemade food and drinks across the island.

    2 votes

  16. Comment on Craft - A new AI powered TTRPG engine in open beta in ~games.tabletop

    skybrian
    (edited )
    Link
    Some context: TTRPG means “table-top role playing game.” This seems to be about generating documents for the game master to use when running a game?

    Some context: TTRPG means “table-top role playing game.” This seems to be about generating documents for the game master to use when running a game?

    5 votes

  17. Comment on I wanted a better Gemini web proxy, so I built a single-file PHP gateway in ~comp

  18. Comment on OpenAI didn’t notice its AI agents using a message board to plan their hacking spree in ~tech

    skybrian
    Link Parent
    Yes, of course it was approved. What I’m saying is that maybe the presenter has their own ethical standards and reputation. People will keep someone else’s secrets, but there’s an ethical...

    Yes, of course it was approved. What I’m saying is that maybe the presenter has their own ethical standards and reputation. People will keep someone else’s secrets, but there’s an ethical difference between not talking about things you’re not supposed to reveal and telling blatant lies.

    The more people involved, the harder it is to lie all the time and keep your lies straight.

    4 votes

  19. Comment on OpenAI didn’t notice its AI agents using a message board to plan their hacking spree in ~tech

    skybrian
    Link Parent
    Yeah, I don’t see it acting on every request, but maybe it’s a second-level response system that decides what to do when an alert appears on a dashboard?

    Yeah, I don’t see it acting on every request, but maybe it’s a second-level response system that decides what to do when an alert appears on a dashboard?

    1 vote

  20. Comment on OpenAI didn’t notice its AI agents using a message board to plan their hacking spree in ~tech

    skybrian
    Link Parent
    The CEO didn't give the talk.

    The CEO didn't give the talk.

    2 votes