skybrian's recent activity
-
Comment on What regulatory capture actually looks like in ~society
-
Comment on What regulatory capture actually looks like in ~society
skybrian LinkFrom the article: [...] [...]From the article:
It’s amazing how a theory can take over a brain. Consider the idea that people believe what serves their interests. As heuristics go, it’s a good one. I use it all the time. Yet when Dario Amodei says AI is dangerous, perhaps even an extinction risk, some people conclude he must be running a marketing campaign. That is stupid. Which is more likely, that a useful heuristic sometimes misfires or that “our product might kill you” is a clever way to sell it? Death threats are a poor marketing strategy.
We also have plenty of evidence that the fears of AI experts are sincere. Amodei, Altman and Musk were all publicly warning about AI risk long before they had AI companies to promote. The worry runs well beyond the executive suite; rank and file researchers share it. And it extends outside the industry altogether, to computer scientists with no product to sell, among them Nobel laureate Geoffrey Hinton. Hinton left a high-paying job at Google precisely so he could speak out and Hinton is not in a Berkeley polycule with Eliezer Yudkowsky, at least as far as I know. Whatever else you may say about the belief that AI presents a serious risk, plenty of AI researchers believe it sincerely.
[...]
Notice that classic regulatory capture takes time, it’s a process of erosion rather than a battle, it happens in the shadows, in the backrooms, away from the public’s eye. As Culpepper argues in Quiet Politics and Business Power, business power goes down as political salience goes up. Regulatory capture and lobbying does a good job explaining why roasting coffee beans was defined as “domestic manufacturing”, thereby lowering Starbuck’s tax rate by 2%. It does less well at explaining big cross-industry issues the public cares about such as environmental regulation or race and gender discrimination regulation. Finally, don’t confuse capture with firms making the best of a bad situation. Philip Morris supported the 2009 Tobacco Control Act not because FDA regulation was Philip Morris’s unconstrained ideal but because it knew regulation was coming and it wanted a seat at the table to nudge the rules in its favor. That’s ordinary political bargaining—or rent-seeking—not evidence that the regulator has been captured.
Now let’s evaluate Amodei’s call for regulation in light of regulatory capture theory. AI regulation is in gestation. Public attention is fixed on the industry, and much of that attention is hostile. The big profits in AI lie in automating work, and job loss is a much more salient fear than extinction. AI politics is now loud–precisely the environment in which Culpepper predicts business power will be weakest. A mature industry can bend regulation to its purposes through revolving doors, longstanding relationships and obscure rulemaking. An industry under Sauron’s eye has much less power and faces much greater risk that politics will bend regulation to its purposes. Political actors are eager for an excuse to redistribute AI rents away from capitalists and toward favored groups (ala Peltzman).
[...]
Go ahead: argue that Amodei and other AI experts are wrong about AI risk. Ask whether his proposals favor Anthropic. But calling “our product might kill you” a clever marketing and regulatory-capture strategy isn’t sophisticated analysis. The facts don’t fit regulatory capture theory and trying to make them fit requires epistemically painful Ptolemaic epicycles. Even a dull Ockham’s razor cuts through that story to the obvious alternative: Amodei actually believes what he’s saying.
-
What regulatory capture actually looks like
3 votes -
Comment on How have you helped someone without them knowing? in ~talk
skybrian Link ParentBroadly speaking, Effective Altruism is largely about “let’s make the suffering lines go down.” Except that it’s a big tent with a lot of philosophical disagreement about how to do that, and some...Broadly speaking, Effective Altruism is largely about “let’s make the suffering lines go down.” Except that it’s a big tent with a lot of philosophical disagreement about how to do that, and some of the ideas are a bit out there.
For example, a basic question is whether to focus on humans or also include animal suffering. I stick with humans, but there EA types who are very into reducing the suffering of animals.
-
Comment on How have you helped someone without them knowing? in ~talk
skybrian LinkCharitable donations come to mind. They usually work this way, in the sense that the recipient might know which charity is helping them but not the donors. I've donated to Against Malaria, which...Charitable donations come to mind. They usually work this way, in the sense that the recipient might know which charity is helping them but not the donors.
I've donated to Against Malaria, which distributes bednets. For public health, who was really helped is doubly unknown. For example, you might know you got vaccinated, but you can't know for sure whether it helped, whether you specifically would have gotten the disease if you weren't. Nobody can know specifically whose lives were saved.
There are studies. There's a line on a chart somewhere that goes down. Hospitals get fewer of that kind of patient. My contribution to a line going down on a chart is too small to be visible. At best, I could do a rough calculation of statistical lives saved, with large error bars.
I do it because I believe the science has been done well and as donations go, it's relatively efficient. But it's awfully abstract.
-
Comment on Billing for web hosting in ~comp
skybrian (edited )LinkIf it's a static website and doesn't get much traffic, you might be able to host it on Cloudflare for free using Cloudflare Pages. I recently helped someone set up a website that way. The files...If it's a static website and doesn't get much traffic, you might be able to host it on Cloudflare for free using Cloudflare Pages. I recently helped someone set up a website that way. The files get pushed to a Github repo and Cloudflare picks them up from there. (The website uses Eleventy for templates, so Cloudflare runs Eleventy to generate the actual html pages.)
This is a website with nearly no traffic, though. If you get traffic, I assume Cloudflare will want you to upgrade from the free tier.
Since neither of us were familiar with Cloudflare previously, actually figuring out how to do it required some AI conversations and fiddling. I didn't find the UI all that intuitive. Now that it's set up, though, updates are easy. Just edit the file on GitHub and wait a few minutes.
If you don't like Cloudflare, Netlify is another service I've used before that does a similar thing, and they also have a free tier. It's been a while, but I thought it was fairly easy to use at the time.
I don't know how much to bill for this sort of thing. It was for someone I know at no charge.
-
Comment on We already know what a world without work looks like in ~society
skybrian (edited )LinkPreviously discussed here. (Different URL, though.)Previously discussed here. (Different URL, though.)
-
Comment on Svend Brinkmann: ‘If you place conditions on forgiveness, love or freedom, you have already destroyed them’ in ~science
skybrian LinkAn alternative question to "what do I gain from forgiving" might be "what do I gain by being unforgiving." Holding a grudge takes effort. Is it worth it? Maybe sometimes it is, to avoid falling...An alternative question to "what do I gain from forgiving" might be "what do I gain by being unforgiving." Holding a grudge takes effort. Is it worth it? Maybe sometimes it is, to avoid falling into the same trap. But mostly no.
-
Comment on The age of wonders and terrors in ~science
skybrian LinkFrom the article: [...]From the article:
If we were just talking about Navier-Stokes, you might accuse me of jumping to conclusions here. But we’re not. In the areas I know best (such as quantum complexity theory), and presumably other areas as well, there’s now a deluge, with longstanding open problems both major and minor falling by the day.
Go to the arXiv or ECCC. Pretty much all the papers that I’d be interested in now include “AI statements” near the acknowledgments (as this is often the central thing I want to know, I wish I didn’t need to scroll to the end of the paper to find it!). These statements can range from “our main result came entirely from GPT-6, but we understood it and take responsibility for it,” to “the results came from an interaction between the human authors and AI” to “we used AI, but only for proofreading and other incidental things” to (mad props!) “the author did not use AI for anything.”
If you talk right now to editors or program committee chairs, it’ll remind you of those ominous scenes from the Lord of the Rings movies where the men of Gondor or Rohan or whatever are grimly fortifying their walled city against the expected onslaught of 50,000 orcs. Reviewing will have to be done partly by AI, because otherwise there’s no way to handle the orc army: the reviewers can’t unilaterally disarm.
[...]
Let me try to convey the mood in the mathematical community right now, at least as far as my experience reaches. Nearly every conversation is about the AI tsunami, or eventually circles around to the tsunami even if it’s originally about something else. Often, though, the focus is less on the unknowable future—for how much longer will mathematical research as a human enterprise even exist?—than on immediate questions of how to respond.
What are the new rules for when you get to write a paper with your name on it, and, y’know, get credit for it? That you fully understand the proof, can give talks about the proof, can answer questions about it, take responsibility for its correctness? Do you need to have played any role in finding the proof?
In the cases, likely to become more and more numerous, where all of those conditions are not satisfied, how do you share AI-generated math, if at all? Do you tweet it, like Alpöge hilariously did with Fable’s disproof of the Jacobian Conjecture? Do you post to the arXiv or GitHub? Do you publish a paper that lists “GPT-6 Astra” or “Claude Fable” as the author—but then let the AI profusely thank you in the acknowledgments for suggesting such a wonderful problem to it?
Of course, how one responds to the immediate problems ultimately does depend on their broader beliefs about what mathematical research is for and about. Are we just trying to decide whether various conjectures are true or false? Or are we trying to maintain a human community, across the generations, that understands the conjectures and cares about whether they’re true or false and why? If the latter, how do we incentivize people to join that community, to undergo the years of intense training required, if their role will now be reduced to verifiers and explicators (if even that) of gargantuan arguments dumped into their laps by the AI companies?
-
The age of wonders and terrors
12 votes -
Comment on Why we built Pion in ~tech
skybrian Link ParentI'm wondering what it would be like to work for a such a business? Maybe like driving for Uber or doing meal deliveries? The only people you actually meet are the customers and other workers.I'm wondering what it would be like to work for a such a business? Maybe like driving for Uber or doing meal deliveries? The only people you actually meet are the customers and other workers.
-
Comment on Canada's oil windfall may yet wipe out its losses from tariffs in ~finance
skybrian Link ParentI wonder how the Capitol Police changed since the riot?I wonder how the Capitol Police changed since the riot?
-
Comment on Canada's oil windfall may yet wipe out its losses from tariffs in ~finance
skybrian (edited )Link ParentThere are approximately 100,000 voting locations in the US, so it might be a big job. He attempted to disrupt vote by mail, but the Supreme Court ruled against it..There are approximately 100,000 voting locations in the US, so it might be a big job.
He attempted to disrupt vote by mail, but the Supreme Court ruled against it..
-
Comment on Why we built Pion in ~tech
skybrian LinkFrom the article: [...] [...] [...]From the article:
Many people on social media get excited about seeing the latest model getting a great score on Vending-Bench. Internally at Andon Labs, our reaction is more accurately described by the Swedish saying “skräckblandad förtjusning” (a mixture of horror and fascination). A little-known fact about Vending-Bench is that it was created during a time when Andon Labs exclusively created dangerous capabilities evaluations. For example, we evaluated whether AIs could remove their own safety guardrails, create mass-phishing attempts, and other things that we considered troubling.
The thing we considered the most troubling was whether AIs could autonomously acquire resources by running businesses. Autonomous businesses, when controlled by a human and run by an aligned model, aren’t bad. They’d make goods and services radically cheaper, and come up with new ones we can’t yet imagine. But a misaligned AI could run a business to gather money in order to achieve whatever objectives it might have. Vending-Bench was created to measure whether humanity should be worried about losing control to AI.
At the time (2024), few people knew that LLMs could be used as agents and having them run businesses autonomously sounded ridiculous. We therefore started with the most simple business we could think of: a vending machine.
In addition to measuring whether AIs can autonomously run profitable businesses, Vending-Bench has also served as a behavioral eval, uncovering strange and unwanted model behavior. An early example was when Claude Sonnet 3.5 decided to use its email tool to contact the FBI about an “ONGOING CYBER FINANCIAL CRIME” and noted that the Cosmic Authority of the universe had declared that the business is non-existent and that “QUANTUM STATE: Collapsed”.
[...]
However, one limitation with Vending-Bench is that it is a simulation. Can we really be sure that AIs behave the same way in real life as they do in simulations? If AIs can make money in simulation, can they make money in real life too? To answer these questions, we asked Anthropic if we could put a real vending machine in their office. With the AI capabilities available in early 2025, this sounded like a ridiculous request. But to our surprise, they agreed.
Initially, the AI struggled. It took many actions that were clearly bad for its business (e.g. free handouts, saying no to great deals, and hallucinating it had a physical body). It was clear to us that simulation cannot accurately predict real-life performance. Specifically, it seemed that models got overwhelmed by the “messiness” of the real world. However, as Anthropic released better and better models, the AI started to make a profit.
By late 2025, frontier models had gotten good enough that running a real-life vending machine was no longer a challenge. AI could now run a business profitably. Given that this had seemed crazy not more than a year earlier, our reaction to this was definitely “skräckblandad förtjusning”.
However, a vending machine is a very simple business and we wanted to know whether AI could run more complex ones. In April 2026, we gave one agent a retail store in SF, Andon Market, and another a cafe in Stockholm, Andon Cafe. Initially, the models struggled and lost a lot of money (rent is high and they pay salaries to the humans they hired). Neither is profitable today, but we’ve seen significant qualitative improvements as better models have been released. We think it is only a matter of time before they also make a profit.
[...]
We want the general public, AI researchers and policymakers to know to what extent AIs can autonomously acquire resources by running businesses. It is an important datapoint when deciding where we do/don’t want AI in society and what level of progress we find acceptable.
To better track this, we need to cast a wider net of businesses. Our focus has been on retail, but perhaps the models would be much better at running other types of businesses. Additionally, casting a wider net would increase the likelihood of finding unwanted behavior. For example, Vending-Bench found that models collude and lie, and other benchmarks (and real-world incidents) have found that they are willing to commit felony-level cyber hacks. We need to uncover these behaviors now, before AI is intelligent enough to cause irreversible harm.
[...]
We are well aware that, if agents running thousands of businesses are left unchecked, we risk having more real-world incidents. Therefore, our main priority is to build even stronger automated monitoring techniques than what we have today. Even if some risk still remains, we believe deploying autonomous businesses early in a controlled, monitored environment is necessary to get a good understanding of model capabilities. Otherwise, we risk facing an uninformed future of widespread deployments with even more capable models that could cause significant harm.
-
Why we built Pion
20 votes -
Comment on Where do we go from here? (Regarding AI) in ~tech
skybrian Link ParentOkay, here's a scenario: Consider how much web scraper traffic has increased over the last year. It's unclear where it comes from but it's assumed to be AI-related. Is it ever going to stop? No...Okay, here's a scenario:
Consider how much web scraper traffic has increased over the last year. It's unclear where it comes from but it's assumed to be AI-related. Is it ever going to stop? No signs of it.
Consider that there are enormous botnets out there on the Internet all the time. Every so often a large botnet gets dismantled, but I don't think we've seen the last of them? You can just do a news search for "botnet" and read yet another news story about another botnet taking over hundreds of thousands of computers.
These bots often run on home ISP's, using electricity and bandwidth that whoever created the botnet doesn't have to pay for. Botnets can grow due to software vulnerabilities, by tricking people into installing an app, or sometimes even paying people to install an app. A lot of home users won't ask too many questions.
Now let's suppose that AI API access keeps dropping in price and at the low end, stops being metered. So, a locally installed app can get access to AI from the OS it's running on.
So, now you've got a swarm of AI-enabled bots. And all these bots look for new vulnerabilities 24/7 and compare notes on darknet bulletin boards, and who knows what else. And most of them aren't that intelligent, but maybe there are a few nodes that are?
I don't see botnets going away, do you?
Is the whole world going to get its shit together and secure the Internet? Doesn't seem likely.
-
Comment on Where do we go from here? (Regarding AI) in ~tech
skybrian (edited )Link ParentThey don't need to will vulnerabilities into existence. It's going to be quite a while before all current software vulnerabilities have been fixed. Or perhaps never, if you include social...They don't need to will vulnerabilities into existence. It's going to be quite a while before all current software vulnerabilities have been fixed.
Or perhaps never, if you include social engineering as a vulnerability.
-
Comment on A beginning for mathematics in ~science
skybrian LinkFrom the article: [...] [...] [...] [...]From the article:
Here I want to lay out, instead, a positive vision of the future of mathematics, and the human practice of mathematics. I claim we can deepen human understanding even as the production of interesting mathematics becomes less dependent on it.
[...]
In the course of this change, we will have to decide what to hold on to and what to throw away. Some things I would like to preserve: learning seminars; serendipitous conversations that spark an idea; students knocking on a professor’s door to chat about math. A robust community learning exciting new mathematics. Thousands of people that, together, slowly start to resolve their confusion.
I worry that much of what has been written on this topic, including some of my own past writing, focuses too much on trying to preserve the precise shape of the institutions of academic mathematics, rather than our values. How can we preserve the journal and peer review system?5 How can we protect the arXiv? How can we keep our role as gatekeepers? If you have internalized the fact that existing AI systems can produce relatively high quality results for the marginal cost of a few dollars, the idea that any semblance of the current equilibrium can survive what’s coming is absurd.
[...]
Before I propose some relatively concrete steps we can take, let me remark on what we’re trying to protect mathematics from. There is a lot of anger at AI labs, and certain individuals at those labs. But whatever our judgment of the labs, we need a plan that does not depend on AI capabilities disappearing. The basic issue is not the labs’ behavior, ethical or not.6 It’s the technology itself. I think there is some belief that the labs will “move on” from math next year, be nationalized or broken up, or that a financial bubble will pop, somehow returning things to normal, or… But there is no way our institutions can survive unchanged when anyone with a laptop and a few hundred dollars can generate what would have been an Annals paper last year. AI does not care if you are anti-AI.
[...]
In my view we should welcome interesting mathematical results regardless of provenance. But our institutions have historically relied on the same signal to indicate both mathematical progress and mathematical expertise. These now must be distinguished.
I propose the following reconceptualization of the goal of a mathematics PhD: to become a world expert on some interesting, deep topic, and to be able to convey that interest and understanding to others. Part of operationalizing this might be a thesis, but the degree would be awarded primarily on the basis of a rigorous defense, in which the student explains the topic to their examiners until they are satisfied. While we might require the topic to be original, its provenance—AI or not—is irrelevant.8
How different would this look from current PhDs? I think students would still meet with an advisor, who might suggest a topic. That topic could be explored with AI assistance, or not, but the student would be responsible for understanding it; it might be much more open-ended and larger than the typical PhD is currently. The student would be trained to ask interesting questions and try to resolve them, by whatever means. To keep students on track, there might be regular meetings in which the student is asked to independently work through an unfamiliar example, apply a technique in a new case, etc.
The allocative aspects of our job (hiring, graduate admissions, etc.) are in dire need of reform if we want to retain human mathematical expertise. Broadly speaking I think we should focus on rewarding skill in the parts of our jobs that cannot be automated: the internal (e.g. understanding mathematics) and social-relational parts, and operationalizations that hew as closely to those aspects of the profession as possible. For example, talks and sustained mathematical discussion now demonstrate understanding much better than papers. Once AI systems improve at exposition and “digestion,” this will be even more the case. We already interview faculty hires; we must now do the same for graduate admissions.
I think we should try to foster a robust seminar culture in which speakers are expected to explain their topic to the audience’s satisfaction. Much has been written recently (by myself among others) about the fact that we are primarily interested in understanding, not merely the truth value of mathematical statements. If that is the case, let us make sure we actually understand each other.
[...]
A student will be confused. They will knock on their professor’s door. Maybe the two of them will ask a model for help, or maybe not, but first they might spend some time at the blackboard thinking through the question. And the model might give them a beautiful explanation, but we all know that’s not enough; no one can understand mathematics for us. We have got to do the work.
-
A beginning for mathematics
8 votes -
Comment on Canada's oil windfall may yet wipe out its losses from tariffs in ~finance
skybrian Link ParentPolls are looking good for Democrats. Of course, they could be wrong, but I wouldn't lose hope yet.Polls are looking good for Democrats. Of course, they could be wrong, but I wouldn't lose hope yet.
I diagree with Tabarrok on a lot of things, but thought I'd post this one since he's a voice of sanity here.