skybrian's recent activity

  1. Comment on TheoPhysEval: Assessing AI agents at autonomous theoretical physics problem solving in ~science

    skybrian
    Link
    Here is the abstract: From a Facebook post by one of the authors, Suvrat Raju:

    Here is the abstract:

    We present a suite of 50 unpublished research-level theoretical physics problems with verifiable answers. Without AI assistance, a typical problem in our collection could be solved by a world expert in about a day or less but would take considerably longer for experts in adjacent specialties. We test the ability of off-the-shelf AI agents to tackle these problems autonomously, without human hints. When given access to tools available in a realistic setting, including the ability to search the literature and write scripts for numerical and symbolic work, the best-performing agent, GPT-6-Astra harnessed in Codex, obtains the final answers with a near-perfect accuracy of 98% at an average cost under USD 3 per problem. On average, agents show only small changes in accuracy when denied web search and access to background research papers, suggesting significant memorization of the literature. These results suggest that state-of-the-art agents can accurately perform concrete calculations required across theoretical physics research. We comment on the broader implications of these findings and discuss possible explanations for the current lack of agent-led breakthroughs in theoretical physics despite their outstanding problem-solving abilities.

    From a Facebook post by one of the authors, Suvrat Raju:

    About 6 months ago, just for fun, I prompted a frontier agent with an unpublished problem whose answer I had worked out. The agent got the answer wrong but it was impressive that it managed to make any progress at all, and could solve the problem after a few hints.

    Praneeth suggested it might be fun to collect a suite of such problems along the lines of the mathematicians' First Proof assessment. Soon we had a small core group and started requesting experts across the world to contribute problems. Problem collection was a slow process, since we were making a significant ask of contributors. We wanted unpublished problems, with clean verifiable answers. So, our contributors had to either sacrifice potential publication elsewhere, or else undertake a small research project just for us.

    Even as problems started coming in, the agents started improving as newer models were released. We kept urging people to push up the difficulty of their problems. But, there is a ceiling to this difficulty since even world experts have only so-many problems lying about whose answers they know and haven't published.

    I don't know of any individual who can solve any significant fraction of our final collection of problems without AI assistance. For instance, I can solve the problems I contributed myself and understand a few others. Most problems don't even make sense to me.

    Our initial evaluation suggested that GPT-6-Astra's score was in the late eighties followed closely by Opus 5.5

    But, as we started to investigate the mistakes that the models might have made, the models, especially Astra, insisted that they were right and that it was the expert-supplied answers that were erroneous. When we went back to the experts it turned out, in multiple instances, that that the experts found some subtle errors buried deep inside their calculations. As they revised their answers, Astra's score started to inch up and it finally ended up with 98% (49/50).

    2 votes

  3. Comment on I teach at a business school. It’s time to shut them down. (gifted link) in ~life

    skybrian
    Link Parent
    Not sure what you mean. Who are they? Did they make the movie?

    Not sure what you mean. Who are they? Did they make the movie?

  4. Comment on Does Tildes welcomes “self-promoting” links to personal blog posts? in ~tildes

    skybrian
    Link
    I don't think it can be self-promoting when you're sharing an opinion and not promoting anything? On a link site, linking to your own blog post to share an opinion shouldn't count, unless you're...

    I don't think it can be self-promoting when you're sharing an opinion and not promoting anything? On a link site, linking to your own blog post to share an opinion shouldn't count, unless you're just linking and not engaging in conversation.

    6 votes

  5. Comment on I teach at a business school. It’s time to shut them down. (gifted link) in ~life

    skybrian
    Link
    From the article: [...] [...] [...] [...]

    From the article:

    When schools like Wharton and Harvard Business School were established, they were seen as overly vocational, and to build legitimacy their founders framed management as a profession, on par with law and medicine. But education in those fields rested on established bodies of knowledge — centuries of case law, natural sciences like biology — that management lacked.

    [...]

    The solution, devised in the 1920s by the Harvard dean Wallace Donham, was to borrow the teaching style used in law and medicine: the case study. It became the signature method of business education. To this day, professors assign 10- to 20-page descriptions of business challenges culminating in a decision point that students then debate in class. The method is pervasive: In 2024, Harvard Business School sold nearly 17 million case studies to various business programs around the world.

    [...]

    I wrote dozens of case studies during my decade at H.B.S., some of which are described as classics and best sellers. These are not products of independent research, but carefully packaged corporate narratives. The disclaimer printed in tiny type on every H.B.S. case concedes this point: Cases “are not intended to serve … as sources of primary data.” The data they include can be selective, biased or even fabricated.

    For the many cases I wrote under the guidance of senior faculty members, events that reflected poorly on the company were softened or omitted and the roles of certain executives were burnished. The companies held final approval, with drafts reviewed by their lawyers and P.R. staff. Rather than being vehicles for disinterested learning, as one might expect at a university, the case studies end up as instruments of reputation management.

    Corporate influence feeds a larger problem: The cases almost always feature winners, making it easy to mistake luck or timing for a proven strategy. Failures, dead ends and nonmarket solutions are rarely considered. Before Enron’s fraud was exposed, its business model was presented in admiring terms by at least five H.B.S. case studies. More damning still, the strategies that later led to its collapse — its asset-light model, its trading platform — were portrayed as innovations. Questions about deregulated energy markets, revolving-door oversight and opaque accounting went unasked.

    Furthermore, edits that push the case narratives away from reality are often meant to make it “work” in a classroom. Complexity is systematically shaved away, and dissenting voices often go unheard. Real organizations don’t move in clearly delineated phases toward a decision point, but business school teaching materials often do — compressing timelines, dropping messy subplots and smoothing over contradictions to produce clean narrative arcs and satisfying problems to debate.

    The result is a curriculum built around false narratives. Some of the most celebrated ideas in management have not been the ones that survived peer review or replication, but the ones that could be turned into TED Talks, consulting frameworks or airport best sellers. Faculty members are rewarded not for knowledge but for performance — selling a big idea with a memorable label that promises “success” or “disruption.”

    Business schools have a divided mission: They seek to legitimize an economic system focused on private gain while also developing guardrails to tame it. They found a remarkable way to manage this tension: the “win-win” story in which the conflict between private gain and public good simply evaporates.

    A classic win-win is the contention that environmental programs like reducing waste or using LED lights can save money or that decent pay and benefits lower costs by reducing employee turnover. Business schools now claim to educate “leaders who make a difference in the world,” as H.B.S. proclaims in its mission, while Wharton pursues a “quest for positive global impact.”

    [...]

    Recent evidence shows that many foundational studies supporting the win-win idea were based on flawed data or couldn’t be replicated. Yet faculty members keep researching, writing about and teaching the topic — suggesting an ideological project more than an academic exercise.

    [...]

    What would that clean-sheet logic mean across an entire university? One starting point is the organization theorist Martin Parker’s call to create “schools for organizing” rather than schools for business, centered on how humans coordinate for shared ends. Students would study co-ops, unions and social movements alongside the shareholder corporation. The central question would shift from “How do we maximize shareholder value?” to “How do different forms of organization distribute power, risk and reward — and which should we strengthen?”

    Dismantling the business school as a single high-status silo that professionalizes private gain may seem a heavy lift. But universities can create cross-disciplinary pathways for students who want to work in business without submitting them to the business school worldview. Some already offer “business and society” majors or master’s programs in organizations and public purpose, drawing on sociology, law and public policy. The point would not be to abolish the study of markets, but to return it to settings where questions of power, history, democracy and ecological limits cannot be so easily avoided.

    12 votes

  7. Comment on Looking for advice: How do I find out what stocks to invest into? in ~finance

    skybrian
    Link Parent
    Similarly, the 2008 recession was a great time to buy. But that’s easier to say in retrospect. You don’t want to wait around for another opportunity like that while the stock market goes up....

    Similarly, the 2008 recession was a great time to buy. But that’s easier to say in retrospect. You don’t want to wait around for another opportunity like that while the stock market goes up. Dollar-cost averaging seems reasonable.

    1 vote

  8. Comment on Looking for advice: How do I find out what stocks to invest into? in ~finance

    skybrian
    (edited )
    Link Parent
    That’s too strong a statement, even for actual gambling. Lottery winners are rare, but they exist. There are also plenty of people in Silicon Valley who became rich by holding company stock. I’m...

    That’s too strong a statement, even for actual gambling. Lottery winners are rare, but they exist. There are also plenty of people in Silicon Valley who became rich by holding company stock.

    I’m still a Boglehead in principle and recommend index funds. But in practice, holding Google stock has been very lucrative, and my investment strategy was procrastinating about diversifying.

    Nowadays, tech stocks are a large part of US index funds and I don’t see a reason to weight them more than the index funds do.

    5 votes

  9. Comment on How drones created a kill zone in the Ukraine war in ~society

    skybrian
    Link
    From the article: [...] [...] [...] [...] [...] [...] [...] [...] [...] [...]

    From the article:

    Henadii and Mykhailo spent more than six months in this so-called kill zone, holed up in a half-destroyed house. They could hear drones buzzing overhead nonstop, like traffic on a busy highway. To avoid detection, they relieved themselves in bags rather than step outside; slept pressed together in winter rather than light a fire; and dragged the bodies of Russian soldiers they had killed into their house, so drones overhead would not detect signs of fighting.

    [...]

    When Russia launched its full-scale invasion in 2022, the war looked markedly different. The battlefield had clear boundaries, similar to what generations of soldiers had known in wars past. The Russian and Ukrainian Armies massed troops and fought from trenches separated by a no-man’s land. Combat was confined to a narrow area, with troops resting safely a few miles behind the front.

    Then came the drones, with their constant surveillance, pinpoint strike precision and a reach that keeps expanding with technological improvements. As a result, the front has blurred into a miles-wide wasteland where neither side holds firm control, and where towns and villages that once stood safely behind the lines now bristle with danger.

    [...]

    Soldiers used to march to the front across open fields.

    Now, they crawl at night under thermal blankets to reduce their heat signatures.

    [...]

    The kill zone has also transformed life at the front. Missions that once lasted a few days stretch into months because rotating troops under a drone-filled sky is too dangerous.

    Henadii, 52, and Mykhailo, 51, described those months as hell. With vehicles unable to reach the front, food and water were dropped by drones. The drops sometimes failed, leaving them hungry and thirsty. Each lost 40 pounds. No medic could treat Mykhailo’s injured heel; his unit could only fly in medicine. For fellow soldiers badly wounded and in need of evacuation, the only option was often a robotic ground vehicle sent to carry them out.

    As technology advances, experts say, the kill zone will grow only wider and deadlier. Drones flying farther will push the fighting into cities once considered safe. Artificial intelligence will sharpen targeting systems, shortening the time between detection and strike. The biggest cost, commanders say, will be borne by the soldiers on the ground, whose presence in the kill zone remains as crucial as ever to capture and hold land.

    [...]

    The mission of the commander, whose call sign is Stepanych, is to create a lethal cordon, impassable for Russian troops in an area near Henadii’s and Mykhailo’s position in Primorske. He has deployed attack drone crews roughly every two miles along the 10-mile stretch of front that is under his command. Together, they launch about 100 drones a day, keeping several in the air at any given time so they can be redirected to new targets within minutes.

    The Russian Army, Stepanych said, “does the same thing.” Russian battlefield reports reviewed by The New York Times show that, on a single day in May, Russia had 199 reconnaissance drones ready to be launched near Primorske, with at least 62 Russian crews operating short-range attack drones in support.

    [...]

    To reach their position, they followed a route mapped in advance that kept them off paths more visible from the sky. Overhead, a Ukrainian reconnaissance drone tracked their progress, streaming live video to officers in the rear who guided them by radio. Intercepted Russian drone feeds helped the officers warn them when enemy drones approached.

    [...]

    Soldiers no longer cluster in trenches — an easy target for drones. Instead, they spread out in small groups, hiding in underground shelters. A position is often held by only two soldiers, as shown in the Russian battlefield reports reviewed by The Times, which detail Russian troop deployments near Kamianske, a village south of Primorske.

    The front has become so fuzzy that opposing troops sometimes shelter in the same building.

    [...]

    From their hideout, Henadii and Mykhailo repeatedly watched Russian troops advancing in small assault groups of only a few soldiers to minimize detection. Not long ago, they would have opened fire. But in the kill zone, using your rifle can be a deadly mistake, revealing your position to drones overhead.

    So they operated as reconnaissance troops, watching for Russian soldiers and radioing their movements. Drones would do the killing.

    “Neither side wants their infantry doing too much of the fighting,” said Rob Lee, a senior fellow at the Foreign Policy Research Institute who studies the war in Ukraine. “They want drones to do the fighting.”

    [...]

    The rise of drones has fueled a debate in Ukraine over whether they could one day replace infantry altogether.

    Dmytro Putiata, a former Ukrainian drone operator, said drones were a formidable force multiplier but would never fully replace infantry. Drones can destroy positions, he said, but cannot seize them. Only soldiers can dislodge troops from entrenched dugouts. As for land drones armed with machine guns that have joined some assaults, he asked: “What happens when the battery dies?”

    [...]

    An infantry company commander in Henadii’s and Mykhailo’s brigade said his unit had stopped delivering food to frontline positions by vehicle in November, switching entirely to drones.

    Russia’s Army also appears to have made the shift. On a single day in mid-May, drones made 678 flights across a large section of the southern front, delivering roughly 4,500 pounds of cargo — mostly food, water and ammunition — according to the battlefield reports obtained by The Times.

    [...]

    The journey home would be the most dangerous part of their deployment. The kill zone had expanded so much in six months that a mile-long walk in had transformed into a six-mile trek out. It would take them days.

    To improve their chances, Maksym, their commander, had planned the route meticulously. He had identified roads poorly visible from the sky and marked houses every 150 yards where they could take cover. Supply packages were dropped in advance at each overnight stop.

    5 votes

  11. Comment on AirTM gains ground in Venezuela despite Nicolás Maduro’s blockade (2020) in ~finance

    skybrian
    Link
    From the 2020 article: [...] [...] [...] Followup: the payments did happen. AirTM came to my attention because GiveDirectly used AirTM to transfer money to Venezuelans after the June 24th...

    From the 2020 article:

    Despite efforts from the Nicolás Maduro regime to block its access, digital wallet provider AirTM continues to gain ground in Venezuela and is set to start processing payments worth millions of dollars from Interim President Juan Guaidó’s team to ill-paid healthcare workers, company founder and Chief Executive Officer Ruben Galindo said.

    Speaking in an interview with el Nuevo Herald, Galindo said that all of the 60,000 doctors, nurses and other healthcare workers in Venezuela have already opened up accounts to start receiving payments through the online platform even though the regime has ordered local Internet providers to impede its access inside the country.

    [...]

    “The ‘health care heroes’ and all of our users have taken steps to evade the censorship,” Galindo said. “Many of them have downloaded access to VPNs (Virtual Private Networks) to evade the restrictions imposed by the dictatorship through the Venezuelan Internet providers.”

    [...]

    Guaidó promised Venezuela’s health workers, some of whom are paid less than $5 a month, a $100 monthly bonus to help compensate for the harsh conditions they face as they battle the coronavirus pandemic.

    The payments, which could initially add up to about $18 million, would come out of the funds seized abroad from the regime.

    Venezuela is by far AirTM’s largest market, representing about half of the company’s operations, as thousands of its citizens living abroad use the company’s services to send remittances to their loved ones suffering under the collapsing economy, Galindo said.

    The Mexico City-based digital wallet provider is also used by Venezuelan freelance workers inside the country to collect wages from their foreign employers, bypassing the restrictions to foreign exchange transactions imposed by the regime.

    [...]

    The payments to Venezuelan healthcare workers could provide an additional $5 million per month, according to press reports.

    Followup: the payments did happen.

    The Health Heroes Aid program met its goals and concluded in January 2021.

    • 59,740 verified doctors and nurses paid
      $17,799,800 digital dollars delivered
      177,995 individual payments to Venezuelan nationals

    AirTM came to my attention because GiveDirectly used AirTM to transfer money to Venezuelans after the June 24th earthquakes:

    How we sent cash after Venezuela’s twin earthquakes using a payment route others thought was impossible

    GiveDirectly had never worked in Venezuela. We had no office, local team, or established payment route—and needed to build a response from scratch in a country where aid organizations face significant barriers.

    Authorities have historically restricted cash assistance and closely scrutinized programs funded from abroad. A 2024 law added further restrictions on nonprofit organizations. Sanctions also complicated moving money into the country, limiting the banks and payment providers we could work with.

    ...

    We found a way to deliver cash through Pago Móvil—the local mobile payment system other responders thought was off-limits. Recipients received local currency directly in their existing accounts, without needing a new app or crypto wallet.

    ...

    This was GiveDirectly’s first use of digital currencies for humanitarian cash transfers.

    ...

    Pago Móvil offered a familiar way for recipients to receive money. The challenge was connecting to it from outside Venezuela while meeting sanctions requirements and finding a reliable supply of local currency. Typically, it would require access to a local bank, which is why many thought it wasn’t possible. So instead, we built a route through two providers: Decaf, a platform for sending stablecoin payments, and AirTM, a U.S.-registered money services business whose marketplace connects payments to local currency.

    It seems like AirTM was the key to the whole thing? AirTM has "cashiers" who accept cryptocurrency (USDC, tied to the US dollar) and send local payments. Here is AirTM's page on becoming a cashier. Most ordinary people don't touch cryptocurrency, but it's playing a role in these payment networks.

    1 vote

  13. Comment on Anthropic says revenue jumped 45% to $16.8 billion in third quarter in ~tech

    skybrian
    Link Parent
    That sounds bad for Yardi, but I don't see what that has to do with Anthropic, which actually is widely used. (I've never heard of Yardi, but apparently they are "a complete AI platform for real...

    That sounds bad for Yardi, but I don't see what that has to do with Anthropic, which actually is widely used.

    (I've never heard of Yardi, but apparently they are "a complete AI platform for real estate.")

    3 votes

  14. Comment on Anthropic says revenue jumped 45% to $16.8 billion in third quarter in ~tech

    skybrian
    Link Parent
    Well, trend lines aren't a surefire way to predict the future anyway. There's a learning curve that could go either way: People figure out how to stop wasting money on AI People figure out new...

    Well, trend lines aren't a surefire way to predict the future anyway. There's a learning curve that could go either way:

    • People figure out how to stop wasting money on AI
    • People figure out new ways to use AI

    Probably both at once, and it's hard to say which will win out. But I think we're only at the beginning of figuring out how to use AI. It's going to take years. Some uses might not be discovered until after AI improves some more.

    2 votes

  15. Comment on Anthropic says revenue jumped 45% to $16.8 billion in third quarter in ~tech

    skybrian
    Link Parent
    I sounds like you think that Anthropic's customers are other AI companies, or something. I'm wondering what evidence there is for that? I did find this Ramp article which suggests that AI spend...

    I sounds like you think that Anthropic's customers are other AI companies, or something. I'm wondering what evidence there is for that?

    I did find this Ramp article which suggests that AI spend per employee and overall token spend may have peaked.

    They also write:

    80% of spend on OpenAI and Anthropic is driven by 1% of firms.

    They do have a much wider customer base (Anthropic has ~40% of all firms in Ramp's data) but the whales are apparently spending a lot more.

    1 vote

  16. Comment on Anthropic says revenue jumped 45% to $16.8 billion in third quarter in ~tech

    skybrian
    Link Parent
    Well, sure, Oracle does look shaky. But I’m guessing most if not all of the big tech firms were founded during your lifetime. If you remember the the dot-com era, you should remember that Amazon...

    Well, sure, Oracle does look shaky. But I’m guessing most if not all of the big tech firms were founded during your lifetime. If you remember the the dot-com era, you should remember that Amazon was founded then. Do you think you would recognize another one if you saw it?

    With the revenue numbers they have and alleged profitability, Anthropic’s trajectory looks quite a bit like the rise of another big tech firm to me, for better or worse.

    10 votes

  17. Comment on Anthropic says revenue jumped 45% to $16.8 billion in third quarter in ~tech

    skybrian
    Link Parent
    I suspect this is going to age as poorly as the idea that "run-rate revenue" is suspicious.

    I suspect this is going to age as poorly as the idea that "run-rate revenue" is suspicious.

    9 votes

  18. Comment on Anthropic says revenue jumped 45% to $16.8 billion in third quarter in ~tech

    skybrian
    Link Parent
    Alternatively, subscription pricing might be sold at close to cost and the "API" rate might have massive profit margins? We'd need to know more about their actual costs to know which scenario is true.

    Alternatively, subscription pricing might be sold at close to cost and the "API" rate might have massive profit margins? We'd need to know more about their actual costs to know which scenario is true.

    9 votes

  19. Comment on Anthropic says revenue jumped 45% to $16.8 billion in third quarter in ~tech

    skybrian
    Link
    Also, Simon Willison comments on a quote from the Financial Times:

    Also, Simon Willison comments on a quote from the Financial Times:

    Anthropic told investors on Wednesday distillation accounted for 5 per cent to 7 per cent of revenue and that they had cut it back “materially” in the third quarter of the year, said four people with direct knowledge of the matter.

    Napkin maths says that if distillation was 5% of their Q2 revenue (before they cut it back in Q3) then unnamed "other labs" spent over half a billion dollars distilling Anthropic's models in that quarter!

    9 votes

  20. Comment on Anthropic says revenue jumped 45% to $16.8 billion in third quarter in ~tech

    skybrian
    Link
    The S-1 isn't out yet so this isn't the formal "road show" for the IPO. Apparently meeting with investors before filing is called "testing the waters." But it seems unlikely that the numbers...

    The S-1 isn't out yet so this isn't the formal "road show" for the IPO. Apparently meeting with investors before filing is called "testing the waters." But it seems unlikely that the numbers they're giving to investors would be much different from the numbers that show up in the S-1.

    I updated my chart to have a new tab for quarterly revenue. Also, there's a tab comparing the actual quarterly numbers to estimates based on previously leaked run rates. Looks pretty close! (The chart is AI-generated.)

    So, despite some skepticism from cynics, the leaked run rates don't seem to have been misleading after all? Everything's consistent.

    8 votes