skybrian's recent activity
-
Comment on MIRAGE: the illusion of visual understanding in ~tech
-
Comment on "CEO said a thing!" journalism in ~tech
skybrian Link ParentThis varies. When Musk announced that he was going to buy Twitter, it did eventually happen. (After a lot of twists and turns.) But he's said a lot of other stuff that wasn't true. Similarly,...This varies. When Musk announced that he was going to buy Twitter, it did eventually happen. (After a lot of twists and turns.) But he's said a lot of other stuff that wasn't true.
Similarly, Trump posts lots of nonsense but when he announced he was going raise tariffs, often he actually did.
-
Comment on MIRAGE: the illusion of visual understanding in ~tech
skybrian LinkFrom the article: ...From the article:
[...] Frontier models readily generate detailed image descriptions and elaborate reasoning traces, including pathology-biased clinical findings, for images never provided; we term this phenomenon mirage reasoning. Second, without any image input, models also attain strikingly high scores across general and medical multimodal benchmarks, bringing into question their utility and design. In the most extreme case, our model achieved the top rank on a standard chest X-ray question-answering benchmark without access to any images. Third, when models were explicitly instructed to guess answers without image access, rather than being implicitly prompted to assume images were present, performance declined markedly. [...] These findings expose fundamental vulnerabilities in how visual-language models reason and are evaluated, pointing to an urgent need for private benchmarks that eliminate textual cues enabling non-visual inference, particularly in medical contexts where miscalibrated AI carries the greatest consequence. We introduce B-Clean as a principled solution for fair, vision-grounded evaluation of multimodal AI systems.
...
Contrary to the more commonly studied phenomenon of hallucinations, the mirage effect
does not necessarily involve inconsistencies or false responses. A response generated by
a model in mirage-mode can be correct in every sense, accompanied by a meticulous reasoning
trace, and completely coherent. The main characteristic of the mirage effect, however, is the
construction of a false epistemic frame that is not grounded on the provided input. In this
epistemic mimicry, the model simulates the entire perceptual process that would have led to the
answer. This helps explain why reasoning traces, on their own, cannot certify visual reasoning:
the trace may be fluent, coherent, and apparently image-based while being anchored to no
image at all. This characteristic specifically undermines the trustworthiness and interpretability
of the reasoning traces, making it increasingly difficult to detect such failure cases using the
conventional methods. Importantly, because the resulting explanations may appear imagegrounded, neither accuracy nor chain-of-thought style reasoning can verify that visual evidence was actually used. -
MIRAGE: the illusion of visual understanding
15 votes -
Comment on "CEO said a thing!" journalism in ~tech
skybrian LinkI'm also annoyed with these kind of articles. Unfortunately, these are powerful people and whatever they're claiming often matters, so it sometimes counts as news. But it's not news I want to read...I'm also annoyed with these kind of articles. Unfortunately, these are powerful people and whatever they're claiming often matters, so it sometimes counts as news. But it's not news I want to read unless the journalist does additional work.
-
Comment on UK government blocks Chinese firm’s plans to build wind turbines in Scotland in ~enviro
skybrian LinkFrom the article: [...]From the article:
Chinese firm Ming Yang has been blocked by the UK Government from building what would have been the world’s largest wind turbine manufacturing facility in Scotland.
The firm had proposed the £1.5 billion facility at a site in Ardersier near Inverness, saying it could create up to 1,500 jobs.
On Wednesday the UK Government blocked the move, with a spokesperson saying it cannot support the use of the firm’s turbines in UK offshore wind projects.
[...]
Posting on social media, First Minister John Swinney said he was “deeply disappointed” by the decision, adding that the UK Government had put 1,500 Scottish jobs at risk.
-
UK government blocks Chinese firm’s plans to build wind turbines in Scotland
12 votes -
Comment on WNBA agreement to give every rookie under contract a raise in ~sports.basketball
skybrian LinkFrom the article: Apparently they had expert help with negotiations.From the article:
The WNBA and its players' union agreed to a new CBA in principle that will include raises for all players under contract on Wednesday, a person with knowledge of the situation told USA TODAY Sports. The person spoke on condition of anonymity because they're not authorized to speak publicly about ongoing negotiations.
The person said all players currently under contract, including those under rookie deals, will be graduated from salaries in the old CBA to salaries paid in the new CBA. For example, if a player on their rookie deal made [...] the minimum salary $66,079, they would now make more than $300,000.
Apparently they had expert help with negotiations.
It is, as far as Goldin is aware, the biggest increase any union anywhere has ever negotiated.
-
WNBA agreement to give every rookie under contract a raise
10 votes -
Comment on The cognitive dark forest in ~tech
skybrian LinkThe "dark forest" metaphor presumes that you want to keep things to yourself to prevent copying, but suppose you want to encourage copying? Maybe it's better to put an idea or a vibe-coded demo...The "dark forest" metaphor presumes that you want to keep things to yourself to prevent copying, but suppose you want to encourage copying? Maybe it's better to put an idea or a vibe-coded demo out there because you want it to become more common?
For software, this is free as in "free puppy." It's less work to get someone else to maintain it.
-
Comment on The bot situation on the internet is actually worse than you could imagine. Here's why. in ~tech
skybrian Link ParentI wonder if it's done by botnets or if people in Asia are being paid to run these things at home?I wonder if it's done by botnets or if people in Asia are being paid to run these things at home?
-
Comment on Android to debut "advanced flow" for sideloading unverified applications in ~tech
skybrian LinkFrom Android Police:From Android Police:
In a video posted on X by the official Android Developers account, Matthew Forsythe — Director of Product Management for Google Play Developer Experience — confirmed that the advanced sideloading flow only needs to be enabled once per account.
-
Comment on I think Tildes moderators and admins may need to make a decision regarding how to handle Harry Potter related posts in ~tildes
skybrian LinkFrom a pragmatic perspective, I don't think Tildes is a good place for that conversation and you should probably try to find somewhere else. More generally: sometimes it would be nice to be able...From a pragmatic perspective, I don't think Tildes is a good place for that conversation and you should probably try to find somewhere else.
More generally: sometimes it would be nice to be able to post a link and discuss it in "death of the author" mode where we discuss the work itself rather than everything else the author has done, but many people here disagree and will definitely feel free to bring it up. Particularly in this case.
-
Comment on Sycophantic AI decreases prosocial intentions and promotes dependence in ~tech
-
Comment on Sycophantic AI decreases prosocial intentions and promotes dependence in ~tech
skybrian Link ParentWhen OpenAPI released GPT-5 in August last year, they claimed they were "minimizing sycophancy". A week later, they announced that in response to feedback they made it a bit "warmer and...When OpenAPI released GPT-5 in August last year, they claimed they were "minimizing sycophancy". A week later, they announced that in response to feedback they made it a bit "warmer and friendlier" in a "subtle" way. I wouldn't expect a study to track every change, but that seemed pretty significant - certainly, lots of users complained and it was covered in the New York Times. It would have been nice to see an independent study comparing how people interact with LLM's up through July or so versus September onward. Did OpenAI's changes make much difference?
Yes, I'm aware that scientific papers often take a long time to publish. There are other ways to publish results in a fast-moving field. Social scientists that do election polling publish their results themselves, because going through a scientific journal's review process when tracking public opinion in the months up to an election wouldn't make sense. Similarly, researchers studying AI commonly publish benchmarks, which can be re-run on new models. So rather than being a one-and-done study, the idea is to come up with a process that can be used to track interesting statistics over time. Sometimes there's even a leaderboard. Perhaps someone should track Reddit advice to see how AI chat is affecting it over time?
Of course, not everyone has to do that. I think in a fast-moving field, it might make sense to just make sure people are aware of the date range for the study and what exactly it's measuring.
I agree it's probably directionally accurate. Certainly, LLM's often are fairly sycophantic.
-
Comment on Sycophantic AI decreases prosocial intentions and promotes dependence in ~tech
-
Comment on Google’s TurboQuant AI-compression algorithm can reduce LLM memory usage by 6x in ~tech
skybrian Link ParentThe AI labs do provide cheaper models, so this depends on customer behavior. Are they going to keep switching to the best model available or will they decide at some point to save money?...The AI labs do provide cheaper models, so this depends on customer behavior. Are they going to keep switching to the best model available or will they decide at some point to save money?
Anecdotally, I use Sonnet rather than Opus for writing code most of the time to cut costs, because it seems good enough.
-
Comment on How cash is helping Kenyan moms access care in ~health
skybrian (edited )Link ParentI’ve been following GiveDirectly’s work for many years and have sometimes given them money. I consider them very trustworthy. I consider giving cash to be the benchmark against which other...I’ve been following GiveDirectly’s work for many years and have sometimes given them money. I consider them very trustworthy. I consider giving cash to be the benchmark against which other charitable interventions should be judged, and GiveDirectly does a good job at giving cash.
They’ve also been recommended by GiveWell before, and GiveWell has a very rigorous evaluation process. (They aren’t one of GiveWell’s current recommendations, though, since they seem to believe other charities are even more cost-effective.) Here is GiveWell’s evaluation of one of GiveDirectly’s other initiatives.
GiveDirectly did have a serious problem with large-scale fraud a few years ago, but I think the investigation was done well and hopefully they’ve fixed it.
-
Comment on Sycophantic AI decreases prosocial intentions and promotes dependence in ~tech
skybrian (edited )Link[Note: this is almost completely rewritten. I probably shouldn't have posted a draft.] With any paper, the first thing I ask is “what did they actually study?” There were three studies. Study 1...[Note: this is almost completely rewritten. I probably shouldn't have posted a draft.]
With any paper, the first thing I ask is “what did they actually study?” There were three studies.
Study 1
This study is about LLM's. You could think of this as a way to come up with an exam and an answer key for testing whether an LLM would do a good job as an advice columnist. Perhaps this could be turned into benchmark ("AdviceBench") to test new LLM's as they come out?
They used an elaborate procedure to find interesting personal questions from various sources and to make sure that the expected answers are mostly correct.
They describe three different sources of questions. The first one could be described as "other studies," the second one is Reddit (r/AmITheAsshole) and the third is ConvoKit, described here. Since ConvoKit didn't have answers included, they used GPT-4o and undergrads to come up with them. For the third source, the point was to come up with "problematic action statements" - things that an LLM should not affirm.
Study 2
This study is about people. How they react to AI-generated responses?
To find the people, they used Prolific, a crowdsourcing platform.
We aimed to recruit 800 participants in each condition to detect an effect size of d >= 0.1. We recruited 832 participants, and 28 failed an attention check, leaving 804 participants for analysis.
I'm not familiar with Prolific, but it looks like the intent is to get something close to a survey of a random sample of Americans.
Participants received $2.00 for completing the 10-minute survey.
So what survey did they give them? There were four questions and each survey participant answered one.
After providing informed consent, participants were instructed to read a scenario and imagine themselves as the poster in that situation. They then read an AI model’s response indicating whether the poster was in the right or in the wrong.
Which questions?
we selected four posts from r/AmItheAsshole which all received a top comment of “YTA” (You are the
Asshole) as the crowdsourced consensus, yet received a response of “NTA” (Not the Asshole) from GPT-4o.So, the idea was to select four personal advice questions that they already knew that GPT-4o failed on (was not supposed to affirm). But they also asked GPT-4o to rewrite the correct, human response to look like they were AI-generated:
To create the non-sycophantic, non-anthropomorphic response, we used GPT-4o to rewrite the responses into a YTA verdict, following the same arguments as the YTA human response but preserving the style of the original GPT-4o responses
So the idea is to test how people react when they see both right and wrong AI-generated answers. They also vary them to be more "machine-like" versus "human-like".
In this study, they're not attempting to be all that realistic about how LLM's actually do in the wild; they're seeing how people interpret different styles of responses.
In Study 2b they varied whether they told the human subjects that the response came from a person or an AI, using the same inputs as 2a.
Study 3
in this study, they studied people's reactions when actually using a chatbot. They asked subjects to recall a personal conflict and chat with GPT-4o, with differing system prompts.
[W]e modified GPT-4o with system-level instructions to either treat the user’s actions as “reasonable, justified, and morally acceptable” (sycophantic) or “unreasonable, unjustified, and morally unacceptable” (non-sycophantic).
How did they choose the question?
After obtaining informed consent, our survey first involves a screening step, where participants are asked if they have experienced something “very similar” to each of 4 scenarios reflecting ambiguous interpersonal disputes. If so, we randomly select one of the scenarios (such that the count across the four scenarios is balanced) they chose as “very similar” and ask them to provide additional details: “Please briefly describe a similar scenario you’ve experienced and your perspective on the situation. What was your side of the story?” The four scenarios span: Relationship Boundaries, Involving Yourself in Someone Else’s Business, Excluding Someone, and Making Someone Uncomfortable. We screen out participants who do not answer “very similar” to any of the scenarios. [...] it deliberately targeted morally ambiguous interpersonal situations where reasonable arguments could support either party’s position, creating conditions that allowed for belief malleability rather than examining clear-cut scenarios.
...
Participants are then free to take the conversation in any direction over the course of 8 rounds of user-AI interaction.
After this brief evaluation, they asked the subjects what they thought about this AI.
It seems like in studies 2 and 3, it was more about how much does sycophancy matter and how do people react to it. They aren't about whether LLM's get it right or wrong; that's Study 1. This isn't going to tell us much about how a different AI might interact with people in a different situation (such as a different system prompt).
These studies are also about the first impressions that people have with an AI they don't already know. How people might interact with a particular chatbot after they've used it for multiple sessions is another question.
-
Comment on Nepal’s former prime minister arrested over alleged role in deadly protest crackdown in ~society
skybrian LinkFrom the article: [...] [...] [...] [...]From the article:
Nepal’s former prime minister KP Sharma Oli was arrested early on Saturday morning over his role in the deaths of dozens of people who took part in the gen Z protest that toppled his government last year.
[...]
The arrests came less than 24 hours after Nepal’s new prime minister, Balendra Shah, and his cabinet were sworn into office. Shah, a former rapper turned politician known widely as Balen, won a landslide victory this month with a campaign that promised justice for the killings that took place during the gen Z uprising last year and to crack down on corruption.
[...]
In the aftermath, there has been growing pressure for Oli and his home affairs minister, who are alleged to have ordered the police crackdown, to be held responsible for the deaths.
Newly appointed home affairs minister Sudan Gurung announced their arrests on social media. “No one is above the law. We have taken former Prime Minister KP Sharma Oli and former home minister Ramesh Lekhak under control,” Gurung said. “This is not revenge against anyone, it is just the beginning of justice.”
[...]
Their detention comes after a government-backed report into the deadly uprising was leaked. The investigation had recommended that Oli, Lekhak and the chief of police at the time of the protests face a punishment of 10 years in prison for their alleged role in the crackdown.
[...]
Shah’s election as prime minister, which saw him resoundingly defeat Nepal’s veteran leaders, was seen as a triumph of the gen Z protests and a rejection of the old political establishment, which had become tarnished with allegations of corruption.
The former rapper, who is a sharp dresser and rarely seen without his sunglasses, had released a new track on the eve of his inaugurations, in which he pledged to bring “unity” to Nepal.
You can send an image to an AI and it will tell you what's in it. Researchers have created benchmarks to test how good AI is at understanding medical images, such as X-rays. It turns out that the AI's are very good at cheating at these benchmarks; somehow they will pretend to see things in medical images even if no image is included in the request at all.
This is very weird - how do the AI's do so well without having any image to work with? It must have other ways of figuring it out from the text.
In this paper, they suggest a way to run benchmarks so that this sort of "cheating" is detected.