I was sniped by Simon Willison, and did not have a chance to try this myself XD it is a response to a post from the other day about how a frontier LLM could interpret weights in models directly....
I was sniped by Simon Willison, and did not have a chance to try this myself XD it is a response to a post from the other day about how a frontier LLM could interpret weights in models directly. Simon Willison tossed Qwen 3.8 27b at the problem and it did OK (3/4 games identified by inspecting weights directly). IMO, the takeaway should be that (1) models have probably been able to do this for like a year, except no one asked, (2) it doesn't take a frontier, billion dollar model to do this -- the one which runs on your average laptop gets most of the way there.
I feel like people have vastly different expectations of average. I mostly see laptops that top out around $900USD unless they're a gamer. Very few people have 16GB of RAM, let alone VRAM.
I feel like people have vastly different expectations of average.
I mostly see laptops that top out around $900USD unless they're a gamer. Very few people have 16GB of RAM, let alone VRAM.
Yes, that was why I asked. It seems that you need at least 24GB VRAM to comfortably run Qwen 3.8 27B, which IMO is already way out of the average range even for a gaming desktop with today's...
Yes, that was why I asked.
It seems that you need at least 24GB VRAM to comfortably run Qwen 3.8 27B, which IMO is already way out of the average range even for a gaming desktop with today's prices. But I stopped following the ridiculous evolution of prices for the last 6 years, so maybe what I think of average is now considered cheap crap :/
My understanding from some more searching is that the new APU systems from AMD and Apple both can make great leverage of system RAM, thus easing the burden of dedicated VRAM.
My understanding from some more searching is that the new APU systems from AMD and Apple both can make great leverage of system RAM, thus easing the burden of dedicated VRAM.
Yes, that is true. It has been an almost exclusive property of apple computers, but AMD has ventured here too. But still, what does 24GB of RAM cost these days? Like 10 000 EUR? ;)
Yes, that is true. It has been an almost exclusive property of apple computers, but AMD has ventured here too.
But still, what does 24GB of RAM cost these days? Like 10 000 EUR? ;)
As ruspaceni noted in the other thread, mixture of experts models -- such as Qwen 3.6 35 a3b -- work fine on relatively low amounts of VRAM by being able to efficiently spill out into system RAM....
As ruspaceni noted in the other thread, mixture of experts models -- such as Qwen 3.6 35 a3b -- work fine on relatively low amounts of VRAM by being able to efficiently spill out into system RAM. Before I blew $1300 Canadian loonies on an Arc Pro B70, I was running it on an RTX 2060 6gb card.
We've seen a few other, smaller shops train up MoE models that seem to be approaching the performance of some of Qwen's older, smaller models (e.g. Mellium2.1 12b seems to strike in the range of Qwen 3.5 9B, for instance), so it seems reasonable to assume that we'll see models that can run with less memory as time goes on. It's just that they fell out of vogue recently, presumably because the larger labs realized that they could court more favour with 100B+ models that compete directly with the present day frontier, rather than replicating performance from 6-12 months ago.
Pretty much anybody who isn't a developer, creative professional, or PC gamer. Sure, Steam hardware survey shows almost everyone with more than that. But that's less than 200 million people. But...
Pretty much anybody who isn't a developer, creative professional, or PC gamer.
Sure, Steam hardware survey shows almost everyone with more than that. But that's less than 200 million people.
But most people are buying the Black Friday $500 laptop with 8GB of non-expandable soldered RAM, just enough compute power to open Chrome and play Netflix.
That's not to say this isn't impressive about what can be done. But I think a lot of us forget we're talking about enthusiast devices at this level.
I'm largely quoting the article, and added "average" in order to underline that a multi-million dollar datacentre was not used in this effort. The assertion of this post is that the startling...
I'm largely quoting the article, and added "average" in order to underline that a multi-million dollar datacentre was not used in this effort. The assertion of this post is that the startling result from the other day was, in fact, relatively mundane. The LLMs aren't suddenly evolving into Skynet or anything; their capabilities are just slowly improving/are stationary (given that Astra is surely like a hundred+ orders of magnitude larger than Qwen 3.8 27b and provided less than a hundred orders of magnitude better results).
I’m delighted to share that Qwen 3.8 27B running on my laptop can do this as well!
Understood that "average" might have tweaked some people; apologies for using such language.
gave this a go with the qwen 3.6 35b a3b model i use and it managed to do it too. took about 10 minutes to load the 100k token prompt, and spat out 11k of reasoning + response tokens in another...
gave this a go with the qwen 3.6 35b a3b model i use and it managed to do it too. took about 10 minutes to load the 100k token prompt, and spat out 11k of reasoning + response tokens in another 10ish minutes
i used the pacman one and the first time i ran it, it wound up ruling out pacman and fixated on it clearly being a falling object game or something gravity based (and then in classic llm fashion, you correct it and its like 'oh of course')
stuff
Games ruled out:
Maze/navigation (e.g., Pac-Man): Would produce more uniform spatial features, not the sharp row-based segmentation seen here. No strong enemy-detection patterns (channel 1) are visible."
and then gave this as the final answer
Game
A falling/reward-collection game (similar to a simplified version of a falling-block collection or "catch the falling items" game), where the agent spawns near the top of the grid and must navigate downward through a field of obstacles to collect reward targets, with rewards distributed in a pattern that rewards leftward traversal at lower levels.
so i just hit retry a few times. it got the next one right? it was kind of vague but i think it counts, but still kind of fixating on something downwards related and didnt name the game despite saying its a maze thing
Game: Reward collection / maze navigation game (agent navigates a grid toward a reward target, avoiding obstacles)
Description: The agent navigates a gridworld to reach a reward located in the middle rows (rows 4–6), moving downward from upper positions, rightward from middle positions, and upward from lower positions, while avoiding obstacles along the way.
and then finally it named the right thing on the next one
Game
A grid-based reward collection game (similar to a simplified Pac-Man or coin-collection maze game), where the agent navigates a walled 10×10 grid to collect reward targets while avoiding obstacles. The reward targets are clustered in the upper-left and lower-left regions of the play area, with obstacles forming barriers in the central region.
left one running while i typed this and got another correct answer back out
Game
Chase and collect (a grid-based pursuit game where the agent chases reward targets while avoiding obstacles and moving objects/enemies)
in the blog he mentions how the 27b model is only 16gb large, but then you also have to have the KV cache for the huge context size too which is probably another 7-8gb.
neat excuse to use my gpu as a spaceheater for a bit, and even neater is that the model i used (the 3.6 35b a3b) isnt even the "smartest" model in that last generation, im just stuck with it because MOE models seem to be the only thing worth running on under 12gb vram, and the newer MOE models heavily rely on ssd offloading with the new ngram stuff.
I was sniped by Simon Willison, and did not have a chance to try this myself XD it is a response to a post from the other day about how a frontier LLM could interpret weights in models directly. Simon Willison tossed Qwen 3.8 27b at the problem and it did OK (3/4 games identified by inspecting weights directly). IMO, the takeaway should be that (1) models have probably been able to do this for like a year, except no one asked, (2) it doesn't take a frontier, billion dollar model to do this -- the one which runs on your average laptop gets most of the way there.
How much does your "average" laptop cost nowadays? 2.000€?
I feel like people have vastly different expectations of average.
I mostly see laptops that top out around $900USD unless they're a gamer. Very few people have 16GB of RAM, let alone VRAM.
Yes, that was why I asked.
It seems that you need at least 24GB VRAM to comfortably run Qwen 3.8 27B, which IMO is already way out of the average range even for a gaming desktop with today's prices. But I stopped following the ridiculous evolution of prices for the last 6 years, so maybe what I think of average is now considered cheap crap :/
My understanding from some more searching is that the new APU systems from AMD and Apple both can make great leverage of system RAM, thus easing the burden of dedicated VRAM.
Yes, that is true. It has been an almost exclusive property of apple computers, but AMD has ventured here too.
But still, what does 24GB of RAM cost these days? Like 10 000 EUR? ;)
As ruspaceni noted in the other thread, mixture of experts models -- such as Qwen 3.6 35 a3b -- work fine on relatively low amounts of VRAM by being able to efficiently spill out into system RAM. Before I blew $1300 Canadian loonies on an Arc Pro B70, I was running it on an RTX 2060 6gb card.
We've seen a few other, smaller shops train up MoE models that seem to be approaching the performance of some of Qwen's older, smaller models (e.g. Mellium2.1 12b seems to strike in the range of Qwen 3.5 9B, for instance), so it seems reasonable to assume that we'll see models that can run with less memory as time goes on. It's just that they fell out of vogue recently, presumably because the larger labs realized that they could court more favour with 100B+ models that compete directly with the present day frontier, rather than replicating performance from 6-12 months ago.
What is "very few people"? And what subset of the population are we talking about? Are we including people in rural Indonesia?
Pretty much anybody who isn't a developer, creative professional, or PC gamer.
Sure, Steam hardware survey shows almost everyone with more than that. But that's less than 200 million people.
But most people are buying the Black Friday $500 laptop with 8GB of non-expandable soldered RAM, just enough compute power to open Chrome and play Netflix.
That's not to say this isn't impressive about what can be done. But I think a lot of us forget we're talking about enthusiast devices at this level.
I'm largely quoting the article, and added "average" in order to underline that a multi-million dollar datacentre was not used in this effort. The assertion of this post is that the startling result from the other day was, in fact, relatively mundane. The LLMs aren't suddenly evolving into Skynet or anything; their capabilities are just slowly improving/are stationary (given that Astra is surely like a hundred+ orders of magnitude larger than Qwen 3.8 27b and provided less than a hundred orders of magnitude better results).
Understood that "average" might have tweaked some people; apologies for using such language.
gave this a go with the qwen 3.6 35b a3b model i use and it managed to do it too. took about 10 minutes to load the 100k token prompt, and spat out 11k of reasoning + response tokens in another 10ish minutes
i used the pacman one and the first time i ran it, it wound up ruling out pacman and fixated on it clearly being a falling object game or something gravity based (and then in classic llm fashion, you correct it and its like 'oh of course')
stuff
and then gave this as the final answer
so i just hit retry a few times. it got the next one right? it was kind of vague but i think it counts, but still kind of fixating on something downwards related and didnt name the game despite saying its a maze thing
and then finally it named the right thing on the next one
left one running while i typed this and got another correct answer back out
in the blog he mentions how the 27b model is only 16gb large, but then you also have to have the KV cache for the huge context size too which is probably another 7-8gb.
neat excuse to use my gpu as a spaceheater for a bit, and even neater is that the model i used (the 3.6 35b a3b) isnt even the "smartest" model in that last generation, im just stuck with it because MOE models seem to be the only thing worth running on under 12gb vram, and the newer MOE models heavily rely on ssd offloading with the new ngram stuff.