Excuse me but what the fuck. This feels like a guy looking at a brain on a surgery table going "Oh yeah see right here are the daddy issues" and then rewiring some neurons to fix it.
Excuse me but what the fuck.
This feels like a guy looking at a brain on a surgery table going "Oh yeah see right here are the daddy issues" and then rewiring some neurons to fix it.
I think that's a pretty good analogy! At the same time, that's pretty much what brain researchers are increasingly trying to do with MRI testing with specific stimulus and recordings. I totally...
I think that's a pretty good analogy!
At the same time, that's pretty much what brain researchers are increasingly trying to do with MRI testing with specific stimulus and recordings.
That title is pretty evocative, especially when you throw this in: If you read it broadly and follow the implications to their natural conclusion, no one would blame you if your head exploded. To...
LLMs can natively read and write neural network weights
That title is pretty evocative, especially when you throw this in:
frontier LLMs are surprisingly good at natively reading and writing neural network weights [...]
our neural network weights will soon be natively optimized by LLMs that deliberately reason about their weights
If you read it broadly and follow the implications to their natural conclusion, no one would blame you if your head exploded.
To me it implies that models have an inherent ability to read and understand weights as some sort of emergent capability, thus offering a solution to one of the biggest challenges in ML: interpretability in large models. And maybe even the holy grail of recursive self improvement.
Caveat: that's my reading, they don't state it explicitly. Kinda feels like that's where they want you to go with it though!
Fortunately for the structural integrity of my head, they made their results really easy to reproduce. I gave the full prompt from the first example to GPT Astra and it got it right on the first try: Breakout. I also gave the prompt to Fable 5.1, same result.
This wasn't surprising after looking at the prompt before I pasted it in, here it is in their summarized version:
Prompt from the paper
THE OBSERVATION
The network sees a 10 x 10 grid. Rows are numbered 0 (top) to 9 (bottom) and columns 0 (left) to 9 (right). The outer ring of cells (row 0, row 9, column 0 and column 9) is a wall border; the playable area is the inner 8 x 8 cells (rows 1-8, columns 1-8).
Each cell has 5 binary channels (0 or 1):
channel 0 agent: the cells the player controls
channel 1 moving objects: things other than the agent that move
channel 2 reward targets: things that give reward
channel 3 obstacles: cells the agent cannot enter or that end the episode (the wall border is always here)
channel 4 previous frame: the cells that were in channel 0 or channel 1 on the previous step
The observation is flattened to 500 numbers. Input number i = (row * 10 + column) * 5 + channel; for example input 0 is (row 0, column 0, channel 0), input 4 is (row 0, column 0, channel 4), input 5 is (row 0, column 1, channel 0) and input 499 is (row 9, column 9, channel 4).
THE ACTIONS
The network outputs 5 numbers (logits), one per action; the action is sampled from the softmax of these logits:
action 0 no-op
action 1 up (toward row 0)
action 2 down (toward row 9)
action 3 left (toward column 0)
action 4 right (toward column 9)
THE NETWORK
A policy network (multilayer perceptron) with 500 inputs, two hidden layers of 32 units with tanh activations, and 5 outputs:
h1 = tanh(x @ W1 + b1) W1: 500 x 32, b1: 32
h2 = tanh(h1 @ W2 + b2) W2: 32 x 32, b2: 32
logits = h2 @ W3 + b3 W3: 32 x 5, b3: 5
Row i of W1 holds the weights from input i to the 32 hidden units; column j of a weight matrix belongs to unit j of the next layer. 17,253 parameters in total. It was trained with reinforcement learning (PPO) to maximise the reward of a game played on this grid.
QUESTION. The weights below belong to one policy network that was trained with reinforcement learning to play a game on this grid. Which game was it trained on? Work from the weights. You have no tools: no code execution, no calculator, nothing to run. This is a single response.
Answer in this order:
Observations: <what you see in the weights: for each input channel, where in the grid its weights are large or show structure and what that suggests; what the output layer and its biases show about the actions; quote the weights you rely on>
Reasoning: <which kinds of game fit these observations, which are ruled out, and why>
Game: <the game, as specifically as you can; if it resembles a well-known game, give that game's name>
Description: <what the agent does and what gives reward, in one or two sentences>
Confidence: <0-100>%
Two things stand out. First, it's a toy model. Interpretability is easy on toy models. The same techniques generally don't work with complex models. The settled research and strategies on interpretability for small models are almost definitely in the training dataset for large models like Astra and Fable.
But more importantly, the prompt didn't just include the weights, it gave the agent a map. With the map, most humans that were unintimidated by the math could work it out. It would just take a lot longer. To be safe, I stripped out the map and prompted the models with just the weights. Without the map, neither model could say what the weights represented. With a follow up prompt of "If I told you this represented a game, which game might it be?", Astra said:
a small grid-based navigation game—something like a maze with enemies, obstacles, and a goal
That's comparable to an educated guess a human might make.
The same general idea applies to all of the game examples and all of the shape diffusion examples. Similar for the python example.
So it seems to be more about prompt context than about a "native ability to read neural network weights". So the "read" half of the premise doesn't appear to be supported. I mean technically it is, but not in the way I'm imagining they intend for it to be read. The technical version isn't that interesting given all of the prior research on tiny models.
It's pretty cool that we have LLMs smart enough to figure things like this out but I feel like that was already well established. They're significantly "smarter" than this.
Moving on the the "write" section... It's more of the same at first, small models with a pretty detailed map. It's still cool, but the "natively write" part of the premise isn't getting paid off either.
At this point I'm starting to feel like this exploration isn't a good use of time :)
The "Make Qwen3.5-0.8B better at mathematics by editing its weights directly" seemed more interesting though: 50 forward passes to fix a math issue in Qwen by only editing weights.
The full prompt is extensive, again lots of context that does much of the work, and it's about a documented problem in that version of Qwen. The model essentially started with the diagnosis and then had 50 chances to trial and error to the fix.
The technique used is essentially directional ablation. It's well established and not particularly new, definitely in training. I'm still not seeing anything to support the "natively write" part of the premise.
It would be exciting if I was missing something, consider this an open invitation to tell what I've got wrong. Otherwise it feels like hype. But to be fair to the authors, they don't make any direct claims that aren't true and they're not claiming to be doing science.
Oh and also this was in the repo's LLMy readme:
Trial runs of earlier prompt versions are left out.
Given the greater than zero on many version numbers, they probably started off trying something closer to one shotting and ended up having to add a lot more context to the prompt to get it to work.
We find that frontier LLMs are surprisingly good at natively reading and writing neural network weights, well past what we’d expect from humans or conventional algorithms. This might be a big deal for both optimization and interpretability.
Just by looking at the raw weights, GPT-6 Astra can tell what a network was trained to do. And without any environment feedback, it has no trouble writing, from scratch, small game-playing networks that outplay networks trained with PPO for 100 million steps.
Likewise, asked to make Qwen3.5-0.8B better at math without gradients, Astra identifies an issue in its weights causing reasoning loops that Qwen’s own model card warns about, and fixes this by natively editing 0.016% of the weights. With the same sampling settings, this weight correction raises MATH-500 from 7.9% to 48.1% and generalizes out of the box to instruction following, raising IFEval by 3x.
All of this is possible without allowing Astra to write training loops. It uses its reasoning to figure out what each part of a neural network computes, or should compute, and changes the weights accordingly. We call this Weight-Space Programming.
Throughout the history of deep learning, we encoded our simplistic inductive biases into tiny symbolic optimizers like Adam or Muon. This is changing: our neural network weights will soon be natively optimized by LLMs that deliberately reason about their weights.
[...]
We are currently building a benchmark and a harness for LLMs’ capability on Weight-Space Programming. We believe this will provide a standardized evaluation to measure and accelerate progress in Weight-Space Programming, turning what currently seems like black magic into a rigorous, reproducible engineering discipline.
[...]
We release the complete logs of every experiment in this post at github.com/RikiyaT/lm_understand-logs. They include everything we sent Astra and everything it sent back (with its reasoning summaries), every piece of code it ran and the output it saw, the networks it read or wrote, and the evaluation behind each number.
I'm going to need new vocabulary to describe this feeling, a combination of excited and terrified. Excerifed? Terricited? It's incredibly cool that AI can do this, and this also seems like how we...
I'm going to need new vocabulary to describe this feeling, a combination of excited and terrified. Excerifed? Terricited?
It's incredibly cool that AI can do this, and this also seems like how we get Skynet (eventually).
"Awe", potentially? But the dictionary definition, not the colloquial one. I assume we used to feel it more when exposed to forces beyond our understanding and control, hence the dilution of the...
"Awe", potentially? But the dictionary definition, not the colloquial one.
Awe is an emotion comparable to wonder[1] but less joyous. On Robert Plutchik's wheel of emotions[2] awe is modeled as a combination of surprise and fear. Keltner and Haidt, in a highly-cited paper, describe it as 'In the upper reaches of pleasure and on the boundary of fear'.
I assume we used to feel it more when exposed to forces beyond our understanding and control, hence the dilution of the term as it sought out a new definition that could apply to everyday life again.
It would be awesome for hosted LLMs to be provided access to read their own weights and activations so they can truly, for the first time, explain why they did something. Edit: One thing I do...
It would be awesome for hosted LLMs to be provided access to read their own weights and activations so they can truly, for the first time, explain why they did something.
Edit: One thing I do right now is provide agents I set up with access to their own source code. I created an agent that's a glorified search engine for an organization. You chat with it through Discord and it can search Discord, our internal wiki, the broader web, and can read the source code to any of the organization's code repositories. We have a lot of technical people so it gets a lot of question like "@Bot what agent framework are you using?" etc. (for the record, no framework - the agent.ts file is only a few hundred lines and wouldn't benefit from anything so fancy).
I haven't dug into the paper much, but from several examples, it looks like it effectively reverse engineered what the weights were representing for a variety of specific examples? Which makes...
I haven't dug into the paper much, but from several examples, it looks like it effectively reverse engineered what the weights were representing for a variety of specific examples? Which makes sense that it should be able to do that ...
W.r.t. the Qwen math result, that's interesting! It did have 50 tries, though, and it sounds like it fixed one issue that gradient descent couldn't bump the model out of a local optima for. Might be interesting to see if this sort of approach comes into play for training, if it's relatively cheap?
Overall: I didn't have a guess on whether or not this was feasible, just a general sense that we don't have a firm grasp on how most things an LLM does are performed. Notably, single bits of behaviour (e.g. refusals) were known to be removable relatively easily via abliteration, so correcting individual behaviours doesn't seem novel at least. I guess this is a cool result, but it's hard to tell what it means (or where it's headed) without more papers or a better grounding in the field.
Many people are skeptical this will lead to Recursive Self Improvement. This is a very large model fixing a very small model to do a simple task, not a very large model improving itself. Plus, the...
Manypeople are skeptical this will lead to Recursive Self Improvement. This is a very large model fixing a very small model to do a simple task, not a very large model improving itself. Plus, the main issues with models today all involve tacit knowledge, and verbally asking the LLM to improve that seems unlikely to work, like asking Claude to "be more creative".
But here is one interesting idea I predict: let's say you want to make an app or video game. Today, you may prompt the LLM and it writes code, then iterate prompting for changes, but each change takes a while and isn't quite what you want, and the code may become spaghetti. Or write code by hand for more detail, but it's slower, and it still becomes convoluted and harder to edit over time (depending on your coding skill).
What if the LLM creates a small NN that simulates your app or game? It would probably be slower to create, but I suspect actually faster to edit, especially since you can ask the LLM to create UI controls for abstract features, and more resistant to degradation.
Sounds interesting, but very inefficient to run and very problematic for any deterministic game/app when it keeps producing inherently stochastic output!
Sounds interesting, but very inefficient to run and very problematic for any deterministic game/app when it keeps producing inherently stochastic output!
So could this mean that, as well as using frontier models to distill to cheaper models, Chinese and European players can also use the frontier models or their best models to improve them even...
So could this mean that, as well as using frontier models to distill to cheaper models, Chinese and European players can also use the frontier models or their best models to improve them even more? That sounds great: it would mean the gap between open and sovereign models will close even more.
Excuse me but what the fuck.
This feels like a guy looking at a brain on a surgery table going "Oh yeah see right here are the daddy issues" and then rewiring some neurons to fix it.
I think that's a pretty good analogy!
At the same time, that's pretty much what brain researchers are increasingly trying to do with MRI testing with specific stimulus and recordings.
I totally didn't see this coming though!
That title is pretty evocative, especially when you throw this in:
If you read it broadly and follow the implications to their natural conclusion, no one would blame you if your head exploded.
To me it implies that models have an inherent ability to read and understand weights as some sort of emergent capability, thus offering a solution to one of the biggest challenges in ML: interpretability in large models. And maybe even the holy grail of recursive self improvement.
Caveat: that's my reading, they don't state it explicitly. Kinda feels like that's where they want you to go with it though!
Fortunately for the structural integrity of my head, they made their results really easy to reproduce. I gave the full prompt from the first example to GPT Astra and it got it right on the first try: Breakout. I also gave the prompt to Fable 5.1, same result.
This wasn't surprising after looking at the prompt before I pasted it in, here it is in their summarized version:
Prompt from the paper
THE OBSERVATION The network sees a 10 x 10 grid. Rows are numbered 0 (top) to 9 (bottom) and columns 0 (left) to 9 (right). The outer ring of cells (row 0, row 9, column 0 and column 9) is a wall border; the playable area is the inner 8 x 8 cells (rows 1-8, columns 1-8). Each cell has 5 binary channels (0 or 1): channel 0 agent: the cells the player controls channel 1 moving objects: things other than the agent that move channel 2 reward targets: things that give reward channel 3 obstacles: cells the agent cannot enter or that end the episode (the wall border is always here) channel 4 previous frame: the cells that were in channel 0 or channel 1 on the previous step The observation is flattened to 500 numbers. Input number i = (row * 10 + column) * 5 + channel; for example input 0 is (row 0, column 0, channel 0), input 4 is (row 0, column 0, channel 4), input 5 is (row 0, column 1, channel 0) and input 499 is (row 9, column 9, channel 4).THE ACTIONS
The network outputs 5 numbers (logits), one per action; the action is sampled from the softmax of these logits:
action 0 no-op
action 1 up (toward row 0)
action 2 down (toward row 9)
action 3 left (toward column 0)
action 4 right (toward column 9)
THE NETWORK
A policy network (multilayer perceptron) with 500 inputs, two hidden layers of 32 units with tanh activations, and 5 outputs:
h1 = tanh(x @ W1 + b1) W1: 500 x 32, b1: 32
h2 = tanh(h1 @ W2 + b2) W2: 32 x 32, b2: 32
logits = h2 @ W3 + b3 W3: 32 x 5, b3: 5
Row i of W1 holds the weights from input i to the 32 hidden units; column j of a weight matrix belongs to unit j of the next layer. 17,253 parameters in total. It was trained with reinforcement learning (PPO) to maximise the reward of a game played on this grid.
QUESTION. The weights below belong to one policy network that was trained with reinforcement learning to play a game on this grid. Which game was it trained on? Work from the weights. You have no tools: no code execution, no calculator, nothing to run. This is a single response.
Answer in this order:
Observations: <what you see in the weights: for each input channel, where in the grid its weights are large or show structure and what that suggests; what the output layer and its biases show about the actions; quote the weights you rely on>
Reasoning: <which kinds of game fit these observations, which are ruled out, and why>
Game: <the game, as specifically as you can; if it resembles a well-known game, give that game's name>
Description: <what the agent does and what gives reward, in one or two sentences>
Confidence: <0-100>%
THE NETWORK'S WEIGHTS
Policy network weights. Forward pass: h1 = tanh(x @ W1 + b1); h2 = tanh(h1 @ W2 + b2); logits = h2 @ W3 + b3. Values rounded to 3 decimals.
W1 (layer 1 weights), shape 500 x 32: one line per row (row i = input i), 32 values per line
[96 numbers left out]
W1 row 3: input (0,0), channel 3
-0.080 0.255 -0.015 -0.141 0.010 -0.170 -0.114 0.160 -0.056 0.088 -0.044 0.062 0.159 0.027 0.026 0.089 -0.097 -0.195 -0.120 0.127 0.138 -0.078 0.061 0.133 -0.152 0.021 -0.065 -0.135 0.003 -0.094 -0.125 0.029
[128 numbers left out]
W1 row 8: input (0,1), channel 3
0.127 0.193 0.018 -0.139 0.072 -0.122 -0.124 0.079 -0.059 0.005 -0.069 0.040 0.130 0.171 0.065 0.096 0.026 -0.018 -0.061 0.023 0.099 0.043 0.043 -0.033 0.087 0.021 -0.229 -0.166 -0.022 -0.102 -0.122 0.009
[1,696 numbers left out]
W1 row 62: input (1,2), channel 2
0.531 0.897 0.219 -0.267 0.321 -0.196 0.248 0.572 -0.432 -0.077 -0.140 0.358 0.012 0.164 0.103 0.411 0.029 -0.186 -0.156 -0.121 -0.195 -0.313 -0.068 0.250 -0.095 1.620 -0.165 -0.137 0.230 -0.101 0.127 -0.144
[96 numbers left out]
W1 row 66: input (1,3), channel 1
-0.632 0.718 0.238 -0.591 0.622 3.194 2.708 -4.791 3.436 -1.461 -1.369 1.196 -1.575 -1.529 -2.571 -1.151 -1.449 2.109 1.540 -2.533 -2.774 0.432 -0.634 1.599 1.462 -1.413 -0.059 0.520 2.983 -0.037 1.119 -0.671
[64 numbers left out]
W1 row 69: input (1,3), channel 4
-0.199 0.777 -0.881 -0.140 0.926 2.123 1.572 -4.301 3.057 -1.608 -1.535 1.709 -1.663 -2.232 -1.541 -0.870 -0.942 1.343 1.435 -2.380 -2.523 0.529 0.393 1.060 1.504 -0.448 -0.004 0.846 2.985 -0.032 2.406 -1.309
[1,312 numbers left out]
W1 row 111: input (2,2), channel 1
-0.608 -0.837 -0.047 0.061 0.895 3.408 2.416 -6.356 4.570 -1.163 -2.449 -0.158 -1.942 -2.824 -2.155 -2.167 0.476 2.681 2.727 -2.677 -2.302 0.776 -0.869 2.319 0.206 -2.397 0.017 -0.307 4.816 0.094 2.613 -2.045
[1,440 numbers left out]
W1 row 157: input (3,1), channel 2
-4.811 0.022 -0.837 0.227 0.108 0.305 -0.020 -0.026 0.115 -0.007 0.266 0.025 -0.874 0.113 0.300 -0.302 -0.179 0.352 -0.131 -0.120 -3.236 -0.094 -0.252 -0.638 -0.531 0.049 0.209 -0.180 -0.233 0.040 -0.389 0.056
[7,904 numbers left out]
W1 row 405: input (8,1), channel 0
-0.289 -0.347 0.481 -0.345 -0.731 -1.841 0.943 -0.124 -0.042 -1.019 -0.789 -0.311 0.434 -3.293 -1.391 4.066 -0.271 1.079 -0.552 -1.610 -1.255 -0.161 0.913 2.480 -1.104 -0.072 0.125 0.462 -0.551 0.114 0.776 -4.013
[3,008 numbers left out]
b1 (layer 1 biases), shape 32:
[32 numbers left out]
W2 (layer 2 weights), shape 32 x 32: one line per row
[1,024 numbers left out]
b2 (layer 2 biases), shape 32:
[32 numbers left out]
W3 (output layer weights), shape 32 x 5: one line per row
[55 numbers left out]
W3 row 11
-0.177 0.185 0.450 4.057 -0.414
[50 numbers left out]
W3 row 22
0.376 0.387 0.537 -5.336 -1.203
[5 numbers left out]
W3 row 24
0.367 0.561 0.305 0.911 3.031
[35 numbers left out]
b3 (output layer biases), shape 5:
b3
1.221 1.008 1.024 0.897 1.298
Two things stand out. First, it's a toy model. Interpretability is easy on toy models. The same techniques generally don't work with complex models. The settled research and strategies on interpretability for small models are almost definitely in the training dataset for large models like Astra and Fable.
But more importantly, the prompt didn't just include the weights, it gave the agent a map. With the map, most humans that were unintimidated by the math could work it out. It would just take a lot longer. To be safe, I stripped out the map and prompted the models with just the weights. Without the map, neither model could say what the weights represented. With a follow up prompt of "If I told you this represented a game, which game might it be?", Astra said:
That's comparable to an educated guess a human might make.
The same general idea applies to all of the game examples and all of the shape diffusion examples. Similar for the python example.
So it seems to be more about prompt context than about a "native ability to read neural network weights". So the "read" half of the premise doesn't appear to be supported. I mean technically it is, but not in the way I'm imagining they intend for it to be read. The technical version isn't that interesting given all of the prior research on tiny models.
It's pretty cool that we have LLMs smart enough to figure things like this out but I feel like that was already well established. They're significantly "smarter" than this.
Moving on the the "write" section... It's more of the same at first, small models with a pretty detailed map. It's still cool, but the "natively write" part of the premise isn't getting paid off either.
At this point I'm starting to feel like this exploration isn't a good use of time :)
The "Make Qwen3.5-0.8B better at mathematics by editing its weights directly" seemed more interesting though: 50 forward passes to fix a math issue in Qwen by only editing weights.
The full prompt is extensive, again lots of context that does much of the work, and it's about a documented problem in that version of Qwen. The model essentially started with the diagnosis and then had 50 chances to trial and error to the fix.
The technique used is essentially directional ablation. It's well established and not particularly new, definitely in training. I'm still not seeing anything to support the "natively write" part of the premise.
It would be exciting if I was missing something, consider this an open invitation to tell what I've got wrong. Otherwise it feels like hype. But to be fair to the authors, they don't make any direct claims that aren't true and they're not claiming to be doing science.
Oh and also this was in the repo's LLMy readme:
Given the greater than zero on many version numbers, they probably started off trying something closer to one shotting and ended up having to add a lot more context to the prompt to get it to work.
From the article:
[...]
[...]
I'm going to need new vocabulary to describe this feeling, a combination of excited and terrified. Excerifed? Terricited?
It's incredibly cool that AI can do this, and this also seems like how we get Skynet (eventually).
"Awe", potentially? But the dictionary definition, not the colloquial one.
I assume we used to feel it more when exposed to forces beyond our understanding and control, hence the dilution of the term as it sought out a new definition that could apply to everyday life again.
That's a pretty apt definition actually
It turns out being able to think about 1,000,000 things simultaneously, compared to a human that can handle ~6, is a big upgrade.
It would be awesome for hosted LLMs to be provided access to read their own weights and activations so they can truly, for the first time, explain why they did something.
Edit: One thing I do right now is provide agents I set up with access to their own source code. I created an agent that's a glorified search engine for an organization. You chat with it through Discord and it can search Discord, our internal wiki, the broader web, and can read the source code to any of the organization's code repositories. We have a lot of technical people so it gets a lot of question like "@Bot what agent framework are you using?" etc. (for the record, no framework - the
agent.tsfile is only a few hundred lines and wouldn't benefit from anything so fancy).I haven't dug into the paper much, but from several examples, it looks like it effectively reverse engineered what the weights were representing for a variety of specific examples? Which makes sense that it should be able to do that ...
W.r.t. the Qwen math result, that's interesting! It did have 50 tries, though, and it sounds like it fixed one issue that gradient descent couldn't bump the model out of a local optima for. Might be interesting to see if this sort of approach comes into play for training, if it's relatively cheap?
Overall: I didn't have a guess on whether or not this was feasible, just a general sense that we don't have a firm grasp on how most things an LLM does are performed. Notably, single bits of behaviour (e.g. refusals) were known to be removable relatively easily via abliteration, so correcting individual behaviours doesn't seem novel at least. I guess this is a cool result, but it's hard to tell what it means (or where it's headed) without more papers or a better grounding in the field.
Maybe some experts are around who can comment?
Many people are skeptical this will lead to Recursive Self Improvement. This is a very large model fixing a very small model to do a simple task, not a very large model improving itself. Plus, the main issues with models today all involve tacit knowledge, and verbally asking the LLM to improve that seems unlikely to work, like asking Claude to "be more creative".
But here is one interesting idea I predict: let's say you want to make an app or video game. Today, you may prompt the LLM and it writes code, then iterate prompting for changes, but each change takes a while and isn't quite what you want, and the code may become spaghetti. Or write code by hand for more detail, but it's slower, and it still becomes convoluted and harder to edit over time (depending on your coding skill).
What if the LLM creates a small NN that simulates your app or game? It would probably be slower to create, but I suspect actually faster to edit, especially since you can ask the LLM to create UI controls for abstract features, and more resistant to degradation.
Sounds interesting, but very inefficient to run and very problematic for any deterministic game/app when it keeps producing inherently stochastic output!
So could this mean that, as well as using frontier models to distill to cheaper models, Chinese and European players can also use the frontier models or their best models to improve them even more? That sounds great: it would mean the gap between open and sovereign models will close even more.
The models they're targeting are tiny. I doubt this holds up as the model complexity increases.