Between this from Google (it’s a very significant optimisation to KV caching, for anyone following along), and DeepSeek’s “Engram” conditional static memory lookups, it’s looking like there’s...
Between this from Google (it’s a very significant optimisation to KV caching, for anyone following along), and DeepSeek’s “Engram” conditional static memory lookups, it’s looking like there’s another decent step up in model performance coming. I’m actually a little surprised to be saying that, I was starting to think we were plateauing a bit and potentially resorting to slightly more brute force approaches to making improvements, so it’ll be interesting to see how much more steam there is in this broad class of LLM architectures!
[Edit] DeepMind’s recent paper about doing actual, publishable mathematical research with an LLM is very significant too IMO. This isn’t some beam search brute force approach to constructing a technically novel but ultimately unreadable thousand page proof, it seems to be a research team genuinely creating new knowledge with the aid of an LLM, which strikes me as an important tipping point in capability.
While I only follow along as a layman, I've seen some interesting discussion in the mathematics community about LLMs helping to find lemmas. Donald Knuth also recently wrote about Claude helping...
While I only follow along as a layman, I've seen some interesting discussion in the mathematics community about LLMs helping to find lemmas. Donald Knuth also recently wrote about Claude helping him solve a problem.
It does seem like there is some genuine value in frontier mathematics.
Anthropic is claiming a step change in an upcoming, larger than Opus, model release. But they'll need both memory and inference optimizations, they're already pushing the limits of their available...
Anthropic is claiming a step change in an upcoming, larger than Opus, model release. But they'll need both memory and inference optimizations, they're already pushing the limits of their available compute due to skyrocketing demand. Maybe they've already integrated some version of similar optimizations. Google's is the latest headline but a lot of groups have been working on it.
Perhaps Google deployed TurboQuant already? They were pretty early with supporting long-context conversations. The Engram paper is pretty interesting too.
Perhaps Google deployed TurboQuant already? They were pretty early with supporting long-context conversations.
Yeah, that absolutely seems plausible - will be interesting to see how (and if) models from the big players change as this newly published stuff cross-pollinates between them and mixes with...
Yeah, that absolutely seems plausible - will be interesting to see how (and if) models from the big players change as this newly published stuff cross-pollinates between them and mixes with whatever tricks and techniques they each haven’t published!
Between this *Google research now and the recent tech of fitting like 400B parameter models on an iPhone, it’s looking to be quite an interesting time for running local models! (I’m not finding it...
Between this *Google research now and the recent tech of fitting like 400B parameter models on an iPhone, it’s looking to be quite an interesting time for running local models!
(I’m not finding it right now, but I saw someone somewhere improving upon this repo, which was about running big models on laptop hardware, and that was already impressive enough in itself if you ask me: https://github.com/danveloper/flash-moe)
In which case used server farm RAM prices really go down, it’s a win-win! Win-win not guaranteed, management takes no responsibility for job losses, riots, rising fascism, economic collapse,...
In which case used server farm RAM prices really go down, it’s a win-win!
Win-win not guaranteed, management takes no responsibility for job losses, riots, rising fascism, economic collapse, plagues of locusts, or indigestion.
I am slightly optimistic for a long term recovery once the surviving companies with actual products realize they actually need proper developers, testers, etc to untangle the LLM induced mess....
for job losses, riots, rising fascism, economic collapse, plagues of locusts, or indigestion.
I am slightly optimistic for a long term recovery once the surviving companies with actual products realize they actually need proper developers, testers, etc to untangle the LLM induced mess. </wishful-thinking>
Unlikely, if you can reduce RAM usage you can now cram more model into the same RAM. It's basically induced demand. More bandwidth available is more bandwidth used.
Unlikely, if you can reduce RAM usage you can now cram more model into the same RAM. It's basically induced demand. More bandwidth available is more bandwidth used.
Yes, exactly that. There's a proposal in my municipality to force trucks onto the bus lane instead of the road which should, according to them, alleviate pressure on traffic. At the same time...
Yes, exactly that.
There's a proposal in my municipality to force trucks onto the bus lane instead of the road which should, according to them, alleviate pressure on traffic.
At the same time they're ditching the idea to add another bus route to a nearby city.
I don't have to tell you what will likely happen if this plan goes through.
I commiserate friend. The only feasible upgrade route for my own PC is to get an AM5 processor and therefore DDR5 RAM. It's prohibitively expensive to the point I'm not sure when it will be...
I commiserate friend. The only feasible upgrade route for my own PC is to get an AM5 processor and therefore DDR5 RAM. It's prohibitively expensive to the point I'm not sure when it will be possible.
I’ve taken to leaving my ancient rig turned off until I really have the urge to play something, then update everything for an hour, forget what I wanted to play and shut down the computer again...
I’ve taken to leaving my ancient rig turned off until I really have the urge to play something, then update everything for an hour, forget what I wanted to play and shut down the computer again for several weeks or months.
The AI labs do provide cheaper models, so this depends on customer behavior. Are they going to keep switching to the best model available or will they decide at some point to save money?...
The AI labs do provide cheaper models, so this depends on customer behavior. Are they going to keep switching to the best model available or will they decide at some point to save money?
Anecdotally, I use Sonnet rather than Opus for writing code most of the time to cut costs, because it seems good enough.
I skimmed through this quickly but I assume this algorithm would be restricted to just Google or corporations? Can I benefit from this if I want to homelab? I’ve got a 1060 and while running...
I skimmed through this quickly but I assume this algorithm would be restricted to just Google or corporations?
Can I benefit from this if I want to homelab?
I’ve got a 1060 and while running ollama on it is kind of impressive the first few times, it is a 6GB VRAM card after all.
Applies a slight penalty to repeated tokens to hopefully discourage idiot doom spirals. Applies a slight penalty to repeated tokens to hopefully disc^C
What's your experience with the performance diff on Ollama vs. llama.cpp? I like that it's easy to use and was assuming it didn't have much penalty, but if I'm leaving a lot on the table I'd be...
What's your experience with the performance diff on Ollama vs. llama.cpp? I like that it's easy to use and was assuming it didn't have much penalty, but if I'm leaving a lot on the table I'd be curious to know.
Wow, thanks for testing it! That seems significant enough that I'll look into setting it up instead. I picked up ollama 2 years ago when it was pretty new, and was the easiest way (especially for...
Wow, thanks for testing it! That seems significant enough that I'll look into setting it up instead. I picked up ollama 2 years ago when it was pretty new, and was the easiest way (especially for AMD cards, at least that I found) and never really considered anything else.
It’s a published paper, that means others can and will reimplement it - and likely pretty quickly, given the speed of OSS development in the field. You might be interested in the links that @tauon...
It’s a published paper, that means others can and will reimplement it - and likely pretty quickly, given the speed of OSS development in the field. You might be interested in the links that @tauon mentioned further up, too? Those are available here and now, and the results look seriously impressive!
I think a good proxy for whether it’s restricted to their own offerings or not is public knowledge: If they wanted Gemini to be the only model/model family that has the tech available, we wouldn’t...
I think a good proxy for whether it’s restricted to their own offerings or not is public knowledge: If they wanted Gemini to be the only model/model family that has the tech available, we wouldn’t even know about this research (or it’d just get published very silently without an accompanying research.google blog post).
Between this from Google (it’s a very significant optimisation to KV caching, for anyone following along), and DeepSeek’s “Engram” conditional static memory lookups, it’s looking like there’s another decent step up in model performance coming. I’m actually a little surprised to be saying that, I was starting to think we were plateauing a bit and potentially resorting to slightly more brute force approaches to making improvements, so it’ll be interesting to see how much more steam there is in this broad class of LLM architectures!
[Edit] DeepMind’s recent paper about doing actual, publishable mathematical research with an LLM is very significant too IMO. This isn’t some beam search brute force approach to constructing a technically novel but ultimately unreadable thousand page proof, it seems to be a research team genuinely creating new knowledge with the aid of an LLM, which strikes me as an important tipping point in capability.
While I only follow along as a layman, I've seen some interesting discussion in the mathematics community about LLMs helping to find lemmas. Donald Knuth also recently wrote about Claude helping him solve a problem.
It does seem like there is some genuine value in frontier mathematics.
Anthropic is claiming a step change in an upcoming, larger than Opus, model release. But they'll need both memory and inference optimizations, they're already pushing the limits of their available compute due to skyrocketing demand. Maybe they've already integrated some version of similar optimizations. Google's is the latest headline but a lot of groups have been working on it.
Perhaps Google deployed TurboQuant already? They were pretty early with supporting long-context conversations.
The Engram paper is pretty interesting too.
Yeah, that absolutely seems plausible - will be interesting to see how (and if) models from the big players change as this newly published stuff cross-pollinates between them and mixes with whatever tricks and techniques they each haven’t published!
Between this *Google research now and the recent tech of fitting like 400B parameter models on an iPhone, it’s looking to be quite an interesting time for running local models!
(I’m not finding it right now, but I saw someone somewhere improving upon this repo, which was about running big models on laptop hardware, and that was already impressive enough in itself if you ask me: https://github.com/danveloper/flash-moe)
Edit – found it! Saw it in this tw*tter post that was shared somewhere: https://xcancel.com/anemll/status/2035901335984611412?s=20. They’ve also forked the above repo at the same name as their twitter handle, and with tweaks for a fork of a fork at Anemll/Flash-iOS.)
RAM prices might be going down? Maybe? I hope? Narry needs a new Linux gaming box...
Or maybe we just move 6 times faster towards the bubble pop.
In which case used server farm RAM prices really go down, it’s a win-win!
Win-win not guaranteed, management takes no responsibility for job losses, riots, rising fascism, economic collapse, plagues of locusts, or indigestion.
Well at least the sticks of RAM will keep us warm and fed.
I am slightly optimistic for a long term recovery once the surviving companies with actual products realize they actually need proper developers, testers, etc to untangle the LLM induced mess. </wishful-thinking>
Unlikely, if you can reduce RAM usage you can now cram more model into the same RAM. It's basically induced demand. More bandwidth available is more bandwidth used.
It’s just like the, ahem, beloved “only one more lane, bro” known from automotive highway planning!
Yes, exactly that.
There's a proposal in my municipality to force trucks onto the bus lane instead of the road which should, according to them, alleviate pressure on traffic.
At the same time they're ditching the idea to add another bus route to a nearby city.
I don't have to tell you what will likely happen if this plan goes through.
A man can dream… a man can dream.
I commiserate friend. The only feasible upgrade route for my own PC is to get an AM5 processor and therefore DDR5 RAM. It's prohibitively expensive to the point I'm not sure when it will be possible.
Here's to hoping none of my current parts break.
I’ve taken to leaving my ancient rig turned off until I really have the urge to play something, then update everything for an hour, forget what I wanted to play and shut down the computer again for several weeks or months.
The AI labs do provide cheaper models, so this depends on customer behavior. Are they going to keep switching to the best model available or will they decide at some point to save money?
Anecdotally, I use Sonnet rather than Opus for writing code most of the time to cut costs, because it seems good enough.
I skimmed through this quickly but I assume this algorithm would be restricted to just Google or corporations?
Can I benefit from this if I want to homelab?
I’ve got a 1060 and while running ollama on it is kind of impressive the first few times, it is a 6GB VRAM card after all.
Well done xD
What's your experience with the performance diff on Ollama vs. llama.cpp? I like that it's easy to use and was assuming it didn't have much penalty, but if I'm leaving a lot on the table I'd be curious to know.
Wow, thanks for testing it! That seems significant enough that I'll look into setting it up instead. I picked up ollama 2 years ago when it was pretty new, and was the easiest way (especially for AMD cards, at least that I found) and never really considered anything else.
It’s a published paper, that means others can and will reimplement it - and likely pretty quickly, given the speed of OSS development in the field. You might be interested in the links that @tauon mentioned further up, too? Those are available here and now, and the results look seriously impressive!
I think a good proxy for whether it’s restricted to their own offerings or not is public knowledge: If they wanted Gemini to be the only model/model family that has the tech available, we wouldn’t even know about this research (or it’d just get published very silently without an accompanying research.google blog post).