I love how good local models are and how optimized the software is. I was gifted a couple of old workstation cards (Turing generation RTX 6000s) and put them to use hosting Qwen 3.8 27B. With...
I love how good local models are and how optimized the software is. I was gifted a couple of old workstation cards (Turing generation RTX 6000s) and put them to use hosting Qwen 3.8 27B. With llama.cpp I can split the weights across the cards so they run in parallel. Multiple requests can batch together. I get 60 tok/s for one request or 40 tok/s each for two concurrent requests. And the model’s way smarter than it should be given the size. In a few years new mid-range consumer hardware will actually be hosting models that are worth running.
Wow. Nice gift! As a happy contrast to the whole 'giant companies holding the economy hostage and growing a huge bubble' downside (to put it mildly) the increasing capability and availability of...
I was gifted a couple of old workstation cards (Turing generation RTX 6000s)
Wow. Nice gift!
As a happy contrast to the whole 'giant companies holding the economy hostage and growing a huge bubble' downside (to put it mildly) the increasing capability and availability of locally hosted AI on, admittedly expensive (but not unreachably fiscally inaccessible) hardware, is very nice and puts tools and capabilities in the hands of the individual instead of megacorps looking to go further down the 'you'll own nothing, pay a subscription for everything, and thank us for it' path.
I’ve been trying to think of a way to explain this to my family (who don’t really pay attention to AI stuff except that it’s on their work computers everywhere now) and I phrased it as “incredibly...
admittedly expensive (but not unreachably fiscally inaccessible) hardware
I’ve been trying to think of a way to explain this to my family (who don’t really pay attention to AI stuff except that it’s on their work computers everywhere now) and I phrased it as “incredibly expensive consumer-grade, or surprisingly cheap business-grade kinds of prices” — which I think feels about right?
If I look at a colour laser printer, for example, I can get decent quality consumer grade version for well under $1000, but as soon as I’m looking at a commercial product, that’s instantly looking at about an order of magnitude higher.
Spending $10k-20k on a home computer seems like incredibly dedicated hobbyist kinds of prices, but for a small business that’s ... okay it’s not easy, but certainly more achievable, especially if it’s a tool you need for your business.
And if trends continue, maybe we'll get PCs for $3-$5k that can host seriously capable AI models at home. Which starts entering into the realm of "regular" hobbyist pricing.
And if trends continue, maybe we'll get PCs for $3-$5k that can host seriously capable AI models at home. Which starts entering into the realm of "regular" hobbyist pricing.
I have a server with those RTX 6000s in it. Yesterday I vibe-coded an agent that can access an organization's mediawiki website as well as search the accompanying Discord server. It can respond in...
I have a server with those RTX 6000s in it. Yesterday I vibe-coded an agent that can access an organization's mediawiki website as well as search the accompanying Discord server. It can respond in Discord channels and DMs citing wiki pages and Discord messages. It can also review your memes since I added the vision part of the model. The hardware necessary to run it would be ~$2500. It's actually pretty useful because you can have it summarize tremendously long arguments between people instead of needing to dig through 10,000 words yourself.
Normally I'd find something like this kind of gross, but because it's 100% self-hosted and has no write access (outside of message replies), no memories or other persistent state, and operates within our k8s cluster where the wiki is it's something I'm happy to have around.
This absolutely sent me. Wild guess but this wouldn’t happen to be about the Eco game would it? I find that, more than any other game I’ve played, Eco players can create interpersonal drama on...
It's actually pretty useful because you can have it summarize tremendously long arguments between people
This absolutely sent me. Wild guess but this wouldn’t happen to be about the Eco game would it? I find that, more than any other game I’ve played, Eco players can create interpersonal drama on nearly unrivalled scales!
Pretty crazy what modern hardware can do. I’ve got a 7900XTX, I ran deepseek a few years ago when it just came out and I was pretty impressed, but at that point it was mostly just a toy. It seems...
Pretty crazy what modern hardware can do. I’ve got a 7900XTX, I ran deepseek a few years ago when it just came out and I was pretty impressed, but at that point it was mostly just a toy.
It seems now, home LLMs are starting to get seriously useful.
Presumably deepseek distilled into a smaller model? There's some interesting stuff happening. People are trying to get large models, like 1 trillion parameters, running within 64GiB of RAM on...
I ran deepseek a few years ago when it just came out
Presumably deepseek distilled into a smaller model?
There's some interesting stuff happening. People are trying to get large models, like 1 trillion parameters, running within 64GiB of RAM on Apple computers by streaming weights off of the very fast storage. Last I saw it ran at like 1 or 2 tokens per second but it's promising.
Edit: We might end up in a world where OpenAI/Anthropic/etc. are unnecessary for most LLM users. Between Chinese models and self-hosting it's not unrealistic.
I had it spilling over into RAM too (64GB DDR5). It was incredibly slow. I can’t recall exactly how many parameters it was. I think maybe 70B and heavily quantized. It was a fun gimmick at the...
I had it spilling over into RAM too (64GB DDR5). It was incredibly slow. I can’t recall exactly how many parameters it was. I think maybe 70B and heavily quantized. It was a fun gimmick at the time, but these newer models seem genuinely useful.
Yes! I think that was exactly it. I remember trying to see if I could run the full deepseek model, and I quickly discovered it was impossible on my hardware.
Yes! I think that was exactly it. I remember trying to see if I could run the full deepseek model, and I quickly discovered it was impossible on my hardware.
We can already build or buy old rackmount servers with a cluster of old Tesla V100 cards for around that price... ish. Craft Computing - Is the Nvidia Tesla V100 still good for AI? - Inspur DGX...
We can already build or buy old rackmount servers with a cluster of old Tesla V100 cards for around that price... ish.
I'm experimenting with tieing locally hosted models into my workflow more. I'm weary of working on anything privacy sensitive like client data in cloud hosted models. Really optimistic about what...
I'm experimenting with tieing locally hosted models into my workflow more. I'm weary of working on anything privacy sensitive like client data in cloud hosted models. Really optimistic about what I'm seeing the local models doing. Just feels a bit clunky still to get stuff running and findng a comfortable workflow.
For coding, currently I talk to Astra to make a plan and tell it to use a Luna subagent to implement each commit. Luna is extremely cheap and I don't worry that using Luna alone will ever exhaust...
For coding, currently I talk to Astra to make a plan and tell it to use a Luna subagent to implement each commit. Luna is extremely cheap and I don't worry that using Luna alone will ever exhaust my $20/month plan. Astra is too expensive for coding use unless you're using it professionally, but I think the planning goes better.
And I haven't even switched to the latest version of Luna yet, which will be half the price.1 It doesn't seem to be available via a ChatGPT subscription yet.
1 Yeah, it's an "introductory" price but I think that's OpenAI hedging their bets in case there's a capacity crunch. These price drops have a tendency to become permanent due to competition.
I've been using Deepseek Flash 4.1 served through fireworks.ai for report development in OpenChamber and its remarkable. Cost-wise, assembling a 45-page report has set me back $2.19. Granted, I'm...
I've been using Deepseek Flash 4.1 served through fireworks.ai for report development in OpenChamber and its remarkable. Cost-wise, assembling a 45-page report has set me back $2.19. Granted, I'm making heavy use of skills, tool calls, and retrieval augmented generation, but DS4.1 is really good, with Fireworks raising prices on October 1st to deal with "unprecedented demand". After I'm done, I might run it through Opus 5.5, as the word is that it writes well, but the total cost is will still be very low. I'm excited to see what lower cost open weights models are going to be capable of in the coming months.
I've been doing a lot of work where I have a GLM 5.3 sub-agent launched for every file in a repo (for small projects) and then hand over findings to Fable or Opus to prune.
I've been doing a lot of work where I have a GLM 5.3 sub-agent launched for every file in a repo (for small projects) and then hand over findings to Fable or Opus to prune.
AI has gotten cheaper more quickly than any other transformative technology in history. The cost of achieving a given level of AI performance has fallen about 47% per quarter since 2023, or 13× per year. That price drop is four times faster than DNA sequencing, six times faster than compute, 18 times faster than lithium batteries, and (in the century up to 1973) 54 times faster than electricity.
[...]
The next chart shows some examples. On January 31, 2025, OpenAI released a new iteration in its series of “reasoning” models, called o3. We estimate that for an average cost of 30 cents per question, it could achieve a 75% score on GPQA Diamond, a multiple-choice exam covering PhD-level physics, chemistry, and biology.1 Just under 18 months later, OpenAI released GPT-5.6 Luna. It scored just as well — for four hundredths of a penny per question ($0.0004). That is a 725-fold drop in the price of thought in under 18 months. It is like the sticker price on a new car falling from $50,000 to $69. No other general-purpose technology in history appears to have gotten so cheap so fast.
[...]
Our analysis comes with major caveats. AI companies may be expressly training their models for some benchmarks (“benchmaxxing”), so that improvement on the benchmarks outstrips improvement for real-world tasks. Even if they are not, doing well on a benchmark is not synonymous with useful work. Because we focus on the frontier — the absolute cheapest model capable of any given level of performance — we implicitly posit an AI user who relentlessly searches for the most cost-effective model for each task, when real users do not switch models so often, and therefore do not reap quite the same savings. Our data are incomplete and noisy: the timeframe is barely three years, and we do not include all combinations of AI model and benchmark. Prices drop differently for different models, benchmarks, time periods, and performance ranges, and there are many reasonable ways to average over this variegated experience. Overall, while we believe that our bottom-line numbers are reasonably representative of reality, they should not be read as exact.
Noise labels. That being said, I really wonder what the people who are labeling it as noise are thinking. We had a whole epidemic of people posting articles with no context on tildes, the result...
Noise labels.
That being said, I really wonder what the people who are labeling it as noise are thinking. We had a whole epidemic of people posting articles with no context on tildes, the result of which was that people are encouraged to post snippets of the article.
I love how good local models are and how optimized the software is. I was gifted a couple of old workstation cards (Turing generation RTX 6000s) and put them to use hosting Qwen 3.8 27B. With llama.cpp I can split the weights across the cards so they run in parallel. Multiple requests can batch together. I get 60 tok/s for one request or 40 tok/s each for two concurrent requests. And the model’s way smarter than it should be given the size. In a few years new mid-range consumer hardware will actually be hosting models that are worth running.
Wow. Nice gift!
As a happy contrast to the whole 'giant companies holding the economy hostage and growing a huge bubble' downside (to put it mildly) the increasing capability and availability of locally hosted AI on, admittedly expensive (but not unreachably fiscally inaccessible) hardware, is very nice and puts tools and capabilities in the hands of the individual instead of megacorps looking to go further down the 'you'll own nothing, pay a subscription for everything, and thank us for it' path.
I’ve been trying to think of a way to explain this to my family (who don’t really pay attention to AI stuff except that it’s on their work computers everywhere now) and I phrased it as “incredibly expensive consumer-grade, or surprisingly cheap business-grade kinds of prices” — which I think feels about right?
If I look at a colour laser printer, for example, I can get decent quality consumer grade version for well under $1000, but as soon as I’m looking at a commercial product, that’s instantly looking at about an order of magnitude higher.
Spending $10k-20k on a home computer seems like incredibly dedicated hobbyist kinds of prices, but for a small business that’s ... okay it’s not easy, but certainly more achievable, especially if it’s a tool you need for your business.
And if trends continue, maybe we'll get PCs for $3-$5k that can host seriously capable AI models at home. Which starts entering into the realm of "regular" hobbyist pricing.
I have a server with those RTX 6000s in it. Yesterday I vibe-coded an agent that can access an organization's mediawiki website as well as search the accompanying Discord server. It can respond in Discord channels and DMs citing wiki pages and Discord messages. It can also review your memes since I added the vision part of the model. The hardware necessary to run it would be ~$2500. It's actually pretty useful because you can have it summarize tremendously long arguments between people instead of needing to dig through 10,000 words yourself.
Normally I'd find something like this kind of gross, but because it's 100% self-hosted and has no write access (outside of message replies), no memories or other persistent state, and operates within our k8s cluster where the wiki is it's something I'm happy to have around.
This absolutely sent me. Wild guess but this wouldn’t happen to be about the Eco game would it? I find that, more than any other game I’ve played, Eco players can create interpersonal drama on nearly unrivalled scales!
No it's for a non profit organization.
Pretty crazy what modern hardware can do. I’ve got a 7900XTX, I ran deepseek a few years ago when it just came out and I was pretty impressed, but at that point it was mostly just a toy.
It seems now, home LLMs are starting to get seriously useful.
Presumably deepseek distilled into a smaller model?
There's some interesting stuff happening. People are trying to get large models, like 1 trillion parameters, running within 64GiB of RAM on Apple computers by streaming weights off of the very fast storage. Last I saw it ran at like 1 or 2 tokens per second but it's promising.
Edit: We might end up in a world where OpenAI/Anthropic/etc. are unnecessary for most LLM users. Between Chinese models and self-hosting it's not unrealistic.
I had it spilling over into RAM too (64GB DDR5). It was incredibly slow. I can’t recall exactly how many parameters it was. I think maybe 70B and heavily quantized. It was a fun gimmick at the time, but these newer models seem genuinely useful.
That was probably Llama 3.3 70B fine tuned on Deepseek. It wasn’t well labeled. True Deepseek at the time was 671B.
Yes! I think that was exactly it. I remember trying to see if I could run the full deepseek model, and I quickly discovered it was impossible on my hardware.
We can already build or buy old rackmount servers with a cluster of old Tesla V100 cards for around that price... ish.
Craft Computing -
Is the Nvidia Tesla V100 still good for AI? - Inspur DGX V100 vs RTX 5090
Seems this is enough to run some of the lighter footprint models.
Thanks for sharing this video! Really cool stuff.
I'm experimenting with tieing locally hosted models into my workflow more. I'm weary of working on anything privacy sensitive like client data in cloud hosted models. Really optimistic about what I'm seeing the local models doing. Just feels a bit clunky still to get stuff running and findng a comfortable workflow.
For coding, currently I talk to Astra to make a plan and tell it to use a Luna subagent to implement each commit. Luna is extremely cheap and I don't worry that using Luna alone will ever exhaust my $20/month plan. Astra is too expensive for coding use unless you're using it professionally, but I think the planning goes better.
And I haven't even switched to the latest version of Luna yet, which will be half the price.1 It doesn't seem to be available via a ChatGPT subscription yet.
1 Yeah, it's an "introductory" price but I think that's OpenAI hedging their bets in case there's a capacity crunch. These price drops have a tendency to become permanent due to competition.
I've been using Deepseek Flash 4.1 served through fireworks.ai for report development in OpenChamber and its remarkable. Cost-wise, assembling a 45-page report has set me back $2.19. Granted, I'm making heavy use of skills, tool calls, and retrieval augmented generation, but DS4.1 is really good, with Fireworks raising prices on October 1st to deal with "unprecedented demand". After I'm done, I might run it through Opus 5.5, as the word is that it writes well, but the total cost is will still be very low. I'm excited to see what lower cost open weights models are going to be capable of in the coming months.
I've been doing a lot of work where I have a GLM 5.3 sub-agent launched for every file in a repo (for small projects) and then hand over findings to Fable or Opus to prune.
From the article:
[...]
[...]
Why is this post hidden by default?
Noise labels.
That being said, I really wonder what the people who are labeling it as noise are thinking. We had a whole epidemic of people posting articles with no context on tildes, the result of which was that people are encouraged to post snippets of the article.
Who is considering that noise, now?
I find the snippets really useful. Laughton click on the comments that very reason before reading the article.
I don’t know what you mean? It doesn’t seem to be for me.
Probably got reported/labeled which gives it virtual negative karma or something like that.