Iroh is announcing Mesh LLM, a distributed LLM serving infrastructure. Inside it runs llama.cpp and offers OpenAI-compatible API for the user. The llama.cpp part seems to be optional - you can...
Iroh is announcing Mesh LLM, a distributed LLM serving infrastructure.
Inside it runs llama.cpp and offers OpenAI-compatible API for the user. The llama.cpp part seems to be optional - you can just run the API
The novel part is that you can distribute the model layers across computers ("Skippy"). Essentially, you can run models that are larger than your VRAM by distributing it over several computers
Previous approach from vLLM and llama.cpp-rpc do exists, but I believe the way Skippy split the model is different than the others, which are intended for very high speed local links.
Iroh is a P2P networking library. Think Steam networking, or Tailscale but a library you embed into your application. The distribution system use Iroh internally, which means that the computers running the model can be anywhere from local to over internet.
Splitting the model this way generally result in poor performance. On Hacker News the author said that running GLM5.2 Q2 MTP using two Mac Studio on 1Gbps local ethernet results in 10 tok/s.
They're running a public mesh which is the default in the software. Users can also freely run a private mesh without setting up a coordinator server.
Personally I'm interested in democratizing LLM, so stuff like this is quite exciting. (I might advertise my own effort soon if it get merged!) However, this mesh seems too naive to work in a real world:
People have different opinion of the model to use, including the quants and finetunes. Without coordination if say you need 10 people to be able to host Kimi, the first person would have to wait for days until all 10 people show up (and never left during that).
There is no system to avoid malicious host in the public pool. Malicious hosts can read inputs and poison outputs.
There is no reward for running the model. Not even guaranteed speed or even faster than non-hosters.
Sounds like it's intended to be used by a single operator with a pool of machines they already control, probably for the reasons you mentioned. I think ultimately this kind of distributed...
Sounds like it's intended to be used by a single operator with a pool of machines they already control, probably for the reasons you mentioned.
I think ultimately this kind of distributed architecture is going to be vital as models get larger and even the beefiest GPUs struggle with workloads. So any progress in that direction is good. But I agree that it's not very practical in this form. Without having tried it, I'm extremely skeptical about performance. Seems like the latency alone would be a deal-breaker. It's hard enough splitting a workload between multiple GPUs in the same system, let alone wrangling separate boxes over a network.
I’d be interested to see concrete numbers on performance vs link speed on this, figure out how much of a bottleneck it really is if running on a local network. I wouldn’t want to run it with...
I’d be interested to see concrete numbers on performance vs link speed on this, figure out how much of a bottleneck it really is if running on a local network. I wouldn’t want to run it with internet-level latency, but for inference the data flow between layers is predictable enough that a couple of ms at two or three sensible breakpoints might not be a killer. Maybe.
Seems like a relatively straightforward experiment to run for someone with access to 2-3 DGX Sparks or similar, at least: just write a script to loop the same test and record the tok/s each time with the network cards in every driver supported mode from baseline 1GbE to full 200GbE with RDMA, PFC, and chocolate sprinkles.
So, this would be excellent for batch use cases that don't need realtime response. I have a giant backlog of stuff that falls under 'Launch a job and come back to it 2 weeks later' which is low...
So, this would be excellent for batch use cases that don't need realtime response. I have a giant backlog of stuff that falls under 'Launch a job and come back to it 2 weeks later' which is low priority enough to not spend actual money on.
If something like this would let me slowly churn through that backlog while helping others do the same (and perhaps improving the tech as it goes), I'll check it out.
Got about 2.5 GB of vendor source code distributed via zip file (no source control). I want summarized and embedded using various models. They release monthly, and once I have a baseline going,...
Got about 2.5 GB of vendor source code distributed via zip file (no source control). I want summarized and embedded using various models.
They release monthly, and once I have a baseline going, the diffs should be more manageable.
Especially since then we build a customization layer on top, and their impact analysis tools leave much to be desired.
I've got a nice little MCP brewing which allows far better diff. I'll probably be able to swing some budget to speed up the process because it's proving useful, but still.
I've been doing my PoC on a tiny little workstation doing CPU embeddings at about 200 token/s.
The public mesh might be ok as a demo, but it sounds like it’s more likely to be used by an organization that has a bunch of computers in a computer lab that they want to use. If you’re concerned...
The public mesh might be ok as a demo, but it sounds like it’s more likely to be used by an organization that has a bunch of computers in a computer lab that they want to use.
If you’re concerned about the resource consumption of LLM’s, it seems like this is going to be worse than anything they’re running in a data center?
As one of its past winners a decade or two ago, I've been invited to judge for a national secondary school-level software competition in Thailand. The past two years I've seen that generative AI...
As one of its past winners a decade or two ago, I've been invited to judge for a national secondary school-level software competition in Thailand. The past two years I've seen that generative AI is widening the gap between top schools and the others a lot.
Back in my day, the field is kinda equalized with software piracy. Everyone use Photoshop regardless of whether you can afford it. The gap is in hardware, but Arduino seems cheap enough that it is not that bad and the top schools who can afford Microsoft Kinect are usually using it for the games category not software.
Fast forward to around 2023 AI become a big thing and people are making it more accessible, especially Google ML Kit/MediaPipe. What I've seen in university level projects, like face detection and pseudo-Kinect has now become smartphone apps developed by middle school kids and you no longer need specialized hardware or high quality webcam.
Now deep learning and large language models dominate the field. I've seen poorer schools use free Colab-style hosting and CPU inference trying to brute force their way to train custom models based on Llama. Last year's competition starts with Gemini 2, which wasn't any good, and ended with Gemini 2.5 Flash. I believe Gemini 2 was free and most teams that needed LLM used that while 1-2 teams used Gemini 2.5 Flash, paid out of their teacher's pocket, and nobody wanted to touch Gemini 2.5 Pro. GPT is out of the question, of course, because there was never a free offering. I would not be surprised if the rich schools (that bought gaming rigs for their game projects in the past) would buy DGX Spark or Mac Studio to run their local LLM this year, while most school does not even own a single dedicated desktop GPU.
So, I believe that building the next generation of engineers this democratized access to LLM is necessary. And it's not just giving them free ChatGPT or Gemini on the web - they needed full API access for their projects and AI coding (I hope they will be taught to use it for augmentation of their skills, not to replace it). Sure, DeepSeek or Mimo is very cheap but it's hard for their mindset (the student, and the poor teachers) to setup credit card and be at risk for cost overruns.
As for environmental impact, personally I don't care for that but as with EV I suppose if I do care I can install solar rooftop in my home. The grid here runs on like 60% natural gas. I don't know whether if you use the AWS/GCP local region are you even using green energy or just paying the premium for carbon offset, but they don't have GPU in their DC at this time of writing anyway.
I don't know if there are other use cases that free LLM will have significant impact on other groups of people. Right now my GPU are freely giving out tokens to roleplayers. Please share your stories if you do.
Ive got a few old computers sitting around not doing anything. (One runs my tv but thats it) I might play around with trying to get this up and running. Though I'm not sure what the advantage is...
Ive got a few old computers sitting around not doing anything. (One runs my tv but thats it) I might play around with trying to get this up and running. Though I'm not sure what the advantage is for an individual user thats just occasionally using the free version of chatgpt anyway.
Iroh is announcing Mesh LLM, a distributed LLM serving infrastructure.
Personally I'm interested in democratizing LLM, so stuff like this is quite exciting. (I might advertise my own effort soon if it get merged!) However, this mesh seems too naive to work in a real world:
Sounds like it's intended to be used by a single operator with a pool of machines they already control, probably for the reasons you mentioned.
I think ultimately this kind of distributed architecture is going to be vital as models get larger and even the beefiest GPUs struggle with workloads. So any progress in that direction is good. But I agree that it's not very practical in this form. Without having tried it, I'm extremely skeptical about performance. Seems like the latency alone would be a deal-breaker. It's hard enough splitting a workload between multiple GPUs in the same system, let alone wrangling separate boxes over a network.
I’d be interested to see concrete numbers on performance vs link speed on this, figure out how much of a bottleneck it really is if running on a local network. I wouldn’t want to run it with internet-level latency, but for inference the data flow between layers is predictable enough that a couple of ms at two or three sensible breakpoints might not be a killer. Maybe.
Seems like a relatively straightforward experiment to run for someone with access to 2-3 DGX Sparks or similar, at least: just write a script to loop the same test and record the tok/s each time with the network cards in every driver supported mode from baseline 1GbE to full 200GbE with RDMA, PFC, and chocolate sprinkles.
So, this would be excellent for batch use cases that don't need realtime response. I have a giant backlog of stuff that falls under 'Launch a job and come back to it 2 weeks later' which is low priority enough to not spend actual money on.
If something like this would let me slowly churn through that backlog while helping others do the same (and perhaps improving the tech as it goes), I'll check it out.
Just curious, what sort of jobs do you have in mind that would run for two weeks unattended?
Got about 2.5 GB of vendor source code distributed via zip file (no source control). I want summarized and embedded using various models.
They release monthly, and once I have a baseline going, the diffs should be more manageable.
Especially since then we build a customization layer on top, and their impact analysis tools leave much to be desired.
I've got a nice little MCP brewing which allows far better diff. I'll probably be able to swing some budget to speed up the process because it's proving useful, but still.
I've been doing my PoC on a tiny little workstation doing CPU embeddings at about 200 token/s.
The public mesh might be ok as a demo, but it sounds like it’s more likely to be used by an organization that has a bunch of computers in a computer lab that they want to use.
If you’re concerned about the resource consumption of LLM’s, it seems like this is going to be worse than anything they’re running in a data center?
As one of its past winners a decade or two ago, I've been invited to judge for a national secondary school-level software competition in Thailand. The past two years I've seen that generative AI is widening the gap between top schools and the others a lot.
Back in my day, the field is kinda equalized with software piracy. Everyone use Photoshop regardless of whether you can afford it. The gap is in hardware, but Arduino seems cheap enough that it is not that bad and the top schools who can afford Microsoft Kinect are usually using it for the games category not software.
Fast forward to around 2023 AI become a big thing and people are making it more accessible, especially Google ML Kit/MediaPipe. What I've seen in university level projects, like face detection and pseudo-Kinect has now become smartphone apps developed by middle school kids and you no longer need specialized hardware or high quality webcam.
Now deep learning and large language models dominate the field. I've seen poorer schools use free Colab-style hosting and CPU inference trying to brute force their way to train custom models based on Llama. Last year's competition starts with Gemini 2, which wasn't any good, and ended with Gemini 2.5 Flash. I believe Gemini 2 was free and most teams that needed LLM used that while 1-2 teams used Gemini 2.5 Flash, paid out of their teacher's pocket, and nobody wanted to touch Gemini 2.5 Pro. GPT is out of the question, of course, because there was never a free offering. I would not be surprised if the rich schools (that bought gaming rigs for their game projects in the past) would buy DGX Spark or Mac Studio to run their local LLM this year, while most school does not even own a single dedicated desktop GPU.
So, I believe that building the next generation of engineers this democratized access to LLM is necessary. And it's not just giving them free ChatGPT or Gemini on the web - they needed full API access for their projects and AI coding (I hope they will be taught to use it for augmentation of their skills, not to replace it). Sure, DeepSeek or Mimo is very cheap but it's hard for their mindset (the student, and the poor teachers) to setup credit card and be at risk for cost overruns.
As for environmental impact, personally I don't care for that but as with EV I suppose if I do care I can install solar rooftop in my home. The grid here runs on like 60% natural gas. I don't know whether if you use the AWS/GCP local region are you even using green energy or just paying the premium for carbon offset, but they don't have GPU in their DC at this time of writing anyway.
I don't know if there are other use cases that free LLM will have significant impact on other groups of people. Right now my GPU are freely giving out tokens to roleplayers. Please share your stories if you do.
Ive got a few old computers sitting around not doing anything. (One runs my tv but thats it) I might play around with trying to get this up and running. Though I'm not sure what the advantage is for an individual user thats just occasionally using the free version of chatgpt anyway.