Iroh is announcing Mesh LLM, a distributed LLM serving infrastructure. Inside it runs llama.cpp and offers OpenAI-compatible API for the user. The llama.cpp part seems to be optional - you can...
Iroh is announcing Mesh LLM, a distributed LLM serving infrastructure.
Inside it runs llama.cpp and offers OpenAI-compatible API for the user. The llama.cpp part seems to be optional - you can just run the API
The novel part is that you can distribute the model layers across computers ("Skippy"). Essentially, you can run models that are larger than your VRAM by distributing it over several computers
Previous approach from vLLM and llama.cpp-rpc do exists, but I believe the way Skippy split the model is different than the others, which are intended for very high speed local links.
Iroh is a P2P networking library. Think Steam networking, or Tailscale but a library you embed into your application. The distribution system use Iroh internally, which means that the computers running the model can be anywhere from local to over internet.
Splitting the model this way generally result in poor performance. On Hacker News the author said that running GLM5.2 Q2 MTP using two Mac Studio on 1Gbps local ethernet results in 10 tok/s.
They're running a public mesh which is the default in the software. Users can also freely run a private mesh without setting up a coordinator server.
Personally I'm interested in democratizing LLM, so stuff like this is quite exciting. (I might advertise my own effort soon if it get merged!) However, this mesh seems too naive to work in a real world:
People have different opinion of the model to use, including the quants and finetunes. Without coordination if say you need 10 people to be able to host Kimi, the first person would have to wait for days until all 10 people show up (and never left during that).
There is no system to avoid malicious host in the public pool. Malicious hosts can read inputs and poison outputs.
There is no reward for running the model. Not even guaranteed speed or even faster than non-hosters.
Sounds like it's intended to be used by a single operator with a pool of machines they already control, probably for the reasons you mentioned. I think ultimately this kind of distributed...
Sounds like it's intended to be used by a single operator with a pool of machines they already control, probably for the reasons you mentioned.
I think ultimately this kind of distributed architecture is going to be vital as models get larger and even the beefiest GPUs struggle with workloads. So any progress in that direction is good. But I agree that it's not very practical in this form. Without having tried it, I'm extremely skeptical about performance. Seems like the latency alone would be a deal-breaker. It's hard enough splitting a workload between multiple GPUs in the same system, let alone wrangling separate boxes over a network.
The public mesh might be ok as a demo, but it sounds like it’s more likely to be used by an organization that has a bunch of computers in a computer lab that they want to use. If you’re concerned...
The public mesh might be ok as a demo, but it sounds like it’s more likely to be used by an organization that has a bunch of computers in a computer lab that they want to use.
If you’re concerned about the resource consumption of LLM’s, it seems like this is going to be worse than anything they’re running in a data center?
Iroh is announcing Mesh LLM, a distributed LLM serving infrastructure.
Personally I'm interested in democratizing LLM, so stuff like this is quite exciting. (I might advertise my own effort soon if it get merged!) However, this mesh seems too naive to work in a real world:
Sounds like it's intended to be used by a single operator with a pool of machines they already control, probably for the reasons you mentioned.
I think ultimately this kind of distributed architecture is going to be vital as models get larger and even the beefiest GPUs struggle with workloads. So any progress in that direction is good. But I agree that it's not very practical in this form. Without having tried it, I'm extremely skeptical about performance. Seems like the latency alone would be a deal-breaker. It's hard enough splitting a workload between multiple GPUs in the same system, let alone wrangling separate boxes over a network.
The public mesh might be ok as a demo, but it sounds like it’s more likely to be used by an organization that has a bunch of computers in a computer lab that they want to use.
If you’re concerned about the resource consumption of LLM’s, it seems like this is going to be worse than anything they’re running in a data center?