Lumabri Enables Napster-like Peer-to-Peer LLM Distribution
TL;DR. Lumabri introduces a pure C engine allowing peer-to-peer distribution and inference of large Mixture-of-Experts models. - The system lets any machine serve or chat with a model, with inference data streamed on demand. - Local mirroring ensures subsequent inferences are served quickly, even if the original server goes offline. - The engine operates identically on CPUs and GPUs, widening access for model contribution and use.
- Lumabri's colibri engine facilitates peer-to-peer serving and chatting with large Mixture-of-Experts (MoE) models.
- Inference data streams from peers on first use and is locally mirrored for subsequent rapid access.
- The system is designed for CPU-first operation, making it accessible even without GPUs for model participation.
- Users can chat, donate disk space for model parts, or contribute compute by running expert nodes within the swarm.
Sources
- Show HN: Lumabri – What if LLMs worked like Napster? — github.com