Running LLMs on a 10-year-old Xeon without GPU
TL;DR. An enthusiast details methods to run 26B-parameter MTP Drafter LLMs efficiently on a decade-old Intel Xeon server lacking a GPU. - The project highlights memory bandwidth as the key bottleneck for LLM inference on vintage hardware. - Custom optimizations go beyond standard tools like `ollama` or `llama-cpp` to achieve functional speeds. - The approach demonstrates that significant computational power is not always required for inference if software is optimized.
- A 10-year-old Intel Xeon E5-2620 v4 server with 128 GB DDR3 RAM can run 26B-parameter LLMs without a GPU.
- Memory bandwidth, not raw processing power, is identified as the primary limiting factor for LLM inference on current and older hardware.
- The presented solution involves custom software optimizations, going beyond generic LLM execution frameworks.
- The project successfully demonstrates practical LLM inference on severely constrained, off-the-shelf hardware.