Kog AI Engine Delivers 3,000 GPT Tokens Per Second on Standard GPUs
TL;DR. Kog AI introduced a tech preview of its inference engine, achieving 3,000 output tokens per second per request on MI300X and H200 GPUs. - The engine optimizes the full software stack for high-speed single-request LLM decoding. - This performance enables faster AI agent interactions and reduces latency for users. - Support for large MoE models is planned, offering similar speed capabilities.
- Kog AI's Inference Engine (KIE) reaches 3,000 tokens/s per request on 8x AMD MI300X GPUs and 2,100 tokens/s on 8x NVIDIA H200 GPUs.
- The optimization focuses on single-request LLM decoding speed, crucial for AI agent performance and user experience.
- KIE addresses software bottlenecks that prevent standard datacenter GPUs from reaching their full decoding speed potential.
- The technology involves co-designing model architecture, runtime, and low-level GPU code for latency optimization.
- Future plans include supporting large third-party Mixture-of-Experts (MoE) models at similar speeds.
Sources
- Real-time LLM Inference on Standard GPUs: 3k tokens/s per request — blog.kog.ai
- github.com — github.com