TokenSpeed Engine Challenges NVIDIA: Almost Half the Latency for AI Inference

The New Engine

LightSeek Foundation TokenSpeed engine is taking on NVIDIA TensorRT-LLM head-on, claiming almost half the latency for everyday decode jobs. The engine is available now under an MIT license, enabling developers to test and adopt it freely.

Performance Numbers

On NVIDIA B200 hardware, TokenSpeed edged out TensorRT-LLM by 9 percent in the lowest latency scenarios and 11 percent in throughput at 100 transactions per second per user. For teams building agentic workflows like Claude Code or GitHub Copilot, these improvements could add up fast when handling millions of requests daily.

Technical Innovation

The clever approach uses a C++ finite-state machine that locks down KV cache issues before runtime, while keeping the coding interface in Python for developer accessibility. This hybrid approach enables high performance without sacrificing ease of use.

Complementary Advances

Alongside TokenSpeed, NVIDIA ModelOpt has demonstrated that CLIP models can drop to FP8 precision while maintaining FP16 levels, as long as patch embedding layers are preserved. Together, these advances pave the way for running bigger models on regular hardware without the usual trade-offs.

Market Impact

As AI inference shifts from a nice-to-have to everyday infrastructure, reducing latency and costs becomes critical. Engines like TokenSpeed and quantization techniques like NVIDIA ModelOpt are making AI more accessible and affordable for a broader range of applications.