Cloudflare Boosts Kimi & GLM AI Model Performance with FP8 Quantization

Cloudflare's Workers AI platform now efficiently serves large Chinese frontier models like Moonshot's Kimi K-series and Z.ai's GLM by quantizing the KV cache from 16-bit (BF16) to 8-bit floating point (FP8), as detailed in a technical deep-dive published on August 3.

Optimizing AI Inference with FP8 Quantization

The core of Cloudflare's optimization lies in quantizing the Key-Value (KV) cache from 16-bit Brain Floating Point (BF16) to 8-bit Floating Point (FP8). This technical advancement is crucial for managing the substantial memory demands of large language models. By reducing the precision of the KV cache, Cloudflare effectively halves memory usage, which directly translates to a doubled context window capacity for the models.

For instance, when running the Kimi K2.6 model on H200 GPUs, FP8 quantization increased the maximum context window from approximately 686,000 tokens to an impressive 1.37 million tokens. This expanded context window is vital for applications requiring extensive memory, such as complex coding tasks or detailed document analysis, allowing the AI to process and retain more information within a single interaction.

Enhanced Throughput and Cost Efficiency

Beyond memory optimization, FP8 quantization also delivers substantial performance and cost benefits. Cloudflare's implementation enabled Kimi K2.6 to handle 64 concurrent requests, a significant increase from the 32 requests possible with BF16. This led to a throughput of 2,192 tokens per second, resulting in approximately 41% higher throughput and a 30% lower cost per token. These improvements are critical for making advanced AI models more accessible and economically viable for a broader range of applications and users.

Leveraging SGLang for Superior Inference Serving

Cloudflare utilizes SGLang as its primary inference serving framework, identifying it as the top-performing open-source option available. SGLang plays a pivotal role in orchestrating the efficient execution of these large models. Cloudflare's commitment to the open-source community is evident as they actively contribute patches and new features back to the SGLang project, fostering collaborative development and continuous improvement.

In addition to KV cache quantization, Cloudflare also employs other optimization techniques, including model weight compression and shared-cache integrity protections. These combined strategies ensure robust and efficient operation of their Workers AI platform, demonstrating that advanced Chinese frontier models can be served effectively alongside Western alternatives on a global scale.

Why This Matters for AI Development

The ability to efficiently serve increasingly large frontier models on constrained GPU memory is a significant infrastructure challenge in the AI landscape. Cloudflare's advancements with FP8 quantization and SGLang highlight a practical solution to this problem. This work not only makes powerful models like Kimi and GLM more accessible but also sets a precedent for how other organizations can optimize their AI deployments. It underscores the ongoing innovation required to scale AI technologies and reduce the operational costs associated with high-performance computing.

Conclusion

Cloudflare's technical deep-dive into KV cache quantization and its application to models like Kimi K2.6 marks a notable step forward in efficient AI model deployment. By achieving significant gains in context window capacity, throughput, and cost efficiency, Cloudflare is enabling broader access to advanced AI capabilities. This development reinforces the importance of continuous innovation in AI infrastructure to support the growing demands of frontier models and their diverse applications.

Sources

• GitHub - sinameraji/kimiflare: Moonshot kimi k3 terminal based coding agent & harness running on your own Cloudflare account. · GitHub
• feat: add reasoning_content, GLM, GPT-OSS, Nemotron, Kimi K2.5 models · cloudflare/langchain-cloudflare@f08d4e8 · GitHub
• "No need for cookie banners" might be incorrect · plausible/analytics · Discussion #1963 · GitHub
• frontend-case-studies/readme.md at master · andrew--r/frontend-case-studies · GitHub