LookAtMyQuant Compiler Achieves 12x Lower KL Divergence for Quantized AI Models
A new quantization compiler, LookAtMyQuant, significantly reduces the size of AI models like Gemma 4 E2B and SmolLM2-360M with 12x lower KL divergence.

By Clara Tan, Artificial Intelligence
3 min read
A new quantization compiler named LookAtMyQuant, developed by ShinMegamiBoson, promises significant advancements in shrinking AI models without compromising performance. The compiler has demonstrated its effectiveness by achieving a 12 times lower Kullback-Leibler (KL) divergence compared to the Q4_K_M quantization format for the Gemma 4 E2B model, all while maintaining the same file size.
Traditional quantization methods often apply a uniform bit width across an entire model, or use a few hand-written exceptions. LookAtMyQuant takes a more granular approach. It measures the impact of compressing each individual piece of a model, such as weight matrices, embedding rows grouped by token frequency, and key-value caches, on the model's overall performance. Based on these measurements, the compiler intelligently assigns specific quantization codes to each piece to minimize total damage while adhering to a target device's size or speed budget.
This innovative approach has shown tangible benefits. For instance, on the Gemma 4 E2B model, a compiled file using LookAtMyQuant not only achieved 12 times lower KL divergence but also improved the GSM8K problem-solving rate to 59%, compared to 50% for Q4_K_M and 61% for the original model. The gains are attributed to the compiler's ability to optimise components like Gemma's 2.35-billion-weight per-layer lookup table, storing it at 4.25 bits instead of 6.56 and reallocating the saved bytes to matrices.
Beyond Gemma 4 E2B, LookAtMyQuant has also shown promising results on other models, including Qwen3.5-4B and SmolLM2-360M, achieving 6 to 81 times lower KL divergence at equal cost when compared to 3-bit models that keep embedding or cache at 16 bits. This indicates a versatile and powerful tool for developers looking to deploy more efficient and performant AI models on various devices.
Sources
- 1LookAtMyQuant GitHubgithub.com
- 2LookAtMyQuant Project Pagelookatmyquant.pages.dev