Hackaday

TurboQuant: Reducing LLM Memory Usage With Vector Quantization

Large language models (LLMs) aren’t actually giant computer brains. Instead, they are massive vector spaces in which the ...

Google's new TurboQuant algorithm speeds up AI memory 8x, cutting costs by 50% or more

Within 24 hours of the release, community members began porting the algorithm to popular local AI libraries like MLX for ...
GitHub

NaN assertion error during quantization

I encountered a runtime error related to NaNs during quantization and would like to ask whether this is a known issue.
Forbes

How Mixed-Precision Quantization Could Break AI’s Power Addiction

It turns out the rapid growth of AI has a massive downside: namely, spiraling power consumption, strained infrastructure and runaway environmental damage. It’s clear the status quo won’t cut it ...
Hackaday

Making The Smallest And Dumbest LLM With Extreme Quantization

The reason why large language models are called ‘large’ is not because of how smart they are, but as a factor of their sheer size in bytes. At billions of parameters at four bytes each, they pose a ...
IEEE

Rejection-Sampled Universal Quantization for Smaller Quantization Errors

Abstract: We construct a randomized vector quantizer which has a smaller maximum error compared to all known lattice quantizers with the same entropy for dimensions 5 ...
GitHub

GPTQ block quantization error implementation

First of all, thank you very much for sharing such great code! It has been incredibly helpful in my research on quantization using NVFP4. The reason I am reaching out ...