Four Bits, Fewer Failure Modes

TL;DR for operators

Post-training quantization methods that look similar on normal 4-bit benchmarks can separate sharply when the numerical problem becomes difficult. In one Mistral-7B v0.1 stress test, QuIP-family variants abort with NaNs and GPTQ reaches WikiText-2 perplexity of 379.73 ± 15.71. Kashin-DCT+H reaches 8.92 ± 0.01, close to the FP16 reference of 8.63.

The paper’s contribution is not simply another quantizer. It restructures Kashin-based quantization so that weights are first represented by bounded factors, replaces an expensive dense random transform with a sign-randomized DCT, and makes the factor geometry predictable enough to initialize four quantization clusters analytically. The transform falls from $O(N^2)$ to $O(N\log N)$ per iteration, while multi-restart k-means disappears.

For teams choosing a PTQ pipeline, this adds another evaluation axis: failure resilience. Kashin-DCT is broadly competitive with established 4-bit methods on ordinary benchmarks and unusually stable in the paper’s targeted failure cases. That does not make it the fastest implementation today. On the reported H100 runs, its quantization wall-clock time remains higher than OPTQ and the restricted QuIP# baseline.

Nor is this a deployed 2-bit model. Each weight is represented by two 2-bit factor codes, for an effective body of about 4 bits per weight plus small metadata. The proposed serving advantage depends on native 2-bit matrix multiplication and a fused kernel that has not yet been implemented and benchmarked end to end.

Similar 4-bit scores can hide very different failure behavior

A deployment team usually has a straightforward objective: reduce model memory and inference cost without damaging model quality enough to erase the gain. That makes perplexity and downstream accuracy natural first filters when comparing post-training quantization methods.

The harder question appears when the conversion itself becomes numerically fragile.

On Mistral-7B v0.1, the stress test in Structured Transforms for Low-Overhead Quantization of Language Models¹ produces a striking separation. The QuIP-family configurations abort because of NaNs during LDL decomposition. GPTQ completes, but WikiText-2 perplexity rises from the FP16 model’s 8.63 to 379.73 ± 15.71. Kashin-DCT+K and Kashin-DCT+H remain at 8.95 ± 0.01 and 8.92 ± 0.01 respectively.

This is not the paper’s main benchmark. It is better read as a targeted robustness test: a difficult configuration exposing sensitivity to weight distributions and Hessian conditioning. It therefore does not establish that Kashin-DCT will dominate across arbitrary pathological models. It does establish that methods occupying roughly the same low-bit design space can differ materially in whether quantization degrades gracefully or collapses.

Pythia-6.9B shows the same distinction less cleanly but at larger scale. OPTQ remains strongest at 12.02 WikiText-2 perplexity against FP16 at 11.41. Kashin-DCT+H rises to 20.64 ± 1.63. Yet QuIP and QuIP-RG exceed 2,200, while the restricted QuIP# configuration reaches 324.93 ± 47.63.

The relevant result is therefore robustness, not universal accuracy leadership.

Bounded factors change where the quantizer absorbs difficulty

The structural idea begins before quantization itself.

Large or poorly distributed weight values are difficult to place on a small number of discrete levels. Kashin decomposition addresses that problem by rewriting a vector as the sum of two components whose maximum coordinate magnitudes are controlled. Instead of asking one low-bit codebook to accommodate the original weight distribution directly, the method quantizes two better-behaved factors.

The paper proves that its revised greedy procedure retains geometric residual decay while keeping the factors’ infinity norms bounded. In simplified form, after successive four-step blocks,

That bound matters because the robustness argument depends on structure created before Hessian-based error compensation operates. Adaptive per-column codebooks then quantize factors with constrained dynamic range rather than forcing a fixed grid to absorb unusually large values.

The paper’s stress experiments are consistent with that mechanism. They do not isolate bounded factorization as a causal treatment, but they show the complete pipeline remaining usable in configurations where several Hessian-driven alternatives become highly unstable.

The expensive random rotation becomes a structured transform

Earlier Kashin-based LLM quantization used a dense random orthogonal transformation. That creates two practical costs: multiplying by the transform costs $O(N^2)$, and storing the transform itself requires $O(N^2)$ memory.

Cherniuk and colleagues replace it with

where $\varepsilon$ is a random sign mask.

The operator remains orthogonal, but the DCT gives it structure. The transformation can be executed in $O(N\log N)$ time and does not require keeping an $N\times N$ random matrix.

This is one place where the paper improves both the mathematics and the implementation path. The structured transform is not merely a faster substitute inserted after the algorithm was designed; it is part of a revised vector-level construction for which the authors recover convergence guarantees.

For quantization infrastructure, the narrower Cognaptus inference is that preprocessing architecture matters once model conversion is repeated across large model inventories. Removing a quadratic transform and its dense storage requirement can make a theoretically attractive representation less awkward to operationalize, even if total end-to-end quantization time is not yet best in class.

Four predictable peaks remove a search loop

The second efficiency gain comes from controlling how the two factors are constructed.

The greedy algorithm uses a fixed four-step block: two updates from one dictionary, then two from its transformed counterpart. Those paired early updates create four natural combinations in each factor,

The practical consequence is unusually concrete. The algorithm already knows approximately where the four dominant value groups should appear, so those locations can initialize the 2-bit clusters directly.

The prior Kashin approach relied on multi-restart two-dimensional k-means. Here, the geometry created by the decomposition substitutes for repeated search. The source package reports roughly a tenfold reduction in clustering wall-clock time on medium-sized models relative to that earlier multi-restart procedure.

This is an ablation-like mechanism result rather than evidence that total model conversion becomes ten times faster. It removes one expensive clustering component; the rest of the PTQ pipeline remains.

Standard benchmarks show competitiveness, not dominance

On ordinary 4-bit evaluations, Kashin-DCT generally sits among credible alternatives rather than above all of them.

For Llama-2-7B, WikiText-2 perplexity is 9.60 ± 0.02 for Kashin-DCT+H, compared with 9.80 ± 0.01 for OPTQ and 9.47 ± 0.01 for the restricted QuIP# implementation. On Llama-2-13B, the corresponding values are 8.37 ± 0.01, 8.48 ± 0.00, and 8.31 ± 0.00. C4 and zero-shot HellaSwag, PiQA, and Winogrande results likewise place Kashin-DCT in the competitive range rather than establishing consistent superiority.

That distinction is central to how the method should be evaluated. Its case is not “better perplexity everywhere.” It is competitive quality combined with a different preprocessing structure and evidence of resilience under difficult numerical conditions.

The QuIP# comparison also requires qualification: the paper disables end-to-end fine-tuning and the E8 lattice vector codebook. These numbers therefore compare against a restricted QuIP# variant, not full QuIP# as originally proposed.

Better scaling does not mean faster quantization today

The implementation evidence draws another useful boundary.

On a single H100, Kashin-DCT+H takes 1,313.6 seconds to quantize Llama-2-7B and 1,544.9 seconds for Llama-2-13B. OPTQ takes 229.9 and 437.1 seconds respectively. The restricted QuIP# implementation takes 763.0 and 1,298.0 seconds.

Kashin-DCT therefore has the most favorable 7B-to-13B scaling ratio of the three—about 1.18× versus 1.90× for OPTQ and 1.70× for QuIP#—but it remains slower in absolute wall-clock time at both reported sizes.

For an operator, these are different claims. Favorable scaling suggests that the architecture may become more attractive as dimensions grow or implementations improve. It does not justify choosing the current implementation when conversion latency is the primary constraint.

Two 2-bit factors are still roughly a 4-bit representation

The most tempting misreading is also the easiest to prevent.

Kashin-DCT represents each original weight using two factors, with each factor carrying a 2-bit code. The effective storage is approximately

bits per weight, including small centroid metadata.

So this is not a conventional 2-bit-per-weight model.

The paper does derive an inference identity that could exploit the two-factor form:

With native 2-bit GEMM support and an appropriate fused implementation, that structure could reduce memory-bandwidth pressure and preserve low-bit computation through inference.

But that is an architectural possibility, not a measured production result. The paper does not implement and benchmark the proposed fused serving kernel. Without suitable hardware support, reconstructing a conventional dense weight for ordinary matrix multiplication may be preferable.

PTQ evaluation should include the shape of failure

The paper supports a narrower operational conclusion than a leaderboard reading would suggest.

A team quantizing a well-behaved model once may reasonably prefer a faster mature 4-bit pipeline. The reported Kashin-DCT implementation does not displace OPTQ on absolute conversion time, and its standard benchmark quality is competitive rather than uniformly superior.

The calculation changes when the deployment process must handle heterogeneous model families, uncertain conditioning, or expensive failure recovery. In that setting, average perplexity is incomplete evidence. A quantizer that occasionally diverges, produces NaNs, or silently destroys model quality imposes rerun, diagnosis, validation, and fallback costs that benchmark tables usually omit.

Kashin-DCT makes that risk visible because its algorithmic structure targets both bounded representations and predictable low-bit clustering while also removing two preprocessing bottlenecks: dense random transforms and restart-heavy clustering.

The remaining uncertainty is deployment-side. The robustness evidence comes from a limited number of targeted stress configurations, activation-aware methods such as AWQ and OmniQuant are absent from the comparison, and the proposed 2-bit inference path has not been implemented end to end. The paper therefore gives teams a stronger candidate for robust PTQ evaluation, not a finished argument for replacing optimized production kernels.

The useful shift is in the selection criterion: ask not only how much quality a quantizer loses when everything behaves normally, but what happens when the numerical assumptions behind the conversion stop cooperating.

Cognaptus: Automate the Present, Incubate the Future.