The Benchmark That Keeps Finding New Failures
TL;DR for operators A model can save inference memory yet lose the information needed to answer correctly. In a constructed ledger-retrieval test, the latest record appears midway through a long context, with competing outdated records later. Uncompressed inference achieves 0.93 accuracy; FastKVZip compression achieves only 0.35. Established benchmarks may not expose such application-specific failures.1 ...