Don’t Give Every Query the Same Context Budget
TL;DR for operators A retrieved prompt can contain one useful sentence or evidence distributed across several passages. Giving both requests the same compressed-context budget assumes that the amount of information worth preserving is constant. The evidence in ATACompressor1 suggests that assumption is costly: on MSMARCO with LLaMA-2-7B, removing adaptive allocation reduced F1 from 50.06 to 47.52 and compression ratio from 27.36x to 19.06x; removing selective compression reduced F1 further to 40.83. ...