Statistics

Generative AI still ships answers without warranties. Edgar Dobriban’s new review, “Statistical Methods in Generative AI,” argues that classical statistics is the fastest route to reliability—especially under black‑box access. It maps four leverage points: (1) changing model behavior with guarantees, (2) quantifying uncertainty, (3) evaluating models under small data and leakage risk, and (4) intervening and experimenting to probe mechanisms. The executive takeaway If you manage LLM products, your reliability roadmap isn’t just RLHF and prompt magic—it’s quantiles, confidence intervals, calibration curves, and causal interventions. Wrap these around any model (open or closed) to control refusal rates, surface uncertainty that matters, and measure performance credibly when eval budgets are tight. ...

When teams push large text workloads through LLMs (contract triage, lead deduping, safety filtering), they face a brutal choice: pay for the “oracle” model (accurate but pricey) or accept quality drift with a cheaper “proxy”. Model cascades promise both—use the proxy when confident, escalate uncertain items to the oracle—but in practice they’ve been fragile. SUPG and similar heuristics often over‑ or under‑sample, rely on asymptotic CLT assumptions, and miss targets when sample sizes are small. The BARGAIN framework fixes this by combining task‑aware adaptive sampling with tighter finite‑sample tests to certify targets while maximizing utility (cost saved, recall, or precision). The authors report up to 86% more cost reduction vs. SUPG for accuracy‑target (AT) workloads, and similarly large gains for precision‑target (PT) and recall‑target (RT) settings—with rigorous guarantees. ...