Data-Leakage

TL;DR for operators An autonomous data science agent can generate executable code, report an excellent predictive score, and still produce an invalid modelling pipeline. In a benchmark covering 1,761 pipelines across 220 tasks, methodological failures were widespread, and data leakage was associated with better evaluation rankings. More sophisticated agents eliminated some basic omissions without consistently preventing leakage. Controlled tests revealed weaknesses in recognizing causal and feature-admissibility constraints. ...

TL;DR for operators A held-out test set does not necessarily tell you how a sign-language translation system will behave with a new user. In Rethinking Sign Language Translation: The Impact of Signer Dependence on Model Evaluation, Artiaga and colleagues re-evaluate three gloss-free systems after excluding the test signer from training and development.1 ...

TL;DR for operators Synthetic data privacy reviews should stop treating every rare match as proof of memorization. That is the useful correction in Phantoms and Disclosures: a Causal Framework for Auditing Synthetic Data, a paper that turns synthetic-data auditing into a controlled experiment rather than an anxious string search.1 The paper’s mechanism is simple enough to be dangerous in the right way: split the source corpus into training and holdout records; generate synthetic data from the training split; extract rare features from training, holdout, and synthetic data; then ask whether synthetic matches are disproportionately concentrated in the training split. Matches against training records are potential true disclosures. Matches against holdout records are phantom disclosures: things that look like leaks but could have appeared even if that record had never been used. ...