Clean Less, Route Better: DataOrchestra Reframes Pretraining Data Curation
TL;DR for operators A pretraining-data pipeline receives millions of uneven records. Some are unusable, some contain removable noise, some need structural repair or added explanation, and some are already valuable enough that further processing may damage them. The operational problem is therefore not how to apply more cleaning, but how to decide which intervention—if any—each example needs. ...