Humans in the Loop, Not Just the Dataset
TL;DR for operators AI-assisted monitoring does not become trustworthy because a human occasionally clicks “wrong label.” It becomes useful when the whole product is designed to capture, validate, resolve, and redeploy human judgement. The paper behind this article studies an open-source Telegram monitoring tool being developed with civil society organisations, using conspiracy-theory classification as the working scenario.1 Its practical contribution is a workflow: Telegram posts are classified, CSO users review labels during their normal monitoring work, their feedback is stored with metadata, and that accumulated feedback becomes a gold-standard dataset for model evaluation and refinement. ...