LLM Annotation

TL;DR for operators A conversation-analysis system can return the wrong label for at least two different reasons: it may have misunderstood what happened in the conversation, or it may have correctly recognized the behavior and then mapped it incorrectly to the construct being measured. A single LLM label hides that distinction. EduBehaviors1 separates the two steps. The system first records human-readable yes-or-no judgments about observable behaviors, then uses a separate rule or classifier to convert those judgments into the final label. In the paper’s Teacher TalkMoves evaluation, the word-only baseline reached macro-F1 0.339 and Cohen’s kappa 0.328. The strongest reported assertion-based configuration reached macro-F1 0.673 with kappa 0.688. ...

TL;DR for operators LLM annotation is not governed by the prompt as cleanly as procurement decks would prefer. The paper behind this article shows that models bring their own internal concept boundary to definition-driven classification tasks, and that boundary can dominate the user’s intended definition even when the prompt looks explicit.1 The practical result is simple: before using an LLM as an annotator, judge, moderator, reviewer, triage engine, or rubric scorer, test whether its internal understanding of the label matches your operational definition. The paper introduces Definition-Specific Familiarity (DSF) as a lightweight proxy for that fit. DSF is positively associated with model accuracy after controlling for dataset difficulty, while three text memorization metrics are not. ...