Summarization

Multimodal AI has reached the point where it can describe videos, summarize documents with images, answer visual questions, and generate outputs that look satisfyingly complete. This is exactly why evaluation is becoming more dangerous. A system that looks competent is not necessarily reliable. It may miss the one-second event that determines the answer. Or it may notice enough evidence but then produce a fluent, attractive, visually decorated summary that quietly distorts the facts. The first failure is upstream: the model did not capture the decisive evidence. The second is downstream: the output did not preserve and present the evidence in a human-useful way. ...

The dashboard says the judge is fine. The document disagrees. Judge is an easy word to trust. It suggests robes, procedure, and someone in the room who is supposed to be less confused than everyone else. In AI evaluation, the word has become dangerously comfortable. Product teams now use LLMs to score summaries, rank chatbot answers, approve RAG outputs, compare model releases, and decide whether another model’s response is “good enough.” The attraction is obvious: human review is expensive, slow, and occasionally insists on context. An LLM judge is fast, scalable, and does not ask why the evaluation rubric was written five minutes before the sprint review. ...

How to design a document summarizer as a lightweight product, with summary types matched to workflow, section-aware processing, and source traceability.

What this demo proves, what it does not prove, how to evaluate it responsibly, and what would be required to turn it into a production summarization workflow.

TL;DR for operators When an LLM summarises a review, policy memo, support ticket, medical note, or news item, the operational question is not only “Did it get the facts right?” The sharper question is: did it change what the user is likely to believe, prioritise, or buy? The paper behind this article studies exactly that problem. It treats LLM-generated content as a decision interface and measures three ways the interface can quietly bend human judgment: changing the sentiment frame of the source, overemphasising the beginning of the source, and fabricating confident answers for events beyond the model’s knowledge cutoff.1 ...