What 99% Review Coverage Fails to Measure

TL;DR for operators

A system can produce structured records for almost every customer review while missing much of the information those records are supposed to capture.

In a study of Pashupatinath Temple reviews, a culturally adapted GPT-4o pipeline generated entity outputs for 99.4% of 2,500 reviews, yet achieved only 28.4% provisional entity micro-F1, with 21.8% recall, against a human-adjudicated reference. Its aspect-sentiment results showed a similar gap: 79.4% polarity agreement when both the model and reference detected an aspect, but only 42.6% F1 when correct aspect detection and sentiment classification were both required.

The contribution is a domain-specific extraction schema, a component-wise evaluation procedure, and a method for constructing semantic networks from the outputs. For heritage managers and tourism-analytics providers, these provide a starting point for organizing feedback about rituals, spiritual experiences, restrictions, and visitor services.

They do not establish that the resulting dashboard accurately represents visitor experiences or is ready to guide operational decisions. That requires separate quality checks for each extraction component, traceable source evidence, and validation in the intended management setting.

Almost every review produces an answer

A heritage manager reviewing thousands of visitor comments wants to know which experiences recur, which services cause dissatisfaction, and which issues deserve attention. A structured dashboard seems attractive: instead of reading every review, the manager can examine categorized experiences, sentiment trends, and relationships among recurring topics.

But record availability does not tell the manager whether those categories represent what visitors actually described.

Hari Sitaula and colleagues examine this problem through English-language Tripadvisor reviews of Pashupatinath Temple in Kathmandu, Nepal.¹ Their extraction pipeline produced entity records for 2,485 of 2,500 sampled reviews. Yet a separate evaluation using 200 human-annotated reviews found substantial disagreement between generated entities and the reference.

Of 1,714 adjudicated entity mentions, the system matched only 373 under the study’s strict normalized text-and-type rule. Precision was 41.0%, meaning 41% of evaluated predictions matched the reference. Recall was 21.8%, meaning the system recovered fewer than one-quarter of reference mentions. Their combined micro-F1 was 28.4%, with a 95% bootstrap confidence interval of 25.9–31.0%.

These figures describe the evaluated extraction workflow, not the general capability of GPT-4o. The entity score is also provisional because the evaluated export cannot be fully regenerated from the available processing code.

The immediate lesson concerns measurement: a populated dashboard measures successful output production. It does not establish successful information recovery.

Sacred heritage requires a different extraction vocabulary

A conventional tourism-review schema can identify facilities, scenery, problems, and overall satisfaction. Living sacred heritage introduces additional distinctions.

A visitor might describe a cremation ritual, a religious actor, a restricted space, and an emotionally complex experience within the same review. A positive spiritual experience can coexist with dissatisfaction about crowding or services. Treating the entire review as positive or negative loses information relevant to different management decisions.

The researchers adapted the existing LLM-Social Computing (LLM-SC) framework by retaining four entity types and adding eight: ritual, religious_actor, sacred_space, spiritual_emotion, festival_event, cultural_rule, sacred_object, and dual_valence.

The resulting prompt specified 12 entity categories, four relation types, and nine aspect-sentiment dimensions. It also included evidence requirements and exclusion rules. The adaptation changed the extraction schema and prompt; it did not fine-tune GPT-4o’s model weights.

This specialization substantially changed what the pipeline could represent. The eight added categories accounted for 7,308 of 11,286 entity records, or 64.8%. Ritual alone contributed 1,741 records, while spiritual emotion contributed 1,462.

That result demonstrates extensive use of the expanded vocabulary. It does not demonstrate accurate recognition.

For example, the dual_valence category generated 220 records but achieved 0% F1 in the reported entity evaluation. The conceptual ability to represent simultaneous positive and negative experiences is therefore separate from the ability to detect them dependably.

Three evaluations reveal different failure points

The study’s component-wise evaluation is particularly relevant to teams designing review-analysis systems.

The sentiment evaluation shows why component boundaries matter.

Across 200 reviews and nine aspects, the reference contained 1,800 review–aspect decisions. The model achieved 53.6% F1 for identifying whether an aspect was present. Among the 267 pairs where both the model and reference identified the aspect, polarity agreed in 212 cases, producing 79.4% conditional accuracy.

When evaluation required the correct aspect and polarity together, F1 fell to 42.6% (95% CI 38.7–46.6%).

A product team reporting only conditional polarity accuracy would therefore omit the effect of missed and incorrectly detected aspects.

Relation extraction introduces another limitation. Human adjudication supported 153 of 194 evaluated relation candidates, or 78.9%. However, only 106 candidates—54.6% of the evaluated set—were both supported and assigned the adjudicated relation type.

Crucially, evaluators selected the first-listed relation candidate from each eligible review. This was neither a random nor an exhaustive evaluation of the 5,590 generated relation records. Whole-output relation precision and recall remain unknown.

These are main component-evaluation results, not interchangeable estimates of a single overall accuracy measure.

The semantic network organizes outputs, not verified facts

The pipeline also converts extracted entities and relationships into semantic networks.

The construction procedure distinguishes several aggregation levels:

• 5,926 nodes in the merged entity inventory, including relation endpoints absent from the entity table.
• 5,110 nodes appearing in the relation network.
• 5,590 relation records, aggregated into 5,341 typed edge groups.
• 5,221 unique undirected connections after collapsing relation labels and excluding self-loops.

These definitions prevent a common analytical error: treating raw extraction records, typed relationships, and simplified graph connections as equivalent quantities.

The largest connected component contained 3,269 nodes, representing 63.97% of relation-network nodes. This describes the structure of model-generated relationships. It cannot independently identify the causes of visitor satisfaction or establish real-world causal connections, even when the model assigns a causal relation label.

A post hoc duplicate-handling sensitivity analysis provides a narrower check. Removing one analysis-sample review reduced undirected connections from 5,221 to 5,220 and increased connected components from 800 to 801. Most aggregate statistics barely changed.

This sensitivity test examines the effect of a particular data-cleaning change. It does not validate relation correctness or demonstrate robustness across models, sites, or collection periods.

What a heritage-monitoring product would need

For heritage-site managers, the plausible application is structured feedback triage.

If a site receives a substantial volume of accessible, permissioned visitor reviews, a specialized taxonomy could help distinguish concerns about cultural restrictions from complaints about facilities, or spiritual appreciation from environmental dissatisfaction. These distinctions could guide which review excerpts managers inspect before deciding whether to investigate a service issue.

For a tourism-analytics provider, the research suggests several product requirements.

First, measure output coverage and correctness separately. Entity recall, aspect detection, polarity classification, relation support, and schema compliance need their own acceptance criteria. Quality should also be examined by category, since aggregate scores can conceal severe weaknesses in culturally distinctive labels.

Second, preserve evidence traceability. Extracted claims should remain linked to source passages where lawful access permits, with uncertain or unsupported labels made identifiable. The decision-maker needs to inspect what the system extracted before treating frequency estimates or graph relationships as meaningful.

Third, preserve model and processing provenance. Recording the model snapshot, extraction prompt, output validity checks, filtering decisions, and derived-data lineage is necessary for interpreting changes between monitoring periods.

These requirements follow from the observed errors and reproducibility gaps. They are Cognaptus operational inferences, not deployment outcomes measured by the study.

The evidence stops before deployment reliability

The evaluation uses one sacred-heritage site, one English-language review source, and a 200-review human-reference subset. Both annotators were study authors. Their initial exact entity mention-and-type pairwise F1 was 38.9%, illustrating the difficulty of constructing a consistent reference for this task.

The bootstrap intervals quantify review-sampling uncertainty, but not every source of annotation, selection, or processing uncertainty. The study also did not track possible overlap between taxonomy-development reviews and the evaluation subset.

Reproducibility has further constraints: the exact dated GPT-4o snapshot was not recorded, original inference responses are unavailable, and the sanitized entity export has incomplete lineage. The research also leaves third-party review-supply authorization unresolved.

Finally, the study provides no matched quantitative comparison against the original Qwen-Max-based LLM-SC implementation or alternative extraction systems. It does not test cross-site transfer, live monitoring latency, or management outcomes.

Consequently, the results cannot establish that the adapted pipeline outperforms competing approaches or improves heritage operations in practice.

Representation is the beginning of reliability

This study makes a concrete methodological contribution: it defines culturally specific information that a general tourism schema may overlook, tests distinct extraction components against human references, and specifies how extracted relationships become a semantic network.

Its evaluation also exposes why those contributions should not be confused with validated operational performance.

For organizations deploying domain-specific review analytics, the appropriate next step is to establish component-level acceptance thresholds, maintain reproducible extraction records, and test whether the resulting evidence improves decisions in the actual operating environment.

A dashboard can contain records for nearly every review. Whether it contains the right information remains a separate, measurable question.

Cognaptus: Automate the Present, Incubate the Future.