Thirteen Buckets and a Warning Light

TL;DR for operators

Most organizations do not have a feedback shortage. They have a conversion problem.

Comments arrive in multiple languages, contain personal information, mix several complaints in one paragraph, and rarely align themselves politely with the categories used in management reports. Human teams then compress this material into a few operational labels, often under time pressure and with rules inherited from whichever spreadsheet survived the last reorganization.

The paper examined here proposes a more disciplined conversion pipeline.¹ It de-identifies English and French feedback, uses a domain-adapted language model to score each comment against 13 predefined Service Quality Elements, turns the model’s prose into structured data, asks specialists to validate the results, and then applies bootstrapped regression to identify demographic-topic relationships that appear to be emerging, persistent, or disappearing.

For operators, the useful conclusions are these:

• The value comes from the pipeline, not the language model alone. Privacy processing, taxonomy design, structured outputs, expert review, and statistical monitoring are all necessary components.
• Domain adaptation appears materially important. Similarity with expert-assigned topics rises from 24.27% for the prompted baseline to 66.64% for the fine-tuned and quantized model.
• That increase is not a clean fine-tuning ablation. The comparison also changes the base model and deployment configuration, so the paper does not isolate which component produced how much of the gain.
• The trend layer is an alerting mechanism. It identifies changes worth investigating; it does not establish discrimination, service failure, or causality.
• The approach is suitable for workflow augmentation, not unattended adjudication. The strongest business case is faster triage and more systematic monitoring while experts retain responsibility for interpretation and intervention.

The central lesson is pleasantly unglamorous: useful AI systems often succeed by placing a probabilistic model inside a process that knows what the model is allowed to mean.

The feedback problem begins before the model sees a word

Consider a familiar complaint:

“I called three times, the French instructions were unclear, and nobody could tell me whether my case had been escalated.”

A sentiment model can confidently announce that the customer is unhappy. This is accurate in roughly the same way that observing smoke establishes that something is warm.

An operations team needs more. Is the complaint about timeliness, clarity, official-language service, access to escalation, professionalism, or all five? Which categories belong in the service dashboard? Which department should investigate? Is this concern becoming more frequent among a particular group, or did three unusually verbose respondents simply have a difficult Tuesday?

The paper works backward from those operational questions. Its system is not asked to discover an elegant latent map of human dissatisfaction. It is asked to translate comments into an existing institutional language: 13 Service Quality Elements covering access to escalation, accessibility, availability, clarity, completeness, consistency, convenience, findability, information accuracy, information format, official languages, professionalism, and timeliness.

That distinction matters. The authors call part of the process topic modeling, but operationally it is closer to taxonomy-constrained, multi-label classification. The categories are known in advance. The model’s job is to judge how strongly each one applies and provide supporting text.

This makes the system less adventurous than unsupervised topic discovery and considerably more deployable. Managers can connect its outputs to existing teams, service standards, and reporting structures. A newly discovered topic called “bureaucratic frustration cluster 7” may delight a visualization dashboard. “Timeliness” already has an owner.

It also creates the first important boundary: the system can only see the service through the taxonomy it has been given. A category set is not a neutral window onto customer experience. It is an organizational theory of what counts.

The architecture is a sequence of controlled translations

The paper’s main contribution is best understood as a chain of translations:

Raw feedback → privacy-safe text → expert categories → structured scores → validated records → demographic trend signals → human investigation

Each stage changes what the data can be used for. Each also creates a possible failure point.

The language model sits in the middle, not at the top. That is good system design. It is also less marketable than putting a chatbot in front of the same data and calling it a transformation strategy.

Privacy is part of the model pipeline, not a legal appendix

The dataset contains 8,161 feedback records: 6,515 in English and 1,646 in French. Tax feedback can include names, phone numbers, email addresses, account details, monetary values, locations, identification numbers, and enough narrative context to make nominally anonymous records rather less anonymous than advertised.

The authors therefore begin with de-identification. They combine a multilingual named-entity recognition model with custom regular expressions designed for structured identifiers and organization-specific patterns. Experts then review flagged examples.

This hybrid design has a simple rationale. A general named-entity model can recognize conventional people and places, but real service feedback is rarely composed for the convenience of a benchmark. Identifiers appear with typos, punctuation, abbreviations, partial numbers, copied email signatures, and local formats. Rules catch predictable structures; the model catches linguistic variation; reviewers catch what both miss.

The preprocessing retains analytically useful information where possible, including percentages, months, generalized geography, and selected non-sensitive numeric content. That is not merely data cleaning. It is a negotiation between privacy and explanatory value.

Mask everything, and the system loses distinctions needed to understand the complaint. Preserve everything, and the analytics program becomes an impressively efficient method for moving sensitive information into new places.

For businesses, this stage is not optional plumbing. Any attempt to reproduce the pipeline requires a field-level retention policy:

1. What must always be removed?
2. What may be generalized?
3. What must remain for the classification task?
4. Which environments may process each version?
5. Who reviews de-identification failures?
6. How are corrections propagated into stored and derived datasets?

The paper describes the de-identification mechanism but does not report formal recall estimates for detecting sensitive information. Organizations should therefore treat the proposed method as an architecture to validate, not as evidence that a particular named-entity model plus a respectable pile of regular expressions has solved privacy.

The taxonomy is the system’s operating contract

After de-identification, the model evaluates every comment against the 13 Service Quality Elements. It assigns relevance scores from 1 to 5 and provides a word or sentence from the feedback as justification.

This output design does three useful things.

First, it permits multi-label interpretation. A comment can concern both clarity and timeliness rather than being forced into one administratively convenient bucket. The paper notes that the existing expert process assigns one primary element, whereas the proposed approach can associate feedback with several.

Second, the numeric scores create inputs for downstream analysis. Text becomes a matrix of feedback records by service categories.

Third, the supporting excerpts give reviewers something to inspect. They do not make the model internally interpretable, but they make each assignment easier to challenge. There is a difference. A model quoting the relevant phrase is evidence of grounding, not a notarized transcript of its reasoning.

The taxonomy also explains why generic language-model performance is insufficient. “Clarity” and “information format” may overlap in ordinary language. “Availability” and “accessibility” may overlap in service operations. “Access to escalation” can be a process state, an emotional demand, or both. The correct distinction depends on institutional manuals and prior labeling practice.

This is where domain adaptation enters.

Fine-tuning teaches the model the organization’s dialect

The paper compares two approaches.

The baseline uses prompt engineering with Zephyr-7B. The prompt describes the categories and asks the model to produce relevant assignments.

The adapted approach uses Mistral-7B-Instruct-v0.2 with parameter-efficient fine-tuning. The authors use LoRA rather than updating the entire model, prepare the base model for low-bit training, use NF4 and bfloat16 during the adaptation process, and subsequently apply 4-bit GPTQ quantization for deployment.

The operational thesis is straightforward: organizations have developed their own terminology, distinctions, priorities, and labeling habits. A generic model may understand the words while missing the local classification rule. Fine-tuning is intended to teach that rule without requiring a very large model or full-parameter retraining.

The main quantitative result supports the general thesis. Similarity with expert labels is approximately:

That is a gain of 42.37 percentage points. It is too large to dismiss as formatting polish.

But it must be described accurately. The experiment is a bundled system comparison, not a controlled ablation. The two conditions differ in base model, domain adaptation, and quantization status. The paper does not report:

• the fine-tuned model before quantization;
• the same Mistral model under prompt-only classification;
• the quantized and unquantized variants under identical evaluation;
• repeated runs or uncertainty intervals for the similarity scores;
• separate similarity results for English and French;
• latency, memory, throughput, or cost measurements.

The evidence therefore supports this statement:

The domain-adapted deployment configuration aligns substantially better with the organization’s labels than the prompted baseline configuration.

It does not support the more convenient statement:

Fine-tuning alone caused a 42.37-point improvement, while quantization preserved all accuracy and delivered a measured infrastructure saving.

The second sentence is the one likely to appear in a procurement deck. It is also the one the experiment did not establish.

Quantization solves a deployment constraint that the paper does not benchmark

The use of 4-bit quantization is commercially relevant because many public institutions and regulated firms cannot casually route sensitive feedback through the largest hosted model available that quarter.

A seven-billion-parameter model compressed to low precision offers a plausible route to deployment on more modest infrastructure. Parameter-efficient fine-tuning also lowers the adaptation burden relative to full-model training.

The paper explains this architectural motivation clearly. What it does not provide is a deployment benchmark. There are no reported measurements for GPU memory, inference speed, energy use, batch throughput, or cost per processed comment.

Quantization should therefore be treated as an implementation detail with credible operational logic, not as experimentally demonstrated ROI.

For a business implementation, the missing evaluation is simple enough to specify:

Until those measurements exist, “resource-efficient” is a design intention. It may be a very sensible one. Sensible intentions remain stubbornly resistant to accounting treatment.

Turning generated prose into statistical inputs

Language models produce text. Regression models prefer columns.

The paper bridges that gap through a digitization step. The generated response is parsed into separate fields for category, relevance score, and justification. Regular expressions convert the formatted output into a dataframe that can be used for trend analysis.

This is easy to overlook because it lacks the glamour of fine-tuning. It is also where many prototypes quietly expire.

A production pipeline must handle malformed responses, missing categories, repeated labels, out-of-range scores, French punctuation, unexpected explanations, truncated output, and model-version changes. If parsing fails, the system should reject or quarantine the record rather than improvise a zero and continue downstream with the serene confidence of a spreadsheet formula.

The paper does not report parsing failure rates or schema-validity metrics. A deployed version should. Once generated text becomes statistical data, output conformance is part of model quality.

A more robust implementation could use constrained generation or a validated JSON schema rather than extracting structure from free-form text after the fact. The business point is broader: an LLM classification system is not complete when the answer looks correct on screen. It is complete when the answer can enter a governed data pipeline without requiring optimism as a serialization format.

Human validation tests usefulness, but not equivalence

The paper evaluates model alignment in two ways.

The first is the dataset-level similarity comparison already described. This is the primary evidence that the adapted configuration better reproduces expert labels.

The second is a small officer evaluation. Five reviewers assess model-assigned Service Quality Elements for 10 randomly selected bilingual feedback texts, producing 51 category instances. Pairwise t-tests compare model and officer scores. For the most divergent evaluator comparison, the paper reports $t=-1.968$ and $p=0.085$, which is not statistically significant at the 5% level.

This is best interpreted as a limited human applicability check. The reviewers did not find a conventionally significant average difference in that small sample.

It is not evidence that the model and experts are equivalent.

Failure to reject a difference is not proof that the difference is acceptably small. An equivalence claim would require a prespecified tolerance and an equivalence-testing design. The effective independence of 51 category instances drawn from only 10 texts is also debatable, because several scores can originate from the same comment and reviewer.

There is an additional wrinkle: the paper says this survey evaluates the model with the lower similarity score. That makes the result interesting—the poorer exact-match model may still produce judgments officers consider plausible—but it complicates any attempt to treat the survey as validation of the fine-tuned configuration specifically.

The two evaluations answer different questions:

The paper is strongest when read as a system demonstration with encouraging alignment evidence. It becomes less secure when every non-significant test is promoted to a certificate of human-level judgment.

The statistical layer turns categories into warning signals

Once each feedback record has structured topic indicators, the pipeline compares demographic-topic relationships over time.

The paper considers age group, gender, and preferred language. It divides the year into earlier and later periods, with either a recent-quarter-versus-prior-year framing or a half-year split. Separate logistic regressions are fitted for each Service Quality Element and time period. The authors use bootstrap resampling to estimate coefficient distributions and 95% confidence intervals.

A relationship is then described as:

• Emerging when a demographic feature becomes statistically significant in the later period;
• Persistent when it remains significant across both periods;
• Disappearing when it was significant earlier but not later.

The illustrative results include disappearing accessibility concerns among respondents under 19, emerging clarity concerns among older and middle-aged groups, and emerging timeliness concerns associated with gender and English-language preference.

These examples show what the mechanism can produce. They should not be read as established findings about taxpayers in general. The paper intentionally anonymizes the authority, uses a single year of feedback, and presents the trend table as an illustrative example.

The correct operational interpretation is:

The system detected a change in the statistical relationship between a demographic field and the probability that submitted feedback was assigned to a particular service category.

That is quite different from:

The organization became biased against that demographic group.

Between those statements lie several unresolved questions:

• Did the service experience change?
• Did the composition of feedback submitters change?
• Did a campaign, deadline, outage, or policy event alter complaint volume?
• Did the model classify one group’s language differently?
• Did the category’s overall frequency change?
• Did sample size affect significance?
• Did seasonal patterns make the two periods incomparable?
• Was the observed transition one of many tests that would occasionally appear significant by chance?

A warning light does not explain why the engine is warm. This is not a defect in warning lights.

“Emerging” means statistically newly visible, not necessarily newly created

The emerging-persistent-disappearing vocabulary is intuitive, which makes it dangerous.

A coefficient can become significant because its estimated magnitude increased. It can also become significant because uncertainty decreased, sample size grew, or the composition of the data changed. Conversely, a relationship can lose significance while remaining similar in magnitude if the later period contains fewer observations.

For business use, the alerting rule should therefore incorporate more than significance status. A mature implementation would report:

• coefficient or odds-ratio magnitude;
• confidence interval width;
• record counts for the group and period;
• absolute topic prevalence;
• change in prevalence;
• classification confidence or review rate;
• sensitivity to alternative time splits;
• correction or prioritization rules for multiple comparisons.

The paper recognizes the seasonality problem. Comparing two semesters from the same year may capture the tax calendar rather than a durable change in service. Comparing the most recent quarter with a much longer baseline can dilute weak new signals. The authors correctly suggest that equivalent periods across multiple years would provide a stronger basis for trend interpretation.

This limitation is not decorative. It changes what the output is for.

With one year of data, the system is appropriate for generating investigation queues. It is not yet a dependable instrument for declaring long-term demographic trends.

The system detects differences in feedback, not bias itself

The likely reader misconception deserves a direct correction.

This is not an autonomous bias detector.

The paper repeatedly frames demographic-topic differences as potential indicators of unequal service experiences. That is reasonable. But the data are submitted feedback, not a randomized measurement of service quality across the population.

Feedback datasets are shaped by at least three processes:

1. who experiences a problem;
2. who chooses and is able to submit feedback;
3. how the system classifies what they write.

A demographic difference can enter at any of those stages.

Preferred-language patterns are especially delicate. They may reflect genuinely different service experiences, different submission volumes, translation or classification error, differences in available channels, or the fact that language preference is itself entangled with service accessibility.

The paper does not report classification accuracy separately by demographic group or language. Consequently, the trend layer assumes that any model error is sufficiently stable and comparable across groups. That assumption should be tested before demographic alerts influence resource allocation or fairness investigations.

The proper business label is not “bias detected.” It is something closer to:

Demographic service-feedback anomaly requiring review

Less exciting, certainly. Also less likely to produce an inquiry six months later.

What the business case actually looks like

The paper’s commercial relevance extends beyond tax administration. Banks, insurers, hospitals, utilities, universities, transport operators, telecom providers, and large marketplaces all receive high-volume feedback in language that does not neatly match internal reporting categories.

The transferable pattern is:

1. Define the decisions before selecting the model

The 13 categories work because they correspond to an operational framework. Another organization should begin with the queues, owners, escalation rules, and service standards the output must support.

An elegant taxonomy without decision rights becomes a new dashboard. Organizations rarely suffer from an insufficient number of dashboards.

2. Treat domain labels as product infrastructure

Definitions, examples, counterexamples, overlap rules, and adjudication procedures should be versioned. When the organization changes a category definition, the model, historical data, and trend baselines may all require revision.

3. Separate semantic automation from statistical claims

The LLM answers: “Which service issues appear in this comment?”

The regression answers: “How does the occurrence of those assigned issues vary across groups and periods?”

Neither component should casually inherit the authority of the other. A polished explanation from the LLM does not validate the trend model. A significant coefficient does not validate the classification.

4. Route uncertainty instead of hiding it

Low-confidence assignments, novel language, conflicting categories, de-identification concerns, and high-impact demographic alerts should receive more human attention. Routine, high-confidence cases can receive less.

The goal is not to remove people from the workflow. It is to stop paying specialists to reread thousands of obvious comments while preserving their attention for the ambiguous and consequential ones.

5. Measure workflow economics, not only model alignment

A production pilot should track:

• reviewer minutes per record;
• percentage of records accepted without edits;
• percentage requiring category correction;
• missed sensitive information;
• parsing and schema failures;
• alert investigation yield;
• time from feedback submission to operational action;
• compute and support cost per thousand records;
• performance by language and relevant demographic group.

A 66.64% label similarity may be useful or inadequate depending on where the remaining errors occur. Misclassifying “convenience” as “findability” is not equivalent to missing an official-language issue that triggers a compliance review.

A practical governance design

The pipeline implies a sensible division of responsibilities.

The human-in-the-loop component should be risk-based rather than ceremonial. Asking an officer to approve a sample after deployment is not governance if the model’s outputs have already entered performance reporting, budget decisions, or fairness claims.

Review should occur at several points:

• before training, to establish label quality;
• during testing, to estimate error patterns;
• during operation, to inspect low-confidence or high-impact cases;
• after taxonomy changes, to recalibrate historical comparisons;
• after alerts, to distinguish model artifacts from service conditions.

The paper embeds experts in de-identification and evaluation. A production system should extend that principle into ongoing monitoring.

The boundaries that matter

The paper demonstrates a credible architecture, but several limitations determine how far its conclusions can travel.

One organization, one operational vocabulary

The model is adapted to a particular tax-service environment. That is the point of the method and the reason generalization cannot be assumed. Another organization would need its own taxonomy, examples, reviewers, privacy rules, and validation.

One year cannot establish durable trends

Seasonality is likely in tax administration. The authors acknowledge that multi-year comparisons between equivalent periods would be stronger. Until then, emerging and disappearing labels should initiate review rather than settle it.

The human evaluation is small

Ten feedback texts and 51 category instances provide a useful qualitative check, not a production-grade estimate of reviewer agreement or model reliability.

The comparison does not isolate components

The reported improvement combines a different model, fine-tuning, and quantization. Additional controlled comparisons are needed before attributing the gain to any single choice.

Efficiency is asserted architecturally, not measured operationally

The low-bit design plausibly reduces resource requirements, but the paper does not report memory, latency, throughput, or cost benchmarks.

Feedback is a selected sample

People who submit feedback are not necessarily representative of everyone who uses a service. Demographic trends in comments should not be treated as population estimates without additional data and weighting assumptions.

The taxonomy can conceal what it does not name

Predefined categories improve operational alignment but constrain discovery. Organizations should retain a mechanism for reviewing uncategorized text, category drift, and recurring explanations that do not fit the existing scheme.

The real product is disciplined escalation

The paper does not provide a machine that reads customer comments and announces where an institution is unfair.

It provides something more useful: a design for converting multilingual feedback into structured, reviewable signals while preserving privacy controls, expert responsibility, and statistical uncertainty.

Its language-model result is encouraging. Moving from 24.27% to 66.64% similarity suggests that organizational language and labeling practice are not trivial details that a sufficiently eloquent general-purpose model will absorb from a prompt. Domain adaptation matters.

Its larger contribution, however, is the placement of that model. The LLM performs the fuzzy semantic task for which it is suited. Rules and specialists protect sensitive information. Structured parsing connects generated outputs to analytics. Regression provides an explicit method for comparing periods and groups. Humans decide what the resulting signal means and what, if anything, should be done.

That is the right hierarchy.

The model categorizes. The statistics alert. The organization investigates.

A surprising amount of responsible AI consists of refusing to let those verbs swap places.

Cognaptus: Automate the Present, Incubate the Future.