Topics Without Coordinates: What CALLIOPE Gains, and What It Costs

TL;DR for operators

A company analyzing thousands of support conversations needs categories that explain what customers are discussing without exposing their individual messages. Conventional systems commonly group numerical representations of messages, then generate descriptions for the resulting clusters. But geometric similarity does not necessarily produce the most informative topic labels.

CALLIOPE¹ offers an alternative: LLMs recursively summarize documents, synthesize categories from aggregated keywords, and assign records to those categories without embedding-based clustering. Its private version provides a formal document-level differential-privacy guarantee for released topic hierarchies.

Across six public benchmarks, CALLIOPE reports stronger semantic descriptions and reference-label alignment than the private embedding-based Urania baseline, particularly under stringent privacy constraints. The tradeoff is substantial inference cost: on MASSIVE, CALLIOPE uses 576 million tokens versus Urania’s 123 million in the reported non-private maximal-depth comparison. For operators, the decision is whether better aggregate understanding warrants that cost—and whether the privacy guarantee matches the information the organization intends to release.

The missing information in a conversation cluster

Consider an enterprise assistant handling questions about billing, account access, product configuration, and unusual customer requests. Managers need to understand not only how many conversations are similar but also what those similarities mean.

Embedding-based clustering represents each conversation as a numerical vector and groups nearby vectors together. This can produce coherent collections of records, but cluster geometry and interpretability are different objectives. A set of nearby conversations may still be difficult to describe with a precise, informative category name.

The distinction becomes more consequential when privacy constraints restrict the information available for clustering.

In the paper’s private evaluations, CALLIOPE produces more informative topic descriptions than Urania according to Guess-Cluster, a semantic evaluation that asks a judge to identify which topic description corresponds to sampled documents (Section 4, Figure 3).

What makes this result interesting is the architecture. CALLIOPE does not first construct geometric clusters and then ask an LLM to explain them. It makes natural-language categorization the organizing process itself.

CALLIOPE constructs categories before assigning conversations

CALLIOPE uses three recurring operations.

Abstraction converts each document into a concise summary and representative keywords. Instead of treating the document primarily as a point in a numerical space, the system creates a textual representation of its meaning.

Synthesis aggregates keyword evidence from documents and asks an LLM to propose meaningful subtopics within the current category. These categories are generated from corpus-level evidence rather than a fixed taxonomy.

Assignment asks an LLM to place each document abstraction into one of the proposed categories, with an unmatched option for records that do not fit.

The process repeats recursively, producing a hierarchy of progressively more specific topics until a configured maximum depth is reached.

For a support operation, the resulting hierarchy could organize conversations at several levels of specificity rather than forcing every message into one predetermined label set. The framework supports this form of categorization; the actual quality and stability of such an enterprise taxonomy remain deployment questions.

This architecture also changes the computational tradeoff. Embedding-based methods can perform clustering in vector space after representation generation. CALLIOPE repeatedly invokes language-model reasoning to classify documents across hierarchy levels.

The next question is whether the textual evidence used to generate categories can be made private without destroying its usefulness.

Privacy enters when keywords become aggregate evidence

CALLIOPE’s private implementation focuses on the information passed from individual documents into topic synthesis.

A keyword occurring in only one conversation might reveal something distinctive about that record. Simply publishing keyword frequencies—or supplying exact frequencies to a process whose outputs will be released—could disclose sensitive information.

CALLIOPE addresses this using private partition selection.

First, each document contributes at most $c$ keywords. This limits how much one document can influence the aggregate. The mechanism then applies calibrated randomness to keyword counts using a truncated discrete Laplace distribution and suppresses keywords that do not satisfy a privacy-dependent threshold.

Topic or partition sizes are also estimated with a discrete Laplace mechanism rather than disclosed as exact counts.

The LLM synthesizes topics from these privatized aggregates. Because generating labels from differentially private information is post-processing, the generated hierarchy can inherit the privacy protection of the underlying mechanism.

The formal guarantee composes privacy budgets across hierarchy levels, while disjoint partitions within a level permit parallel composition (Section 2.2, Theorem 2.2; Appendix A).

The resulting guarantee is document-level $(\varepsilon,\delta)$-differential privacy: adding or removing one document changes the probabilities of released outputs only within the specified privacy bounds.

This distinction matters because the privacy mechanism protects the released hierarchy under its stated conditions. It does not make the individual input conversations private throughout their entire processing lifecycle.

There is also an interpretability consequence at stringent privacy levels. Stronger filtering can leave fewer informative keywords, producing broader categories. If no keywords survive, the LLM can still propose topics using prior knowledge—but those topics are no longer supported by dataset-specific keyword evidence. A plausible category name should not then be mistaken for an observed pattern.

The benchmark advantage is strongest in semantic description

The authors compare CALLIOPE with Clio and Urania across six public text datasets: MASSIVE, CLINC-OOS, Banking77, Emotion, IMDB, and Reuters.

The evaluated private settings use reported $\varepsilon$ values of 0.1, 1, and 10, with $\delta=10^{-6}$. Configurations are repeated ten times, with repeated semantic evaluations.

The comparisons address three distinct questions.

Evidence: Section 4; Figure 3; Appendix E.2, Figures 9–10. The supplied research record does not provide exact numerical values for these figure-based comparisons.

Guess-Cluster is especially relevant to operational analytics. It tests whether the topic description communicates enough information for a judge to recognize the corresponding documents. This is closer to a manager’s need for comprehensible categories than a measure based exclusively on distances between vectors.

Guess-Odd-One provides a useful qualification. CALLIOPE’s advantage in describing categories is more pronounced than its advantage in grouping coherent documents. Better descriptions therefore do not establish universally better partitions.

The stronger agreement with benchmark labels is additional evidence, although reference labels may not represent the taxonomy an enterprise actually wants.

The paper also examines alternative LLM judges and includes human side-by-side assessments on Banking77, CLINC-OOS, and MASSIVE. These checks strengthen the semantic findings beyond dependence on one judge, but do not eliminate evaluation subjectivity.

One comparison detail needs particular care: Urania’s reported epsilon applies separately to clustering and keyword aggregation, yielding a total budget of $2\varepsilon$. CALLIOPE’s reported epsilon is its total allocated budget. Equal displayed epsilon values therefore do not represent equal total privacy expenditures.

Overall, the benchmark supports CALLIOPE as a competitive semantic alternative under the tested configurations, not a general demonstration that language-based categorization dominates geometric clustering.

Better descriptions require substantially more tokens

CALLIOPE pays for repeated LLM assignments throughout its hierarchy.

Appendix G, Table 1 reports token consumption for non-private runs at the maximal considered hierarchy depth using Gemini 3.5 Flash.

CALLIOPE consumes more tokens on every dataset. On MASSIVE, its consumption is approximately 4.7 times Urania’s. The paper estimates corresponding run costs of approximately $175 versus $40 under the reported pricing assumptions.

These are not measurements of private production deployment. They are evidence of the inference-cost consequences of the architecture in the evaluated non-private configurations.

For a team producing an occasional strategic taxonomy, that expenditure might be acceptable. For one continuously analyzing large volumes of conversations, repeated LLM assignment could become a significant cost driver.

The framework currently processes static datasets and does not establish a streaming update mechanism with controlled cumulative privacy expenditure.

A private hierarchy is not a private conversation system

The formal privacy guarantee establishes an important property of CALLIOPE’s published output. Enterprise deployment introduces several additional boundaries.

First, document-level protection is not person-level protection. A customer contributing many conversations may have a substantially larger presence in the dataset than a single protected document. Person-level guarantees require additional mechanisms and privacy accounting.

Second, the guarantee does not authorize publication of individual document-to-topic assignments. Nor does it automatically extend to supplementary evaluation statistics. The paper explicitly notes that its reported quality metrics are not themselves differentially private and would need additional protection before being published from sensitive data.

Third, differential privacy of the final hierarchy does not protect upstream collection, centralized storage, third-party model inference, or other processing stages. An organization must separately establish how the underlying conversations are handled.

Finally, the experiments use public text datasets and a particular principal LLM configuration. They do not establish equivalent outcomes for sensitive enterprise conversation logs, different model families, or evolving production workloads.

These are scope boundaries of the demonstrated system, not evidence that its privacy mechanism fails.

When the additional inference cost earns its place

For Cognaptus, the paper suggests a practical way to evaluate enterprise conversation analytics.

A customer-support or AI-product team could compare a conventional embedding-based taxonomy with a language-driven hierarchy on the same representative corpus. The evaluation should examine whether managers can identify genuine issue categories and emerging patterns more reliably, while separately measuring partition coherence, token expenditure, and privacy-accounting requirements.

The decisive measure is not simply the number of discovered clusters. It is whether the hierarchy improves a specific operational decision: prioritizing recurring support problems, identifying underserved assistant workflows, or discovering product needs that a fixed taxonomy misses.

That benefit must be weighed against the cost of repeated inference and the protections required for the underlying records.

CALLIOPE’s contribution is to demonstrate that interpretable topic hierarchies and formal document-level privacy do not require embedding-based clustering. Its benchmarks provide encouraging evidence that a language-driven approach can retain semantic usefulness under strong privacy constraints.

Whether that advantage justifies deployment depends on how much better the resulting categories inform decisions—and whether the organization can afford the computation while protecting everything beyond the published hierarchy.

Cognaptus: Automate the Present, Incubate the Future.