TL;DR for operators

Hierarchical question-based embeddings offer two distinct ways to manage interpretability: use fewer semantic questions in the representation, or summarize detailed questions through broader explanations.

Across 20 benchmark datasets, a representation with 3,462 questions retains nearly all the semantic similarity performance of one with 13,914 questions, but its average retrieval score falls more substantially. The optimal level of detail therefore depends on what the embedding must accomplish.

A separate alignment mechanism reduces the number of explanatory questions without changing the underlying embedding or its predictive performance. This improves a question-count-based interpretability measure, not demonstrated human comprehension.

For enterprise search and document-classification workflows, the research supports testing configurable semantic granularity and progressively disclosed explanations. Retrieval quality, explanation accuracy, user comprehension, and operating costs still require separate evaluation.

Removing 75% of the Questions Has Unequal Consequences

An enterprise search system retrieves two related documents. An analyst can see that the system considers them semantically similar, but cannot immediately identify which attributes connect them.

One approach is to represent each document through answers to understandable yes/no questions. The analyst can then inspect shared attributes rather than infer meaning from numerical embedding coordinates. Yet when the representation contains thousands of questions, the explanation itself becomes difficult to examine.

Reducing those questions seems attractive, provided the system retains the distinctions needed to distinguish relevant from irrelevant documents.

Kohei Oda and colleagues investigate this problem in Interpretable Text Embeddings Based on Hierarchical Questions.1 Their results reveal that reducing semantic detail has markedly different consequences for similarity measurement and retrieval.

At a binary-answer threshold of 0.5, reducing the representation from 13,914 questions to 3,462 lowers average semantic textual similarity (STS) performance from 79.27 to 78.77. Average retrieval nDCG@10, however, declines from 31.62 to 27.29.

The smaller representation retains approximately one quarter of the dimensions and loses only 0.50 STS points, but sacrifices 4.33 retrieval points.

This is a task-dependent trade-off. Broad semantic overlap may be sufficient to estimate whether two texts are similar, while retrieval benefits more from detailed attributes that discriminate among competing documents.

Building Semantic Representations at Four Levels of Detail

Conventional dense embeddings encode text into numerical vectors whose individual coordinates are difficult to interpret. Question-answer embeddings instead associate each dimension with a natural-language question.

For a given document, each question receives a predicted probability of a yes answer. A threshold converts that probability into a binary feature.

Oda and colleagues extend the earlier CQG-MBQA approach by organizing these questions hierarchically.

Their construction process begins with approximately 6.8 million MEDI2 texts represented using UAE-large-v1 embeddings. Recursive k-means clustering creates progressively narrower semantic groups, beginning with 40 clusters and expanding through four levels.

GPT-4o-mini generates discriminative questions using positive examples and contrasting texts. Candidate questions are scored according to how often they distinguish their intended clusters from negative examples, then filtered for redundancy.

The resulting hierarchy contains:

Representation Questions Avg. STS Avg. clustering Avg. retrieval
Level 1 213 66.94 26.27 6.39
Level 2 861 74.28 32.61 16.71
Level 3 3,462 78.77 36.93 27.29
Level 4 13,914 79.27 38.04 31.62
Flat CQG-MBQA reproduction 16,419 79.28 38.16 30.93

Source: Section 4.1, Table 1; Appendix H, Tables 4–6. All performance results use a yes-answer threshold of 0.5. STS is Spearman correlation, clustering is V-Measure, and retrieval is nDCG@10, each displayed on a 0–100 scale.

These are the principal downstream benchmark comparisons, not measurements of human understanding.

Level 4 performs almost identically to the reproduced flat baseline on STS and clustering, and slightly better on average retrieval. Level 3 achieves a substantial dimensionality reduction with comparatively small losses on the first two task categories.

The authors identify level 3 as a favorable performance–interpretation trade-off. The retrieval results nevertheless show why that judgment cannot be transferred automatically to every application.

An enterprise search service concerned with ranking relevant passages may require deeper representations than a system primarily grouping documents by general topic.

Compressing Explanations Without Compressing Embeddings

The second contribution changes what an analyst needs to inspect rather than what the model uses to represent a document.

Several detailed questions may express related semantic attributes. A question about whether an article supports academic study and another about whether it explains a procedure might both be summarized under a broader question about learning or skill development.

The method uses mDeBERTa-base-v3 to estimate whether a detailed question’s declarative meaning entails a broader parent question. Each detailed question is assigned to the highest-scoring candidate within its parent cluster.

During interpretation, multiple positive dimensions pointing to the same parent can be presented as one broader explanatory question.

Crucially, the downstream embedding vector remains unchanged.

This separates two operations:

  • Representation reduction: selecting a shallower hierarchy decreases the dimensions used for downstream tasks and can change accuracy.
  • Explanation compression: aligning detailed questions with shared parents reduces the distinct questions counted during interpretation without changing predictive performance.

The paper’s cognitive-load proxy counts relevant positive questions: those shared between texts, shared across a cluster, or shared between a retrieval query and its top-ranked passage. After alignment, it counts distinct parent questions instead.

The main performance–cognitive-load comparison, reported at a threshold of 0.3, shows an improved measured Pareto frontier relative to the reproduced flat baseline across STS, clustering, and retrieval.

This finding demonstrates fewer counted explanatory questions at comparable benchmark performance. It does not demonstrate that analysts read faster or reach better decisions.

Broad Questions Can Be More Informative Than Detailed Ones

A further experiment addresses whether the apparent value of broader questions comes entirely from their smaller number.

The authors randomly subsample deeper-level questions to match the dimensionality of shallower representations. This is a controlled granularity comparison, rather than another test of the full hierarchy’s overall performance.

At a threshold of 0.3, matched-question-count STS performance differs relatively little across levels. At 0.7, shallower questions tend to outperform more detailed questions, especially when the available question count is limited.

The mechanism concerns answer sparsity.

A restrictive threshold marks an attribute present only when its predicted yes probability is high. Fine-grained questions may then receive mostly negative answers, leaving few positive features for comparing texts. Broader questions can still capture shared semantic information.

The matched-count STS analysis uses five random seeds. It supports a threshold-dependent granularity effect, not the conclusion that broad questions are universally superior.

The study also examines whether its question hierarchy reflects the intended semantic ordering. For many level combinations, more than 67% of aligned child questions have model-estimated entailment probabilities above 50% toward their parents, with less frequent reverse entailment. Lexical concreteness generally increases with depth.

These are semantic validation analyses. They support the plausibility of the hierarchy but do not guarantee that every parent correctly summarizes its children.

The Enterprise Design Decision Is About Two Separate Budgets

Cognaptus inference: An enterprise search or knowledge-base system could maintain detailed question-based embeddings while displaying broader questions first, allowing analysts to inspect specific attributes when a result needs closer examination.

That design separates the semantic information used for retrieval from the amount displayed during explanation.

For an enterprise search owner, the first decision is the minimum representation depth that maintains acceptable ranking quality on the organization’s documents and queries. The reported retrieval decline between levels 4 and 3 makes task-specific testing essential.

For a document-classification workflow, the relevant trade-off may differ. Broader questions could help analysts inspect shared classification themes, with detailed questions available for borderline cases. The paper does not evaluate this operational workflow.

A deployment evaluation should distinguish three outcomes:

  1. Task quality: Does the selected hierarchy preserve retrieval relevance or classification accuracy on representative documents?
  2. Explanation usefulness: Do analysts correctly understand why documents were connected, and can they identify misleading semantic alignments?
  3. Operational efficiency: Does the explanation interface improve decision time or error rates, and do any infrastructure savings exceed hierarchy-construction and inference costs?

These outcomes require different evidence. Smaller embeddings do not establish lower end-to-end operating costs, and shorter explanations do not establish better comprehension.

What the Benchmark Cannot Establish

The computational evidence is reasonably broad: 20 datasets, a reproduced interpretable baseline, threshold sensitivity experiments, matched-dimensionality comparisons, and semantic alignment analyses.

The human-centered evidence is narrower.

Cognitive load is approximated by question counts rather than measured reading effort or decision quality. Parent-child relationships depend on an NLI model and need not be logically valid. The implementation also depends on dense embeddings and LLM-generated supervision.

The paper provides no comprehensive human interpretability experiment, multilingual validation, or complete construction-cost and latency assessment. Its objective is a better performance–question-count trade-off, not superior absolute accuracy against leading black-box embeddings.

Fewer Questions Are a Design Choice, Not a Validation Result

The study establishes that interpretable embedding dimensions need not be treated as one flat collection. Hierarchical construction makes semantic granularity configurable, and parent-question alignment offers a separate way to reduce the explanations presented to users.

Its most operationally relevant finding is that these choices have different consequences: reducing dimensions can affect retrieval substantially, while summarizing explanations leaves prediction unchanged but has not yet demonstrated improved human understanding.

For enterprise document systems, the next decision is not simply how few questions can be displayed. It is how much semantic detail the task requires, whether broader explanations remain faithful, and whether people actually make better decisions with them.

Reference

Cognaptus: Automate the Present, Incubate the Future.


  1. Kohei Oda and Po-Min Chuang and Kiyoaki Shirai and Natthawut Kertkeidkachorn (2026). Interpretable Text Embeddings Based on Hierarchical Questions. arXiv:2611.06402. https://arxiv.org/abs/2611.06402 ↩︎