Benchmarks Without Borders: Inside the Moduli Space of AI Psychometrics

Procurement Has a Benchmark Problem

Procurement teams love benchmark tables. They are clean, sortable, and emotionally comforting. Vendor A beats Vendor B by 3.7 points on a reasoning suite; Vendor C wins on code generation; Vendor D claims better tool use under “realistic agent workflows,” a phrase that usually means someone added a browser, a calculator, and optimism.

The trouble is not that benchmarks are useless. The trouble is that fixed benchmark scores are too easy to overinterpret. An autonomous agent is not just a model answering questions. It may plan, call tools, remember state, revise its own work, coordinate with other agents, and consume resources while doing so. A single leaderboard score cannot tell us whether we are seeing transferable capability, interface scaffolding, benchmark familiarity, or a very expensive magic trick with a nice dashboard.

Przemyslaw Chojecki’s paper, “Psychometric Tests for AI Agents and Their Moduli Space,” tries to formalize a better answer.¹ Its core move is not to introduce yet another benchmark. Mercifully. Instead, it treats benchmark batteries themselves as mathematical objects, identifies which changes should count as irrelevant relabelings, normalizes scores into comparable canonical forms, and then studies the resulting “moduli space” of possible evaluations.

That sounds abstract because it is. But the business interpretation is surprisingly practical: enterprises should stop asking whether one benchmark suite is sacred and start asking whether their evaluation panel covers the right space of task variation, scoring thresholds, dependency patterns, and resource constraints.

In plainer language: you do not need perfect tests. You need structurally diverse tests, calibrated scoring, and an honest error bar. Radical stuff, apparently.

The Paper’s Main Move: Turn Benchmarks Into Objects, Not Trophies

The paper begins from a simple observation: a psychometric test battery is not merely a list of tasks. For AI agents, a battery includes tasks, task families, scoring maps, thresholds, sampling laws, seeds, drift conditions, and resource measurements such as time, tokens, or cost.

Formally, the paper defines a battery as an octuple:

where $T$ is the finite task set, $F$ partitions tasks into families, $S$ contains task-specific scoring maps, $Q^\ast$ defines thresholds, $\mu$ is the sampling law over tasks, seeds, and drifts, $D$ and $\Pi$ represent drift and seed spaces, and $R$ records resources.

The point of this machinery is not notation for notation’s sake. It allows the paper to ask a question that ordinary benchmark discussions rarely ask cleanly:

When are two benchmark batteries meaningfully the same evaluation, even if their labels, score scales, seeds, or resource units differ?

That question matters because many surface changes should not change the evaluation meaning. Reordering tasks within a family should not affect capability. Relabeling random seeds should not change the score. Transforming a raw score through a strictly increasing function should not make a weaker agent stronger. Measuring cost in cents rather than dollars should not suddenly create artificial intelligence. Accountants would notice.

So the paper defines morphisms and symmetries of batteries. These include within-family task permutations, measure-preserving relabelings of seeds and drifts, strictly increasing score reparameterizations, and positive resource rescalings. Once these irrelevant transformations are factored out, what remains is not a single benchmark. It is an equivalence class of benchmarks.

That equivalence class is the beginning of the moduli-space view.

The Moduli Space: What Actually Varies After You Remove Cosmetic Differences

The paper’s most useful conceptual move is to separate cosmetic variation from structural variation.

After quotienting by evaluation-preserving symmetries, a battery can still vary in important ways. The paper identifies three major continuous coordinates within a fixed discrete “skeleton” of task families and threshold structure:

This is where the moduli-space language earns its keep. A benchmark battery is not only “hard” or “easy.” It has a family structure, thresholds, task-dependence patterns, and resource geometry. Two agents can have the same average score while behaving very differently across that space. One may be robust across families; another may be a specialist wearing a generalist costume.

The paper also distinguishes discrete and continuous moduli. The discrete part includes the task-family skeleton: which task families exist, how anchors are chosen, and how thresholds are ordered. Within a fixed skeleton, the continuous part is described by thresholds, copulas, and resource rays. Moving between skeletons corresponds to deeper structural changes: families merge or split, thresholds collide, anchors appear or disappear.

For business users, this distinction is useful because not every benchmark update should be treated the same. Recalibrating thresholds is not the same as adding a new task family. Changing cost accounting is not the same as changing the task-dependence structure. A mature evaluation system should track these separately instead of throwing everything into a release note called “benchmark v2.” Very enterprise. Very doomed, unless formalized.

PIT Normalization: The Small Technical Trick That Makes Comparison Possible

A central problem in cross-task evaluation is that raw scores are not naturally comparable. A 0.82 on one task and a 0.82 on another may not mean the same thing. One score may come from a nearly deterministic task; another may come from a noisy evaluation with many borderline cases.

The paper uses randomized probability integral transform, or PIT normalization, to solve this. Given a scalar task score and its conditional distribution, the score is transformed into a normalized value $u(t)$ in $[0,1]$. Under the relevant conditions, this normalized score becomes uniform conditional on seed and drift, and it is invariant to strictly increasing score transformations.

That matters because once each task score is normalized, the evaluation can focus on the dependence structure among tasks. This is where copulas enter. A copula captures how normalized task performances move together. If an agent’s success on one task family predicts success on another, that dependence matters. If performance collapses under mild drift or only holds for one narrow family, that also matters.

The business translation is straightforward: score normalization is not cosmetic. It is the difference between comparing calibrated capability patterns and comparing arbitrary measurement scales. Without normalization, a procurement dashboard can become a spreadsheet-shaped hallucination.

The Central Guarantee: Dense Panels Can Determine Regular Evaluation Functions

The paper’s main theorem is a determinacy result. It proves that regular families of AAI functionals are determined by their values on a countable dense subset of simple battery-law pairs.

That sentence is mathematically compact and cognitively rude, so let’s unpack it.

An AAI functional is a scoring rule that assigns an autonomy or general-intelligence score to an agent representation on a battery. The paper requires such functionals to satisfy several axioms:

The determinacy result then says: if the scoring functional is regular, meaning Lipschitz-continuous with respect to the paper’s metric over battery-law pairs, and if two such functionals agree on a dense set of simple cases, they agree everywhere.

Operationally, the paper reads this as a certification principle. If a finite panel is dense enough in the canonicalized battery space, strong performance across that panel can bound worst-case performance across the broader continuum of related tests. In the introduction, the paper states the practical version: if an agent’s AAI score exceeds a threshold on every panel instance with margin $m$, then its worst-case AAI over the whole moduli is at least approximately the threshold minus a controllable Lipschitz error term, written there as $\text{threshold} - 2L\delta$.

This is the sentence people will be tempted to abuse. Please resist. The paper does not show that adding more benchmarks magically certifies AGI. It shows that if the scoring functional is regular, if the panel is dense in the right canonical space, if the distance metric is meaningful, and if finite-sample uncertainty is handled properly, then finite testing can support global claims with bounded error.

That is still valuable. It is just not a vending machine for intelligence certificates.

AAI Becomes a Functional, Not Just a Composite Score

The paper connects its framework to the AAI score developed in the author’s earlier work. The AAI-Index is treated as a geometric aggregator over axes such as Autonomy, Generality, Planning, Memory, Tool Economy, Self-Revision, Sociality, Embodiment, World-Model quality, and Economics.

The paper defines a geometric AAI functional of the form:

where each $\pi_x$ is an axis score, $w_x$ is its weight, and $W$ is the total weight.

The paper’s Proposition 7.1 shows that this geometric functional satisfies the proposed axioms and reduces to the AAI-Index for deterministic axis values. Then Theorem 7.3 and Corollary 7.4 show regularity under design assumptions: each axis map must be Lipschitz in the canonical variables and clipped below by some $\epsilon > 0$.

The clipping condition is not a footnote-level detail. The logarithm behaves badly near zero. If an axis score can collapse to zero, the geometric mean becomes unstable. Clipping keeps the score bounded away from zero, allowing the Lipschitz argument to go through.

For evaluation teams, the lesson is not “use this exact AAI score tomorrow.” The lesson is more general: composite agent scores should disclose their regularity assumptions. If a score is too discontinuous, too threshold-hacky, or too sensitive to arbitrary formatting changes, it will not support broad claims about general capability.

A benchmark score without regularity is not a measurement instrument. It is a mood.

Cognitive Core: Separating Capability From Scaffolding

The paper’s second major practical contribution is the idea of a cognitive core. In the framework, a cognitive core is the minimal factor needed to make threshold-aligned success indicators measurable. Less formally, it is the smallest structure that still explains whether the agent crosses success thresholds across tasks.

The paper defines a core score, $AAI_{core}$, by projecting the full agent representation onto this core factor and evaluating the induced law through a lifting map. It also proves that, under the relevant minimality condition, the cognitive core is unique up to isomorphism.

This matters because modern agents are full of non-core aids: tools, memory stores, prompt wrappers, retrieval systems, workflow templates, guardrails, and orchestration layers. Some of these are useful engineering. Some make the system look smarter than its underlying transferable capability. The paper does not tell us how to perfectly separate those in every deployed system, but it gives a formal language for describing the separation.

The continuation theory in Section 8 extends this idea. A full score can be seen as a continuation of a core score. The minimal continuation depends only on the core. Other continuations add non-core invariants: interface advantages, scaffolding effects, operational quirks, or other fiber-level features that vanish on canonical lifts.

That gives businesses a clean diagnostic distinction:

This is not anti-tool-use. Tool use is part of real agent performance. The point is not to shame scaffolding. The point is to know when you are buying cognition, when you are buying infrastructure, and when you are buying a very expensive autocomplete wearing a hard hat.

What Counts as Evidence in This Paper

This is not an empirical benchmark paper. There are no leaderboards, no model comparisons, no ablation table showing that System X beats System Y by 4.2 percentage points. The evidence is mathematical: definitions, propositions, theorems, proof sketches, and operational procedures.

That changes how the paper should be read.

This distinction matters because the paper’s value is architectural, not experimental. It gives evaluators a language for building more defensible evaluation systems. It does not deliver a ready-made compliance regime.

The Business Use Case: Evaluation as a Governed Measurement System

The business relevance is strongest for organizations that need to evaluate autonomous agents repeatedly, not once for a press release.

Think of a company deploying agents for customer support, compliance review, financial analysis, software maintenance, or logistics planning. The relevant question is not “Which agent scores highest on Benchmark X?” The better question is:

Does this agent maintain calibrated, resource-aware performance across a structured space of task variants, thresholds, dependencies, seeds, and drifts?

The paper suggests an evaluation operating model with five components.

1. Define Task Families Before Selecting Tasks

A serious evaluation starts with task families, not random benchmark items. For a support agent, families might include factual retrieval, policy interpretation, escalation judgment, multilingual handling, adversarial user behavior, and tool-mediated resolution. For a coding agent, they might include bug localization, patch generation, test writing, dependency reasoning, and repository navigation.

The paper’s framework rewards this because generality is measured across families, with dispersion penalties discouraging narrow excellence. A model that crushes easy retrieval but fails escalation judgment should not receive a free “general agent” badge. Badges are cheap. Incidents are not.

2. Normalize Scores and Track Thresholds

Raw scores should be transformed into comparable canonical representations. Thresholds should be explicit and calibrated. The paper’s threshold sensitivity axiom captures an important business reality: moving from below acceptable to acceptable is more valuable than improving already-safe performance by a decorative margin.

For governance, this means evaluation reports should not only publish averages. They should show where agents sit relative to operational thresholds and how sensitive decisions are near those thresholds.

3. Model Dependence Across Tasks, Not Just Average Performance

The copula component is especially relevant for risk teams. If task successes are highly correlated, the evaluation may reveal a single underlying capability. If they are weakly correlated, the same average score may hide fragmented competence.

A business evaluation should ask whether success in one family predicts success elsewhere, whether failures cluster under certain drifts, and whether new benchmark variants change the dependence structure. This is where the moduli-space view becomes more than mathematical decoration.

4. Include Resource Accounting

The paper repeatedly includes resource coordinates and cost terms. That is correct. An agent that solves tasks by using extreme token budgets, excessive tool calls, or slow multi-step search may be capable but commercially unattractive.

For enterprise deployment, capability per unit cost matters. The economics axis in the AAI discussion captures this by relating quality-adjusted throughput to monetary cost. A procurement team should treat “better but 9x slower” as a trade-off, not a triumph.

5. Recalibrate Under Drift

The operational section proposes drift and recalibration logic: define robustness regions around copula and threshold variation, monitor anchor agents, and trigger recalibration when rank concordance falls below a pre-registered threshold.

This is one of the more business-ready ideas in the paper. Evaluation systems decay. User behavior changes, task distributions move, models are updated, wrappers are modified, and tool APIs quietly become new failure surfaces. A static benchmark panel is a photograph. Agent governance needs a weather station.

Where the Guarantee Bends

The paper’s framework is powerful, but its practical use depends on assumptions that should not be politely hidden under the rug.

First, the regularity condition is doing heavy lifting. The determinacy result needs Lipschitz-style stability. If the scoring functional has sharp discontinuities, arbitrary thresholds, or hidden nonlinear jumps, dense-panel reasoning becomes weaker. This is not a minor implementation preference. It is the hinge of the guarantee.

Second, density is not free. A panel must be dense in the right canonical space, not merely large. Adding 500 tasks from the same narrow family does not cover the moduli space. It covers one neighborhood very enthusiastically.

Third, estimating copulas and score distributions can be difficult. PIT normalization assumes access to useful conditional score distributions. In real evaluations, those distributions may be noisy, under-sampled, or affected by changing prompt formats, tools, or evaluator models.

Fourth, the cognitive core is formally elegant but operationally hard. The paper defines uniqueness up to isomorphism under minimality conditions, but deployed agents are messy compositions of model weights, prompts, retrieval, memory, tools, APIs, and orchestration logic. Identifying what belongs to the “core” will require measurement design, not just notation.

Fifth, the paper is not an empirical validation of the AAI axes. It shows that AAI-style geometric aggregation can fit into the axiomatic framework under assumptions. It does not prove that those axes are complete, optimally weighted, or accepted by regulators, customers, or reality. Reality remains annoyingly unstandardized.

These limitations do not weaken the paper’s conceptual value. They locate it. The contribution is a formal blueprint for evaluation design, not an off-the-shelf certification stamp.

The Better Procurement Question

The old procurement question was:

Which model has the highest benchmark score?

The better question is:

What structured region of agent behavior has been tested, under which thresholds, dependencies, drifts, and resource constraints, with what uncertainty?

That is a less convenient question. It does not fit neatly into a one-row vendor comparison. It also has the advantage of being closer to how deployed AI systems fail.

Chojecki’s paper gives a mathematical vocabulary for that shift. Batteries become structured objects. Equivalent surface transformations are factored out. Scores are canonicalized. Benchmark families live in a moduli space. Regular scoring functionals can be determined from dense panels. Cognitive cores separate transferable capability from interface scaffolding. Finite-sample and drift results provide the beginnings of operational assurance.

The deepest business lesson is not that moduli spaces will appear in every board deck. They will not, unless the consultants get very ambitious. The lesson is that AI evaluation should become less like leaderboard watching and more like governed measurement engineering.

A serious enterprise agent should not merely win a benchmark. It should demonstrate stable, calibrated, resource-aware performance across a deliberately sampled structure of tests.

Benchmarks are still useful. They just need borders, coordinates, and passports.

Cognaptus: Automate the Present, Incubate the Future.