Rank the Work, Not the Model: Meta-Benchmarks for Bank LLM Screening

TL;DR for operators

A bank must choose a model for a particular workflow, but it cannot run a full internal evaluation whenever a new model appears. General leaderboards offer a useful starting point—not a reliable answer to which model best fits the banking work that matters.

The reported relationship between global and banking-domain rank varies substantially. Spearman correlation is 0.62 for Customer Management and 0.71 for IT Management, but 0.95 for Market Operations and 0.97 for Regulations and Compliance. The strongest model overall may therefore be less compelling for a specific domain, while an apparently precise domain score may rest on limited or indirect evidence.

Blair Hudson’s framework reduces the initial search cost by reusing results from many existing public benchmarks rather than creating and rerunning one new test suite.¹ Because those benchmarks use incompatible scoring scales, the method converts them into repeated model-versus-model outcomes and allows more current, discriminating, and widely reported benchmarks to influence the ranking more strongly. It then translates technical test performance into recognizable work activities and banking operations. This is the paper’s meta-benchmarking layer.

For operators, the practical sequence is:

1. Define the workflow and the banking activities it requires.
2. Use the relevant domain profile to narrow the candidate set.
3. Inspect how much evidence supports each score and where that evidence came from.
4. Run targeted internal evaluations on the shortlist.
5. Complete separate privacy, security, legal, compliance, risk, and governance review.

Used this way, the framework can inform procurement, evaluation budgets, provider-concentration analysis, and fallback planning. It cannot establish that a model is safe, compliant, reliable, or ready for production.

Global rank and banking rank answer different questions

A bank evaluating models for a customer-support workflow faces an expensive search problem. New systems arrive faster than most institutions can design, execute, review, and maintain full internal test suites. A general leaderboard offers a tempting shortcut: start with the highest-ranked models and assume the ordering will mostly carry over.

The paper’s domain comparisons show why that assumption needs qualification. Global composite rank correlates only moderately with Customer Management rank at 0.62 and IT Management rank at 0.71. The relationship is much closer for Regulations and Compliance at 0.97 and Market Operations at 0.95.

These figures do not mean that broad leaderboards are defective. They mean that their aggregation target differs from the bank’s decision. A global score blends coding, reasoning, knowledge, language, document processing, and other capabilities. A business workflow requires a particular combination of activities, and the combination varies by domain.

The procurement implication is narrow but consequential. The highest overall model may remain a strong candidate, but it should not receive an automatic presumption of domain leadership. Where the global-to-domain correlation is lower, screening directly against the relevant capability profile becomes more valuable.

The framework reuses public evidence rather than rerunning every test

The paper constructs its June 2026 snapshot from publicly reported benchmark results aggregated through the LLM Stats API. It covers 288 models from 25 organizations, 452 benchmark identifiers, 41 O*NET Generalized Work Activities, and 38 BIAN banking business domains.

This is meta-benchmarking: the method organizes evidence generated elsewhere rather than creating and administering a new universal examination. That distinction matters operationally. A bank can use the framework to narrow the field before committing resources to proprietary evaluation, but it inherits the reporting practices, omissions, contamination risks, and measurement choices of the public benchmark ecosystem.

The first technical problem is that public benchmarks do not share a common unit. A percentage on one task cannot be averaged meaningfully with an exact-match score, pass rate, preference result, or task-specific metric from another.

The paper avoids direct averaging by converting each benchmark into model-versus-model outcomes. When two models both report results, the higher score becomes a win, the lower score a loss, and an equal score a draw. Missing results generate no comparison; they are not treated as failures.

That ordinal conversion discards the size of the difference between two benchmark scores. It gains comparability across heterogeneous scales, but a narrow win and a large win have the same immediate outcome.

Benchmark influence depends on current evidential value

Not every benchmark result deserves equal influence. Some tests are saturated, some are reported for only a small subset of models, and some persist in datasets after providers have largely stopped using them.

The paper assigns each benchmark a composite weight:

The three factors represent:

• Discrimination: whether the benchmark still separates recent leading models.
• Coverage: how much of the recent model population reports a result.
• Recency: whether the scored models are themselves recent releases.

Because the factors are multiplied, weakness on one dimension can sharply reduce a benchmark’s influence. A widely reported benchmark contributes little if nearly every leading model has saturated it. A highly discriminating result also receives limited weight if only a few models have attempted it.

The default calculation examines models within a 12-month rolling window. Discrimination is estimated from the top ten recent scorers, and benchmarks with fewer than three recent scored models receive zero discrimination weight. These are configurable methodological choices, not fixed properties of the benchmarks.

The operational benefit is lifecycle management. Benchmark influence can decline automatically as a test becomes less informative, without requiring an administrator to declare it obsolete. The corresponding limitation is that rankings depend partly on window length and public reporting behavior. A shorter window reacts faster but has less data; a longer one may preserve evidence from models that no longer represent the competitive frontier.

Pairwise Elo creates relative work-activity scores

The benchmark weight scales an Elo update rather than a raw-score average:

Comparisons conducted on stronger current evidence move the ratings more than comparisons conducted on weak evidence. All models begin at 1500, the base K-factor is 32, and the tournament runs for 20 passes.

The converged rating for each work activity is then normalized:

A score of 50 is therefore the participating-model average for that activity in the current snapshot. Ten points represent one population standard deviation. It is not a permanent capability threshold, and the value can change when models, benchmarks, or reported results change.

The paper’s stability analyses are best read as robustness checks on this aggregation procedure:

The results indicate that the framework is not excessively sensitive to the tested weighting choices. They do not show that its rankings predict operational performance.

Standardized mappings connect technical tests to banking decisions

Benchmark results become relevant to banking only after two translation steps.

First, the paper maps each benchmark to one or more of 41 O*NET Generalized Work Activities according to the cognitive activity it primarily measures. Second, those work activities are mapped to 38 domains from the BIAN Service Landscape.

A banking-domain score is a weighted average of the relevant work-activity scores:

The default gives each mapped work activity equal weight. Institutions could change those weights where a workflow depends more heavily on one activity than another.

This mapping layer makes the screening process traceable. A reviewer can move from a business-domain rank to the contributing work activities and then to the public benchmarks underneath them. That is more auditable than accepting a provider’s general capability claim or an unexplained composite score.

It also introduces judgment. The mappings are manually maintained candidate mappings, not validated ground truth. Equal weighting ignores correlations among activities and can dilute a serious weakness in one critical component.

Coverage gaps determine how much confidence a score deserves

The taxonomy contains 41 work activities, but only 24 have any public benchmark coverage. The 17 uncovered activities are concentrated in physical, managerial, and interpersonal work.

Evidence is also uneven among the covered domains. Coding and general-reasoning activities have relatively dense support. Specialist financial question answering, AML and KYC work, compliance reasoning, and jurisdiction-specific regulatory obligations have fewer direct benchmarks and rely more heavily on proxies.

This creates two distinct cases:

• A domain score supported by numerous current, discriminating benchmarks can provide a defensible basis for preliminary comparison.
• A domain score assembled from sparse or indirect evidence should mainly indicate where internal evaluation is still missing.

The framework makes absence visible by declining to score domains whose contributing activities lack public evidence. That is preferable to filling gaps through unsupported imputation. It does not eliminate the cold-start problem for newly released or sparsely evaluated models.

The business case is a smaller internal test set

What the paper directly shows: domain-specific rankings can differ from global rankings; the tested weighting variants produce broadly stable orderings; and public evidence density varies substantially across banking activities.

Cognaptus interpretation: a financial institution could use these profiles as a triage mechanism. Begin with the proposed workflow, identify its relevant BIAN domains and constituent activities, and select a limited group of models for proprietary testing. This concentrates evaluation effort on the models most plausibly suited to the work rather than testing every prominent release.

The same profiles could support two secondary decisions. First, they may reveal provider concentration when all models above an internal screening threshold come from one vendor. Second, they may identify cross-provider fallback candidates and quantify the expected capability difference during outages, deprecations, rate-limit incidents, or provider-policy changes.

Those uses require institution-defined thresholds and current data. The paper does not validate a universal Elo gap that makes one fallback acceptable or another unacceptable.

A domain score is not production evidence

The most consequential misconception would be to treat a high banking-domain score as certification.

The framework does not evaluate institution-specific data, permissions, tool access, workflow controls, latency requirements, failure recovery, privacy exposure, cybersecurity posture, legal obligations, or jurisdictional compliance. It also combines provider-reported and independently reproduced benchmark scores without independently verifying each result.

Composite scores create an additional risk. A strong average can coexist with unacceptable performance on one constituent activity. For AML investigation, regulatory reporting, customer remediation, or other high-consequence workflows, a critical failure cannot be neutralized by stronger performance elsewhere in the domain.

The appropriate boundary is therefore procedural: use the ranking to decide which models deserve closer examination, not which model deserves deployment.

Screening becomes stronger when uncertainty remains visible

This paper addresses a real constraint in bank model selection: the institution needs domain-relevant evidence but cannot rebuild a comprehensive evaluation system for every model release.

Its contribution is a reproducible method for turning fragmented public results into relative, traceable capability profiles. Dynamic benchmark weighting limits the influence of stale or weak evidence. Pairwise Elo avoids averaging incompatible raw scales. Standardized work and banking taxonomies connect the resulting scores to recognizable operational decisions.

The framework is strongest when it reduces search cost and exposes where the evidence is thin. It becomes unreliable when a relative score is treated as a complete statement about deployment readiness.

For operators, the practical standard should remain clear: rank the models against the work, inspect the evidence beneath the rank, and reserve approval for institution-specific validation.

Cognaptus: Automate the Present, Incubate the Future.