Black Box Is Too Blunt: What AI Interfaces Reveal to Attackers

TL;DR for operators

What exactly should a deployed AI system reveal to users, vendors, insiders, or connected applications?

Hiding model architecture and weights does not make every interface equally opaque. Mahbub and colleagues separate access into six operational categories—None, Metadata, Decision-Only, Score/Rank, Embedding, and White-Box—because each exposes a different signal to an adversary.¹ A binary decision permits probing, while numerical confidence or similarity values provide directional feedback; internal feature representations expose still richer information. The paper’s synthesis suggests that richer signals generally reduce attacker uncertainty and query burden while enabling additional risks such as model extraction and biometric template inversion.

For procurement and deployment, ask what the richest signal an external attacker, insider, contractor, compromised device, or reverse engineer could plausibly obtain—not simply whether the system is called “black-box.” Use that access level to set API-output restrictions, adversarial tests, monitoring, access controls, contractual requirements, and incident-response expectations, and reassess it as effective access changes over the system lifecycle.

The boundary is important: SMART is a conceptual framework built from heterogeneous prior studies. It does not estimate the probability that a deployment will be attacked or provide standardized risk multipliers for moving between access levels. :contentReference[oaicite:0]{index=0}

A black-box API can still disclose very different attack signals

Suppose a procurement team is choosing how much information an AI service should return.

One design emits only “accept” or “reject.” Another returns a confidence or similarity score. A third exposes the internal feature vector used for matching. All three may hide the underlying model, yet they do not present the same adversarial surface.

This is the problem behind the paper’s Signal-based Model Access Risk Taxonomy, or SMART. Instead of classifying systems primarily as white-box, gray-box, or black-box, SMART asks what information the adversary can actually observe.

The six categories are:

The classification is useful because the paper treats information richness as the underlying mechanism. More informative signals reveal progressively more about model behavior or internal geometry.

Richer feedback changes attack efficiency, not just attack possibility

At the lowest access level, an adversary may still succeed without interacting with the target model. The review points to transferable adversarial examples and physical attacks as evidence that proprietary or inaccessible models are not automatically insulated from evasion.

Metadata is the next step. Technical specifications, documentation, latency, error behavior, or other artifacts can narrow the space of likely models and preprocessing pipelines. But the paper is careful here: direct evidence that metadata exposure alone improves evasion success remains limited, and some of the argument is extrapolated from model-stealing research.

Decision-only access supplies a more direct signal. Repeated binary outputs can be used to infer where a classification or verification boundary lies. The interface leaks less information than a numerical score, but it does not reduce the attacker to guessing.

Continuous scores change the optimization problem more substantially. If small input changes produce numerical changes in confidence or similarity, those responses provide directional information about the scoring function. The reviewed literature includes examples where attacks requiring more than 20,000 decision-only queries are described as succeeding with roughly 200–500 score-based queries.

Those figures illustrate mechanism rather than a universal multiplier. They come from different prior studies, not from a standardized experiment. The paper itself also contains inconsistent wording about the magnitude: one section describes the reduction as typically two to three orders of magnitude, while another summarizes it as often one to two.

The defensible conclusion is narrower: score feedback can materially reduce the online search burden compared with hard labels.

Scores and embeddings also widen the type of risk

The progression is not only about making evasion faster.

Score-based interfaces can support model extraction. If an attacker can query a service and learn enough about its responses to construct a surrogate, much of the subsequent experimentation can move offline. The resulting risks can extend from adversarial-example development to privacy attacks and intellectual-property loss.

Embedding access goes further by exposing an internal representation directly. In face recognition, these representations are commonly high-dimensional vectors used for matching, comparison, or clustering. The review argues that they can become direct optimization targets for impersonation attacks and can retain enough biometric information to support template inversion.

That makes an embedding a security asset as well as a data-processing artifact. A stored face representation may be less visually interpretable than an image, but the reviewed inversion evidence means it cannot be assumed to contain no recoverable identity information.

The paper also makes an informal scaling observation. Pairwise score comparisons across $N$ items can require work growing approximately as $N^2$, whereas obtaining one embedding for each item grows approximately as $N$. This is not presented as a formal complexity result, but it helps explain why embedding access can provide a more direct interface for large-scale comparison or attack workflows.

Procurement determines plausible access, but access can change later

SMART becomes operational when the paper maps model access to procurement choices.

Its procurement table assigns open-source base models a plausible White-Box level for both external and insider actors. Commercial cloud APIs are mapped up to Score/Rank externally, while insiders may plausibly reach Metadata through Embedding. Commercial on-premise systems may expose little externally but richer internal access. Contracted development, in-house systems, and edge deployments can also create effective White-Box conditions for insiders.

These are maximum plausible access levels, not predictions of what every deployment exposes.

Cognaptus’ business inference is to use the richest credible access state—not the intended customer interface—as the security planning baseline. For a buyer, that means examining at least three actor classes: ordinary external users, insiders or development partners, and actors who gain access after compromise or reverse engineering.

That framing changes several decisions. An API team may decide whether numerical scores are genuinely required. A biometric deployment may apply stronger protections to stored embeddings. Procurement specifications can require adversarial testing, access controls, query telemetry, rate limits, model-update commitments, breach notification, and incident-response responsibilities appropriate to the plausible access level.

The source also argues that access is dynamic. Outsourcing, employee turnover, device compromise, leaked models, or reverse engineering can move a system toward richer adversary access after launch. Threat modeling therefore needs lifecycle reassessment rather than a one-time classification at deployment.

Use SMART as a risk framework, not a risk calculator

The paper gives operators a more precise vocabulary for model-access risk, but its evidentiary form sets clear limits.

It is a narrative review of 39 cited works rather than a systematic review. There is no reported study-selection protocol, quality assessment, or common experimental benchmark. Attack-success rates and query counts come from heterogeneous models, datasets, goals, and threat assumptions.

Face recognition supplies most of the operational examples. The authors argue that the taxonomy generalizes to other AI systems, but that broader applicability is conceptual rather than empirically demonstrated.

SMART therefore cannot tell a procurement team that exposing scores raises risk by a specific percentage, or that an embedding interface will produce a particular attack success rate. It can do something more modest and still operationally useful: prevent materially different interfaces from being collapsed into one “black-box” category.

For security teams, that creates a clearer chain from architecture to action. Identify the richest signal a plausible actor can obtain, determine which attack mechanisms that signal enables, and then set controls and evaluation requirements against that condition. The paper’s contribution is not a numerical risk score. It is a sharper definition of what access actually means.

Cognaptus: Automate the Present, Incubate the Future.