Adversarial Machine Learning

TL;DR for operators What exactly should a deployed AI system reveal to users, vendors, insiders, or connected applications? Hiding model architecture and weights does not make every interface equally opaque. Mahbub and colleagues separate access into six operational categories—None, Metadata, Decision-Only, Score/Rank, Embedding, and White-Box—because each exposes a different signal to an adversary.1 A binary decision permits probing, while numerical confidence or similarity values provide directional feedback; internal feature representations expose still richer information. The paper’s synthesis suggests that richer signals generally reduce attacker uncertainty and query burden while enabling additional risks such as model extraction and biometric template inversion. ...

A factory can know exactly which machine submitted a model update and still train on a lie. The device may possess a valid cryptographic identity. Its software may have booted from an approved configuration. Its network connection may be encrypted. None of that proves that the update it sends is harmless—or that the resulting intrusion-detection model will recognize an attack crafted specifically to deceive it. ...