Adversarial ML

Security teams like to search for suspicious strings. That habit is understandable. Strings are visible. They can be logged, filtered, matched, scored, and proudly displayed in dashboards. A bad suffix at the end of a prompt looks like a bad suffix at the end of a prompt. Convenient. Almost too convenient. The problem is that prompts are not flat text boxes. They are transformed into token sequences, wrapped in chat templates, and passed through attention layers that do not treat every position equally. Some positions receive more influence over the model’s next-token behavior than others. Put adversarial tokens there, and the same amount of “badness” can travel farther. ...

A fraud model does not only learn from transactions. It learns from sequence. Who interacted with whom. When. How often. After what previous event. Before which next event. In temporal graph systems, the order is not metadata. It is the thing being modelled. That is why LoReTTA is an uncomfortable paper.1 It does not argue that Temporal Graph Neural Networks can be broken only by a powerful adversary with model access, expensive surrogate training, and a theatrical pile of fake edges. It argues something more operationally annoying: a continuous-time graph can be poisoned by removing influential interactions and replacing them with plausible ones. The resulting history still looks enough like history. The model quietly learns the wrong temporal structure. Very civilised, as crimes go. ...