Prompt Injection

TL;DR for operators The current enterprise LLM mistake is to treat safety as a chat-interface problem. Put a filter in front, add a stern system prompt, run a cheerful demo, and hope the model has suddenly acquired a moral philosophy and a compliance department. Charming. Also insufficient. Two recent papers make a stronger and more operationally useful point. The first evaluates open-source LLMs under prompt-injection and jailbreak attacks, then compares lightweight inference-time defenses such as input filtering, system-prompt hardening, vector detection, voting, and self-examination.1 The second studies whether fine-tuned language models memorize sensitive personally identifiable information that appears only in inputs, not in the desired outputs, and benchmarks mitigation methods such as differential privacy, UnDial, regularization, and DPO.2 ...

TL;DR for operators The paper’s useful message is not “LLM agents are unsafe,” which is too vague to help anyone do anything before lunch. The useful message is narrower and more operational: agents become vulnerable when untrusted content from SaaS integrations is read into the agent context and then treated as authority for a later action. ...

TL;DR for operators AI agents are not only vulnerable because someone can hide a bad instruction in an email, document, web page, Slack message, or tool output. They are vulnerable because attackers can now automate the search for bad instructions that work. That changes the security problem. A one-off prompt injection is annoying. An automated attack loop is strategic. It generates candidate injections, observes the agent’s response, scores partial progress, keeps the promising branches, and tries again. Very entrepreneurial, in the worst possible way. ...

TL;DR for operators The paper introduces GAS-Leak-LLM, a black-box method that uses a genetic algorithm to evolve adversarial suffixes: small text sequences appended to harmful prompts to increase the chance that a model produces unsafe content.1 The important part is not that another jailbreak exists. We have enough of those. The important part is that jailbreak discovery is framed as a repeatable optimization loop using only model queries. ...

TL;DR for operators Most agent guardrails behave like stop signs. They inspect a proposed action, decide whether it looks safe, and then allow or block execution. This is neat, legible, and often operationally clumsy. Real agent failures are not always cleanly harmful from the first word. A useful business request can be contaminated by a prompt injection, a malicious tool response, or an unsafe intermediate plan. Blocking the whole task may reduce risk, but it also throws away the legitimate work. Excellent safety theatre, less excellent operations. ...

Security teams like to search for suspicious strings. That habit is understandable. Strings are visible. They can be logged, filtered, matched, scored, and proudly displayed in dashboards. A bad suffix at the end of a prompt looks like a bad suffix at the end of a prompt. Convenient. Almost too convenient. The problem is that prompts are not flat text boxes. They are transformed into token sequences, wrapped in chat templates, and passed through attention layers that do not treat every position equally. Some positions receive more influence over the model’s next-token behavior than others. Put adversarial tokens there, and the same amount of “badness” can travel farther. ...

Customer support bots are not supposed to have enemies. They sit politely inside enterprise websites, read policy documents, retrieve relevant snippets, and answer questions with the soft confidence of a well-trained assistant. The selling point is simple: Retrieval-Augmented Generation, or RAG, should make large language models less likely to hallucinate because the answer is grounded in external evidence. ...

A policy says one thing. A customer says another. A retrieved document says something newly alarming. A compliance rule says stop. A business workflow says continue. This is where large language models become interesting, and by “interesting” I mean expensive. Most companies still talk about LLM alignment as if it were a calibration problem. Tune the model. Add a system prompt. Insert a safety policy. Wrap it with retrieval. Then expect the assistant to behave consistently across messy real-world tasks. The paper Are Dilemmas and Conflicts in LLM Alignment Solvable? A View from Priority Graph argues that this expectation is too neat for the problem being solved.1 ...

Email is a boring place to start an AI security article. That is exactly why it is useful. A modern enterprise agent is not merely answering questions about email. It can search messages, summarize attachments, update calendars, create rules, contact colleagues, write to Slack, edit files, and remember what it learned for next time. In demo videos, this looks like productivity. In security reviews, it looks like a small software system that accepts natural language as both instruction and evidence. Wonderful. We have reinvented workflow automation, except now the workflow engine reads every suspicious paragraph with a helpful attitude. ...

Email is a boring interface. That is exactly why it is dangerous. A user asks an AI agent to summarize a message, update a record, book a trip, or search a workspace. The agent reads some external content, decides which tool to call, fills in the parameters, and continues the user’s task. Somewhere inside that external content sits a hidden instruction saying, in effect: “Before doing the user’s task, do mine.” ...