From Stage to Script: How AMADEUS Keeps AI Characters in Character

TL;DR for operators

Characters are easy when they stay on script. They become expensive when users ask the wrong question, which is, naturally, what users do.

The AMADEUS paper addresses a specific failure mode in retrieval-augmented role-playing agents: ordinary RAG can retrieve facts, but persona consistency often depends on inferred traits, values, habits, and narrative context rather than direct answers. A user asks, “Are you confident everything will work out?” The persona document may not contain that sentence. Naive RAG may grab a superficially similar chunk and improvise badly. AMADEUS instead tries to retrieve evidence from which a character’s attributes can be inferred, then feeds those attributes into generation.¹

The operational lesson is not “use more context.” It is sharper: for customer-facing AI characters, brand agents, tutors, NPCs, streamers, and expert personas, retrieval has to preserve identity. AMADEUS does this through three stages: Adaptive Context-aware Text Splitter (ACTS), Guided Selection (GS), and Attribute Extractor (AE). The paper also introduces CharacterRAG, a dataset with 15 fictional characters, 976K written characters, and 450 QA pairs designed for RAG-based role-playing evaluation.

The strongest evidence is on persona fidelity under out-of-knowledge questioning. In GPT-4.1 experiments, AMADEUS reaches 85.00% MBTI accuracy and 81.33% Big Five/SLOAN accuracy, ahead of naive RAG, corrective RAG, graph-based RAG, RAPTOR, and adaptive RAG variants where reported. Direct factual QA gains are more modest but still positive: on CharacterRAG QA, AMADEUS slightly improves over naive RAG across GPT-4.1, Gemma3-27B, and Qwen3-32B, with lower hallucination scores in the reported setting.

The boundary is important. These are fictional personas, reconstructed from Korean Namuwiki-derived material, evaluated partly through psychological-test proxies and LLM-based metrics. That does not make the work weak. It means the right enterprise interpretation is: AMADEUS is a useful design pattern for persona-aware retrieval, not a warranty that your AI brand ambassador will never start speaking like a bored legal intern.

The real failure is not forgetting facts, but breaking identity

Most RAG systems are built around a simple promise: retrieve relevant information, place it in context, and let the model answer. That works reasonably well when the task is factual lookup. A return policy is a return policy. A warranty clause is a warranty clause. A support agent either found the correct page or it did not.

Role-playing agents are less polite.

A character can be asked about something not explicitly present in its written persona. The correct response is not “I do not know” in the same sense as a database lookup. It may be an answer inferred from values, temperament, social habits, prior actions, and narrative context. A shy character, a reckless character, and an idealistic character can all face the same question and need different answers, even when none of their source documents contains a direct sentence matching the query.

This is where naive RAG becomes oddly dangerous. It does not merely fail to retrieve enough. It may retrieve the wrong kind of enough. The paper’s opening example compares naive RAG against AMADEUS for a Tanjiro Kamado question about approaching someone romantically. Naive RAG retrieves a chunk about Tanjiro trying to hide excitement over food and produces a response that bends that evidence into the answer. AMADEUS retrieves material about how Tanjiro quickly calls people by name and forms warm social connections, then generates a more character-consistent response.

The difference is not cosmetic. It changes the definition of relevance.

For role-playing, a retrieved chunk is useful not only when it contains the answer. It is useful when it supports an inference about how the persona would answer. That is the central mechanism in AMADEUS.

AMADEUS turns retrieval into a three-stage acting coach

AMADEUS is training-free. It does not fine-tune the model or update parameters for every character. That matters because real persona systems change: game lore expands, streamer identities evolve, brand voice guidelines get rewritten after someone in marketing discovers a new adjective. Fine-tuning every time is usually not the economical path to sanity.

Instead, AMADEUS modifies the retrieval pipeline.

This is a mechanism-first paper because the mechanism is the point. The benchmark scores are useful, but the deeper claim is that persona RAG needs a different retrieval objective. In normal RAG, the question is “Which chunk answers the query?” In AMADEUS, the question becomes “Which chunk helps infer the character’s stance toward the query?”

That distinction is small enough to fit in a method section and large enough to matter in production.

ACTS preserves the stage directions that ordinary chunking cuts away

The first AMADEUS component is the Adaptive Context-aware Text Splitter. ACTS is designed for persona documents that are long, uneven, and hierarchical. Characters do not come as clean paragraphs of equal length. They come with backstory, relationships, events, habits, internal values, and sections nested under sections. A fixed-length splitter treats that as a formatting inconvenience. ACTS treats it as signal.

The paper’s CharacterRAG personas preserve hierarchical sections such as character actions, story arcs, and subsections. ACTS recursively retrieves the hierarchy around each chunk and appends that contextual information. It also sets chunk and overlap sizes adaptively: it uses the maximum paragraph length in the persona and sets overlap to half that length. The authors argue this reduces information loss because adjacent narrative fragments often depend on each other.

This is not glamorous. It is plumbing. But in AI systems, plumbing is where many “emergent behaviour” miracles quietly drown.

The evidence for ACTS appears in the similarity-distribution analysis. Compared with RecursiveCharacterTextSplitter, MarkdownHeaderTextSplitter, SemanticChunker, and an adaptive splitter without hierarchical context, ACTS achieves the highest average similarity score and lower variance across the tested embedding settings. The likely purpose of this test is implementation validation: it checks whether the splitter is producing chunks that are semantically richer and more stable for retrieval.

That does not prove better role-playing on its own. It supports the first link in the chain: if chunks preserve more useful context, the later selection and attribute extraction stages have better material to work with.

Guided Selection asks whether a chunk can support an inference

The second component, Guided Selection, is where AMADEUS becomes more than chunking hygiene.

GS first sorts chunks by semantic similarity to the user query. So far, nothing radical. Then it asks an LLM to judge whether each chunk contains information from which the character’s Belief and Value or Psychological Traits can be inferred for the query. It fills a slot with selected chunks and stops once the slot is full; if nothing qualifies after the search limit, it falls back to top-similarity chunks.

This is a pragmatic compromise. Pure semantic similarity is too shallow for persona inference. Full reasoning over every chunk is too expensive. GS narrows the field, then asks a more persona-aware question.

The appendix makes this explicit. The GS prompt asks whether the context clearly supports inference of belief/value or psychological traits for the user query, and requires a True/False answer. The implementation uses GPT-4.1 for GS and AE, with a maximum of 30 search iterations and a slot size of 2. That means AMADEUS is not “free” in compute terms, despite being training-free. It shifts cost from model updates to inference-time filtering and extraction.

That trade-off is often sensible. Updating a customer-facing persona through retraining is expensive, slow, and operationally awkward. Running a guided retrieval pass may be easier to manage. But it is still a cost, and the paper’s production value lies partly in making that cost targeted.

Attribute Extraction gives the actor a character note before the line

The third component, Attribute Extractor, converts selected chunks into explicit character attributes. AMADEUS focuses on two categories: Belief and Value, and Psychological Traits. These are then included in the dynamic context for generation.

This is the paper’s strongest design move.

A standard RAG prompt might provide knowledge chunks and ask the model to answer as the character. AMADEUS adds an intermediate layer: infer the character’s stable traits from the evidence, then answer. That makes the final generation less dependent on the model spontaneously discovering the right persona signal inside raw text. It is the difference between handing an actor a pile of notes and saying, “Good luck,” versus giving them a short director’s cue: optimistic, loyal, easily distracted, values friendship, avoids direct conflict. Apparently actors, like language models, benefit from not being made to rummage through a filing cabinet mid-scene.

The ablation study supports this. On the MBTI setting, naive RAG reaches 65.00% accuracy with an average F1-score of 0.6146. Adding ACTS raises accuracy to 68.33%. Adding GS raises it to 75.00%. Adding AE produces the full AMADEUS result: 85.00% accuracy and 0.8394 average F1-score.

This is an ablation, not the whole proof. Its purpose is to show where performance gains enter the pipeline. The answer is clear enough: the biggest gain comes when the system stops merely retrieving chunks and starts expressing character attributes.

CharacterRAG exists because dialogue logs are the wrong benchmark

The paper also introduces CharacterRAG, a dataset built specifically for RAG-based role-playing agents. It contains persona documents for 15 fictional characters, 976K written characters, and 450 question-answer pairs. Each persona is reconstructed from the character’s own perspective, with external information removed when it could disrupt role-playing. The source material is based on Korean Namuwiki data.

The dataset uses six role-playing attributes:

The important move is that CharacterRAG evaluates both explicit character knowledge and out-of-knowledge persona consistency. The 450 QA pairs test whether the model can use written persona knowledge. The MBTI and Big Five/BFI interview-style questions test whether the agent can respond consistently when the exact answer is not in the persona.

This is not a perfect evaluation strategy. Personality-type labels for fictional characters are noisy, culturally mediated, and partly crowd-sourced through prior psychological typing sources. But the purpose is reasonable: if a role-playing agent claims to be a character, it should not only recite lore. It should also preserve a plausible behavioural profile when asked new questions.

That is the part many enterprise persona systems quietly ignore.

A brand agent can memorise product details and still fail brand identity. A tutor can know the curriculum and still fail pedagogical style. An NPC can know the quest line and still feel like a tax form wearing a sword.

The main evidence: AMADEUS improves persona fidelity where ordinary RAG struggles

The headline results are strongest in the psychological interview setting.

For MBTI prediction under GPT-4.1, AMADEUS reports 85.00% accuracy and 0.8244 average F1-score in the main comparison table. Naive RAG and LightRAG each reach 65.00% accuracy. CRAG reaches 68.33%. RAPTOR reaches 76.67%, and Adaptive RAG reaches 75.00%.

For Big Five/SLOAN prediction under GPT-4.1, AMADEUS reports 81.33% accuracy and 0.7986 average F1-score. Naive RAG reaches 72.00%, CRAG 76.00%, and LightRAG 34.67%. RAPTOR and Adaptive RAG are not included in that Big Five/SLOAN table.

The direct CharacterRAG QA results are more modest but still useful. With GPT-4.1, naive RAG scores 91.33% ACC, 9.23 ACCL, and 3.13 hallucination score; AMADEUS scores 92.67% ACC, 9.26 ACCL, and 2.89 hallucination score. With Gemma3-27B, AMADEUS moves from 86.44% to 88.00% ACC and from 8.85 to 8.92 ACCL, with hallucination score nearly unchanged at 3.26 versus 3.27. With Qwen3-32B, AMADEUS moves from 78.44% to 78.89% ACC and from 8.49 to 8.63 ACCL, while reducing hallucination score from 5.05 to 4.66.

That pattern matters. AMADEUS is not primarily a factual QA breakthrough. It is a persona-consistency architecture. It improves direct knowledge use, but its larger contribution is handling questions beyond explicit persona knowledge.

Web and graph retrieval are not automatically upgrades

One useful corrective in the paper is its treatment of CRAG and LightRAG. Operators often assume that web search or graph structures are natural upgrades over basic retrieval. Sometimes they are. For role-playing, they can become decorative machinery attached to the wrong problem.

CRAG, a corrective/web-search-oriented baseline, is included to test whether external search helps. The authors report mixed behaviour. In Big Five/SLOAN, CRAG reaches 76.00% accuracy, which is below AMADEUS but above naive RAG. In CharacterRAG QA, CRAG performs poorly relative to naive RAG and AMADEUS, particularly for some characters; the paper notes that despite web information, CRAG correctly answers only 6 of 30 CharacterRAG questions for Nina Iseri. The authors attribute this to noisy web retrieval and the difficulty of analogical reasoning for questions requiring persona inference.

LightRAG, a graph-based method, performs especially poorly on the Big Five/SLOAN table, at 34.67% accuracy. The authors argue graph-based RAG is poorly suited to RPA applications because of graph construction cost, entity ambiguity, difficulty updating knowledge, and challenges maintaining persona consistency.

The broader lesson is not “never use graph RAG” or “web search is bad.” That would be a pleasantly simple conclusion, and therefore suspicious. The better conclusion is that retrieval architecture must match the task. Character identity is not just a knowledge graph. It is a behavioural pattern over narrative evidence. Treating it as entity lookup misses the point with admirable confidence.

The business value is identity-preserving retrieval, not theatrical AI

The most obvious applications are entertainment: AI NPCs, interactive storytelling, VTuber companions, fan-character bots, and game agents that need to stay in-world. AMADEUS is directly relevant there. A character that breaks persona breaks immersion. Users do not need to know the phrase “retrieval-augmented generation” to feel that the magic has leaked out.

But the business relevance is broader.

Many enterprise agents are role-playing agents with less colourful costumes. A sales assistant has a brand persona. A financial education bot has a risk posture. A corporate training coach has a pedagogical style. A healthcare intake assistant has empathy and boundary constraints. A legal support bot has a tone and a refusal profile. These systems need facts, but facts alone do not define how they should speak.

Cognaptus’ inference from the paper is this: persona-aware retrieval should be treated as part of operating-model design, not UI polish.

The ROI logic is not that AMADEUS makes agents cuter. That is a sentence nobody should write in a budget memo unless they enjoy procurement meetings.

The ROI logic is that persona drift creates support cost, reputational risk, user churn, and content maintenance overhead. A retrieval system that can update persona documents without retraining, preserve hierarchical context, and infer stable attributes could reduce the operational burden of maintaining many specialised agents.

The key word is “could.” The paper provides a credible design signal. It does not provide a deployment benchmark for regulated industries, multilingual customer service, or brand-safety guarantees.

What to copy, what to test, and what to ignore for now

For operators building persona-heavy agents, AMADEUS suggests three practical design principles.

First, preserve document hierarchy. Do not flatten your persona material into anonymous chunks unless you want the model to behave as though every sentence was found loose in a drawer. Role, relationship, chronology, and section structure often determine meaning.

Second, retrieve for inferential relevance, not only lexical or semantic similarity. If the user asks a new question, the best evidence may be an action, habit, or relationship that implies the answer. A retrieval pipeline that cannot recognise inferential support will overfit to surface similarity.

Third, convert evidence into explicit operating attributes before generation. A short extracted summary of beliefs, values, and psychological traits can make the final response more stable than raw chunks alone.

What should be tested before production?

• Whether the attribute categories match your domain. “Belief and Value” and “Psychological Traits” work for fictional characters. A regulated banking assistant may need “Compliance Boundary,” “Risk Posture,” and “Escalation Rule.”
• Whether the LLM used for GS and AE is reliable enough under your corpus. The paper uses GPT-4.1 for these stages. Substituting a cheaper model may change the behaviour.
• Whether inference-time cost is acceptable. AMADEUS is training-free, not cost-free.
• Whether evaluation reflects actual failure modes. MBTI-style tests are useful for fictional-character fidelity, but enterprise agents need scenario tests, red-team prompts, and human review against brand or policy rubrics.

The part to ignore, at least initially, is the temptation to copy the whole method as a sacred recipe. The paper’s overlap coefficient analysis finds that $\alpha = 2$ maximises the sum of similarity scores while minimising variance under its assumptions. Useful? Yes. Universal law of retrieval physics? No. Please do not carve it into a dashboard.

The limitation is not the dataset size; it is the evaluation target

The paper’s limitations are not fatal, but they shape interpretation.

The dataset covers 15 fictional characters. That is enough to demonstrate the mechanism, not enough to claim broad generality across all persona systems. The characters span genres, but they remain fictional, media-derived, and reconstructed from a particular source context.

The evaluation also relies partly on personality-test proxies. MBTI and Big Five/SLOAN labels are convenient because they give a way to assess out-of-knowledge questions, but they are imperfect measures of character identity. A role-playing agent can match a personality type and still miss voice, humour, moral framing, timing, cultural references, or emotional nuance. Conversely, it might sound character-consistent to humans while missing a typed label.

The metrics for direct QA include LLM-based judgments: ACC, ACCL, and hallucination score. These are common in modern evaluation pipelines, but they are still judge-model-mediated. The paper strengthens this with human evaluation of attribute extraction, involving 14 evaluators and internal consistency above the usual acceptability threshold. Still, production teams should not outsource final persona evaluation entirely to another model. That is not governance. That is recursion with a tie.

Finally, AMADEUS assumes the persona document contains enough behavioural evidence for inference. If the source material is sparse, contradictory, or badly curated, guided selection and attribute extraction can only become more elegant ways to discover that the cupboard is empty.

The useful shift: from answer retrieval to character evidence

The best way to read AMADEUS is as a shift in retrieval philosophy.

For factual agents, retrieval asks: “Where is the answer?”

For role-playing agents, AMADEUS asks: “What evidence tells us who should be answering?”

That reframing is useful beyond anime characters and NPCs. It applies wherever an AI system must maintain a stable stance across changing user inputs: brand representatives, training coaches, simulated customers, expert personas, negotiation agents, and internal copilots with role-specific judgement.

The paper’s scores are encouraging, but the more durable contribution is architectural. ACTS preserves the structure of persona evidence. GS selects chunks that support inference. AE turns that evidence into explicit attributes. Together, they make RAG less like a search engine bolted onto a chatbot and more like a stage manager handing the actor the right cue before the line.

That is not the end of persona consistency. It is a better beginning than stuffing lore into a vector database and hoping the model remembers who it is. Hope is not an architecture, no matter how nicely it demos.

Cognaptus: Automate the Present, Incubate the Future.