FormuLLA: When LLMs Stop Talking and Start Formulating

Formulation is where AI enthusiasm usually goes to sober up.

In a slide deck, “AI-assisted drug development” sounds clean: feed the model a drug, get back a formulation, reduce experiments, accelerate personalisation, everybody nods. In a lab, the problem is less polite. A formulation is not just a sentence with chemical names. It is a physical recipe with roles, proportions, processing constraints, and mechanical consequences. A model can sound fluent while quietly omitting the lubricant, mangling the unit, or inventing a polymer that belongs more to fantasy literature than pharmaceutics.

That is why FormuLLA is more interesting than its title first suggests. The paper is not merely asking whether large language models can discuss pharmaceutical 3D printing. It asks whether they can generate plausible excipient recommendations for fused deposition modelling, given an active pharmaceutical ingredient and dose.¹ That is a harder, colder question. It moves the LLM from commentary into formulation design, where language is only useful if it survives contact with material reality.

The most useful way to read the paper is through comparison. General model versus biomedical model. Fine-tuning versus forgetting. BLEU and ROUGE versus formulation-aware evaluation. Generation settings versus training settings. This is where the paper’s real business lesson sits: in technical domains, model choice matters, but validation design matters more.

The task is not prediction; it is reverse formulation

Most earlier AI work in pharmaceutical 3D printing has been discriminative. Given a formulation, predict whether it is printable. Given ingredients and process data, estimate dissolution, mechanical properties, or processability. That is valuable, but it leaves the human scientist doing the hard upstream search.

FormuLLA reverses the direction.

Instead of asking:

Here is a formulation. Will it work?

the model is asked:

Here is an API and dose. What excipients should I try?

That reversal changes the difficulty. The input becomes short. The output becomes broad. The model must generate a multi-component formulation with excipient names, quantities, printability, and filament aspect. It is not enough to classify. It must propose.

The authors take an existing FDM pharmaceutical formulation dataset, originally structured for conventional machine-learning use, and transform it into instruction-response data. The dataset contains more than 1,400 formulations, structured across more than 339 columns, covering 61 APIs and 276 excipients. Each row is converted into an Alpaca-style training example: an instruction such as “Recommend excipients for 20 w/w% Ciprofloxacin,” followed by a response listing excipients, proportions, printability, and filament mechanical aspect.

That conversion is the first major contribution. It is easy to underestimate because it does not look glamorous. But in applied AI, the supposedly boring step is often the whole product. A dataset designed for tabular prediction does not automatically become useful for generative formulation. Someone has to decide what the model is allowed to see, what it must produce, and how domain knowledge is represented as language without becoming vague theatre.

Here, the output is not a generic paragraph. It is a structured recommendation. That design choice is what lets the authors compare models not only as language generators, but as candidate formulation assistants.

Four models enter; the biomedical one does not win

The paper fine-tunes four open-source models: Llama 2 7B, Mistral at roughly 7B parameters, T5-XL or Google-FLAN-XL at roughly 3B parameters, and BioGPT at roughly 350M parameters. BioGPT is the obvious temptation. It is trained on biomedical literature, including PubMed abstracts, and one might expect it to handle drug-related text more naturally.

That expectation does not survive the experiments.

Llama 2 performs best overall. During training, Llama 2 and Mistral show the strongest convergence after four epochs, with final average losses of 0.1325 and 0.1830 respectively. T5-XL converges at a much higher loss of 0.9339. BioGPT starts high and remains high, converging at 2.9047.

The cleaner interpretation is not “biomedical models are bad.” That would be lazy, and therefore popular. The better interpretation is that domain pretraining is not a substitute for architecture, capacity, tokenisation behaviour, fine-tuning stability, and evaluation discipline. BioGPT may know biomedical language better in some settings, but this task is not merely biomedical language. It is structured generation over specialised excipient names, formulation roles, and quantities.

The practical point is sharp: model selection cannot be reduced to “which model has the most relevant pretraining corpus?” In technical workflows, the right model is the one that can be adapted without losing the structure the task requires.

Fine-tuning can teach the model, or erase it

The learning-rate experiments are not just hyperparameter tuning. They are a useful warning about technical-domain adaptation.

At a learning rate of $10^{-2}$, performance largely collapses. BLEU scores are close to zero for most models, and BioGPT fails to fine-tune under that setting. At $10^{-6}$, training becomes too weak to produce the desired adaptation, and outputs degrade in different ways. The viable point in this study is $10^{-4}$.

At $10^{-4}$, Llama 2 and T5-XL perform best on standard language metrics. Llama 2 achieves a BLEU score of 0.62 and T5-XL 0.53. Their ROUGE-1 scores are also relatively strong, at 0.78 ± 0.068 and 0.70 ± 0.12 respectively. Mistral and BioGPT are weaker on those language metrics, with BLEU scores of 0.26 and 0.24, and ROUGE-1 scores of 0.48 ± 0.14 and 0.47 ± 0.056.

But the more revealing evidence is qualitative. BioGPT produces corrupted excipient names and nonsensical alphanumeric strings. At one point, it generates a formulation containing a malformed “Polyvinyl Carbatesuccinatesuccinatesuccinate” and stray non-English or corrupted text. Llama 2 and T5-XL, by contrast, can produce recognisable excipient names and more coherent formulation structures at the viable learning rate.

This is where “catastrophic forgetting” stops being a textbook phrase and becomes a procurement risk. Fine-tuning on a relatively small dataset can overwrite useful prior behaviour. Smaller models appear especially vulnerable in this paper. The issue is not that the model does not know enough. It is that the adaptation process can damage what it already knows.

For a business team, this distinction matters. A failed fine-tune does not always announce itself with an error message. Sometimes it produces something that looks structured, contains technical nouns, and is still wrong. That is the annoying category: outputs too plausible to ignore and too unreliable to trust.

More adapters help Mistral, but they do not magically fix everything

The paper also tests LoRA adapter configuration. The baseline applies LoRA to the attention Query and Value projections. The authors then add Key and Output projection adaptation to see whether expanding the adapted layers improves generation.

This is best read as an ablation. The question is not “Are more adapters always better?” The question is “Does giving the fine-tune more representational access improve this task for this model?”

For Mistral, yes. With additional K and O adapters, Mistral’s BLEU score rises from 0.26 to 0.51. ROUGE-1 improves from 0.48 ± 0.14 to 0.66 ± 0.14. ROUGE-2 rises from 0.39 ± 0.14 to 0.51 ± 0.14, and ROUGE-L from 0.48 ± 0.15 to 0.64 ± 0.15. The paper reports this as statistically significant for Mistral, but not for BioGPT, Llama 2, or T5-XL.

That result is easy to misuse. The lesson is not “always add more LoRA layers.” The better lesson is that adapter placement interacts with architecture and task. Some models need more trainable access to represent the domain shift. Others already perform well enough under the simpler configuration, or fail for reasons additional adapters do not solve.

BioGPT is the useful cautionary case. The extra adapters improve some outputs, but corrupted text remains. A model can move closer to the right excipient space while still failing as a clean language interface. In a lab workflow, that distinction matters. A scientist can sometimes interpret ugly output; an automated pipeline cannot afford to “sort of understand” corrupted text.

BLEU and ROUGE measure fluency, not formulation competence

The paper’s most important contribution is arguably not that Llama 2 wins. Model leaderboards age badly. Evaluation lessons age better.

BLEU and ROUGE are useful for measuring similarity between generated text and reference text. They tell us whether the model preserves linguistic and token-level patterns. That is not the same as asking whether a formulation is scientifically sensible.

The authors notice this gap and introduce VELVET: Validation Excipient List Verification and Evaluation Tool. VELVET parses predicted and reference excipient lists, normalises ingredient names, places ingredients into an embedding space based on formulation co-occurrence, and computes distances between predicted and reference excipient compositions. Lower VELVET scores are better. Missing ingredients are penalised using the maximum observed pairwise embedding distance, reported as 9.4475.

This is not a perfect metric. It is still based on similarity to historical co-occurrence, not wet-lab validation. But it is a move in the right direction because it evaluates the output as a formulation candidate, not as a paragraph.

Under VELVET, Llama 2 again performs best, with the lowest reported score at $4.03 \pm 1.40$ under the $10^{-4}$ learning rate without additional adapters. BioGPT produces the second-lowest reported VELVET score at $4.7 \pm 2.95$, even though its language output can be visibly corrupted.

That apparent contradiction is the paper’s best teaching moment. BioGPT can sometimes land near historically plausible excipient choices while failing to produce clean text. BLEU and ROUGE can miss formulation relevance. VELVET can miss linguistic corruption. No single metric is enough.

A production system would need layered evaluation:

VELVET is therefore not the final answer. It is the necessary protest against using generic language metrics as if they were domain validation.

Generation settings are tuning knobs, not a rescue mission

The paper then varies temperature and top-p. This is best treated as a robustness or sensitivity test, not as a second thesis.

The result is uneven. Llama 2 is stable across these generation settings, with little effect across metrics. T5-XL is more sensitive: changing temperature to 0.3 or 0.5 reduces BLEU from around 0.50 to around 0.12. Mistral also suffers when top-p or temperature deviates from the default setting. BioGPT, interestingly, improves under some generation-parameter changes, especially top-p.

The practical interpretation is simple: inference settings matter, but they are downstream of adaptation quality. Temperature and top-p can alter diversity, determinism, and output stability. They cannot reliably convert a poorly adapted model into a dependable formulation assistant.

This matters because many business deployments overfocus on prompting and inference settings. That is understandable. Those settings are visible, cheap to change, and psychologically satisfying. But in FormuLLA, the larger performance drivers are model architecture, learning rate, adapter configuration, and evaluation design. The knobs on the dashboard are not the engine.

The experiment stack is small, but it is correctly aimed

The paper conducts 60 experiments across training and generation settings. That is not enormous by industrial AI standards, but it is enough to reveal the core failure modes.

The training-time results are also worth noting. At the optimal learning rate, reported fine-tuning times range from 3 minutes 42 seconds for BioGPT to 16 minutes 32 seconds for Mistral. Llama 2 takes 14 minutes 28 seconds, while Google-FLAN-XL takes 5 minutes 32 seconds. These are GPU-based experiments and should not be blindly generalised to every infrastructure setting, but they show why PEFT is operationally attractive: the adaptation loop is short enough to support repeated experimentation.

That is the upside. The downside is that fast iteration also makes it easy to generate many superficially plausible models without deeply validating any of them. Cheap training is useful only if paired with expensive seriousness.

What businesses should actually take from FormuLLA

The business value here is not “replace formulation scientists with an LLM.” Please do not do that. Reality has suffered enough.

The realistic value is candidate generation and screening. A system like this could help formulation teams move from a large search space to a smaller, ranked set of plausible excipient combinations. It could be especially useful when paired with historical formulation databases, rule-based constraints, and domain-native evaluation metrics.

What the paper directly shows:

• LLMs can be fine-tuned on structured pharmaceutical FDM formulation data converted into instruction-response format.
• Model choice and fine-tuning configuration materially affect output quality.
• Llama 2 performs best among the tested models under the reported conditions.
• Biomedical pretraining alone does not guarantee superior formulation generation.
• Standard language metrics do not adequately evaluate excipient recommendation quality.
• A domain-aware metric such as VELVET can expose a different performance dimension.

What Cognaptus infers for business use:

• LLMs are most useful here as early-stage formulation assistants, not autonomous decision-makers.
• The main ROI pathway is reducing experimental search cost, not eliminating experiments.
• Domain evaluation should be designed before deployment, not patched on after a demo looks impressive.
• A strong formulation AI system will likely be hybrid: LLM generation, structured rules, similarity metrics, domain databases, human review, and lab validation.

What remains uncertain:

• Whether the recommended formulations perform well in wet-lab validation beyond similarity to historical references.
• Whether the approach generalises beyond FDM and beyond the dataset’s API/excipient distribution.
• Whether VELVET correlates strongly with actual printability, manufacturability, stability, or regulatory acceptability.
• Whether longer training schedules, alternative learning rates, richer prompts, or newer models would change the ranking.
• Whether a formulation-specific model trained on excipient-rich corpora would outperform general models under controlled conditions.

That uncertainty is not a weakness of the paper. It is the boundary of the evidence. Good applied AI work should make the boundary visible. Otherwise it is not science; it is sales enablement with equations.

The hidden bottleneck is not generation; it is evaluation design

FormuLLA belongs to a broader shift: LLMs are moving from text production into technical decision support. The moment that happens, generic evaluation becomes dangerous.

In a normal chatbot, a fluent but shallow answer is annoying. In formulation design, a fluent but wrong answer can redirect experiments, waste materials, distort priorities, or create false confidence. The model does not need to be malicious. It only needs to be plausible.

The paper’s comparison-based evidence points to a more mature operating model for technical LLM deployment:

1. Start with a task that has a clear operational decision.
2. Convert historical data into instruction-response examples without destroying structure.
3. Compare multiple model families instead of trusting domain labels.
4. Treat fine-tuning as an engineering experiment, not a ritual.
5. Evaluate both language integrity and domain validity.
6. Use generation settings as stability controls, not magic.
7. Keep human and laboratory validation in the loop.

This is less glamorous than “AI discovers new formulations.” It is also much closer to something a serious organisation could use.

From fluent assistant to formulation co-pilot

The paper’s central warning is nicely inconvenient: the model that sounds most domain-specialised is not necessarily the model that works best for the domain task. BioGPT looks attractive because biomedical pretraining feels relevant. Llama 2 wins because adaptation quality and architecture matter more in this setup.

That should make buyers more cautious and builders more disciplined. The next useful pharmaceutical LLM will not be the one with the most impressive biomedical label. It will be the one whose outputs can be checked against physical, chemical, and operational constraints.

FormuLLA does not prove that LLMs can autonomously formulate 3D-printed medicines. It shows something narrower and more useful: with the right data transformation, careful PEFT, and domain-aware evaluation, LLMs can begin to participate in formulation search.

That is enough. In serious technical domains, “begin to participate” is already a meaningful achievement. The machine does not need to wear the lab coat. It just needs to stop confusing language with competence.

Cognaptus: Automate the Present, Incubate the Future.