The Protein Signal Is Not the Whole Gain

TL;DR for operators

Adding protein-sequence intelligence to a biological annotation system does not produce uniform gains. In SoftGene’s experiments, pretrained protein embeddings barely improve Gene Ontology biological-process annotations, but substantially improve cellular-component annotations. Removing learned soft prompts causes a much larger deterioration across both major benchmarks.

For bioinformatics platform teams, the relevant decision is which components justify their additional complexity in a particular biological domain. The evidence supports evaluating sequence embeddings, soft prompting, and auxiliary context separately. Generated descriptions can assist research interpretation, but benchmark similarity and plausible attention rankings do not establish biological correctness.

The biological input does not always improve the description

A research software team wants to generate short functional descriptions of groups of genes. Adding information about the proteins those genes encode seems sensible: amino acid sequences contain biological signals that gene names alone may not reveal. But collecting, encoding, and maintaining those representations adds computational and engineering requirements. The investment depends on how much additional annotation quality they actually deliver.

Gene set annotation is the task of describing the biological function or pathway represented by a collection of genes. Researchers use such descriptions to interpret functional genomics results, often comparing them with curated resources.

In SoftGene: Protein Language Model-Enhanced Soft Prompting for Interpretable Gene Set Annotation, Drew Ross and coauthors introduce a framework that incorporates protein-sequence representations into language-model-generated annotations.¹

Its most revealing result is not the aggregate benchmark improvement. It is how differently the pretrained protein representations perform across biological domains.

In the Gene Ontology (GO) biological-process domain, replacing randomized embeddings with pretrained protein embeddings changes METEOR from 0.222 to 0.223. METEOR measures similarity between generated text and a reference description, accounting for word alignment and matching.

For cellular-component annotations, however, the same architectural comparison raises METEOR from 0.102 to 0.164. Molecular-function annotations improve from 0.213 to 0.236.

These are component-ablation results from Section 3.4.5, Table 5. They indicate that pretrained protein representations offer very different incremental value depending on the target annotation domain.

A platform focused on cellular localization therefore has stronger benchmark evidence for investing in sequence representations than one focused primarily on biological processes. The paper does not explain conclusively why the difference arises, but it establishes why aggregate performance cannot settle that engineering decision.

How SoftGene connects proteins to language generation

SoftGene combines three mechanisms that contribute different information to the final annotation.

First, a pretrained protein language model, ESM-2, converts amino acid sequences into numerical representations. The system averages residue embeddings into protein representations, then uses hierarchical attention to aggregate protein isoforms into genes and genes into a complete gene set.

This matters because genes can produce multiple protein isoforms. Treating every gene as a single symbolic identifier would discard that distinction.

Second, a trainable projection converts the aggregated gene-set representation into continuous soft prompt tokens. These are learned vectors placed in the input embedding space of a frozen language model. Rather than updating the generator’s parameters, the system learns representations that condition its output.

Third, GPT-4.1 Nano generates auxiliary biological context from gene symbols. This text accompanies the soft prompts and task instructions when the local language model generates its annotation.

The principal configuration uses a frozen Llama 3.1 8B generator. Training updates the hierarchical encoder and soft prompt projector to improve prediction of reference annotation tokens. The authors use ten soft prompt tokens in their selected configuration.

The resulting architecture combines biological sequence representations, learned task conditioning, and textual biological knowledge. Its benchmark advantage cannot automatically be assigned to any one of those components.

Removing soft prompts hurts far more than removing pretrained protein embeddings

The component ablation in Section 3.4.2, Table 3 provides the clearest evidence about what drives SoftGene’s performance.

Reported means across three experimental runs. The randomized-embedding condition retains the hierarchical architecture but replaces pretrained protein representations.

Removing soft prompts reduces GO METEOR by 0.200 points and MSigDB METEOR by 0.151 points. These are very large deteriorations relative to the complete system.

The result identifies learned continuous conditioning as central to SoftGene’s performance under the tested architecture. It does not establish that soft prompts would dominate every alternative architecture, since removing a component also changes how the remaining components interact.

Removing auxiliary context causes smaller declines: 0.220 to 0.182 on GO and 0.157 to 0.145 on MSigDB. The system remains above the evaluated baseline methods on METEOR without that context, suggesting that an externally generated contextual description is beneficial but not indispensable for competitive reference matching.

Replacing pretrained protein embeddings has a more uneven effect. GO performance declines modestly, from 0.220 to 0.199, while MSigDB falls from 0.157 to 0.097.

The stronger interpretation is therefore architectural: SoftGene’s combined prompting system performs well, while sequence-derived features provide additional gains that depend on the biological dataset and domain.

This distinction matters commercially. A research platform should not assume that its most specialized input representation is responsible for most of the measured improvement.

Better reference matching is not verified biological understanding

SoftGene is evaluated on 18,889 GO gene sets and 2,461 MSigDB pathway sets, using random 70/30 training and testing partitions.

With Llama 3.1 8B, it achieves GO METEOR of 0.220 and MSigDB METEOR of 0.157, compared with 0.148 and 0.113 for P-Tuning. It also leads the evaluated baselines on ROUGE-L and BERTScore F1.

Those comparisons are the paper’s main performance evidence. They cover prompting methods, prompt-tuning alternatives, and biological annotation agents, including GeneAgent. However, baseline model configurations and external tool access are not fully matched. The results establish superiority within the reported experimental protocol, not a fully controlled ranking of architectures at identical capacity and resources.

Additional semantic evaluation narrows the apparent advantage. On GO, SoftGene reaches biomedical semantic similarity of 0.888 versus 0.882 for P-Tuning. Its ontology-aware hierarchical similarity is 0.900 versus 0.896. These differences are considerably smaller than the corresponding METEOR advantage.

A separate robustness experiment attempts to minimize overlap between training and testing genes. GO METEOR declines from 0.220 under random splitting to 0.205 under the gene-disjoint procedure. MSigDB declines from 0.157 to 0.136.

The modest mean reductions are consistent with some transfer beyond repeatedly observed gene compositions. But complete gene-level disjointness is not guaranteed, and the MSigDB gene-disjoint estimate has a wide reported 95% confidence interval of ±0.082.

Most importantly, the evaluation rewards agreement with curated descriptions. Even a high similarity score can conceal a biologically meaningful omission.

One high-scoring GO example predicts “diacylglycerol kinase activity” when the reference specifies “ATP-dependent diacylglycerol kinase activity.” The prediction receives BERTScore F1 of 0.95 despite omitting the ATP-dependence qualifier.

Other qualitative failures involve incorrect substrate specificity, confusion between related pathways, and mismatched annotation granularity.

For a research platform, these errors affect whether a generated annotation is suitable for preliminary interpretation or sufficiently precise to support a downstream biological claim. The benchmark measures the former much more directly than the latter.

Attention rankings offer inspection, not causal explanation

SoftGene’s hierarchical attention mechanism also exposes which genes and protein isoforms receive greater learned weight.

In a zinc transport case study, prominent attention falls on known transporter-family members, including SLC39A5 and SLC39A4. In a smooth muscle contraction example, the model emphasizes CALCA, ADRB2, ADRB3, and a functionally relevant PRKG1 isoform.

These examples support the plausibility of the learned relevance rankings. They are not experiments showing that the highlighted features causally produced the annotation.

For an analyst-facing product, attention weights could help users identify which inputs the model prioritized and investigate suspicious descriptions. They should not be presented as independently established evidence that a particular gene or isoform determines a biological function.

Deployment should follow domain-specific gains and complete operating costs

The paper suggests a conditional deployment strategy for bioinformatics vendors and AI-assisted research teams.

A team generating mainly biological-process annotations should first establish whether pretrained protein embeddings outperform a randomized-embedding configuration enough to justify their operational costs. A team working on cellular-component descriptions or MSigDB pathways has stronger evidence of incremental benefit, although it still needs validation on its own data.

The same reasoning applies to auxiliary context. Removing GPT-4.1 Nano context reduces measured performance, but the reduced system remains competitive. An organization requiring a fully local workflow can evaluate that tradeoff instead of assuming the commercial model is mandatory.

Runtime measurements also require care. SoftGene averages 103.18 milliseconds per GO evaluation sample, compared with 84.51 milliseconds for a zero-shot baseline, on a single NVIDIA A6000 GPU. Protein embeddings are computed and cached separately.

Those figures exclude the auxiliary-context generation pass required for previously unseen gene sets. They therefore do not represent complete end-to-end production latency. The reported training peak memory of 46.39 GB across two A6000 GPUs also indicates that parameter-efficient adaptation does not eliminate substantial hardware requirements.

A production evaluation should consequently measure performance by biological subdomain, component-level improvement, context-generation cost, caching requirements, and error categories that matter to downstream users.

None of these operational outcomes, including analyst productivity gains or research cost savings, is directly established by the paper.

A useful annotation is still a candidate interpretation

SoftGene advances a credible design for combining specialized biological representations with frozen language models. Its benchmark results show better reference matching, while its ablations reveal that the contribution of protein-sequence information is less uniform than the complete system’s performance might suggest.

For research platforms, the strongest adoption principle is selective: retain components that improve the target biological workload under comparable evaluation conditions, and distinguish learned textual agreement from independently verified biological knowledge.

SoftGene provides a potentially effective way to generate and inspect candidate functional annotations. Its reported evidence does not establish new biological functions, causal explanations, or clinical reliability. Those conclusions require forms of validation beyond reference similarity and attention analysis.

Cognaptus: Automate the Present, Incubate the Future.