Easy First, Not Best Overall

TL;DR for operators

A team choosing a sequence-labeling model with limited annotated data should compare both prediction accuracy and the cost of producing it. With only 10% of English CCG Rebank training data, TagInsert achieves 90.99% accuracy against 86.78% for a left-to-right Transformer, but BERT still leads at 92.35%. TagInsert’s confidence-guided insertion strategy makes the largest difference against conventional autoregressive decoding when labeled data are scarce.

The advantage is not universal. TagInsert performs extremely poorly on rare English CCG categories, despite numerically outperforming both baselines on rare categories in three small multilingual datasets. Its reported training cost is also substantially higher. For document-processing or low-resource language systems, dynamic insertion deserves targeted benchmarking, not automatic preference over encoder-only classification.

When training data are scarce, prediction order becomes a design choice

A document-processing system may need to assign grammatical or structural labels to individual words. With limited annotated examples, some decisions are straightforward while others depend on relationships elsewhere in the sentence. Predicting labels strictly from left to right can force the system to resolve an ambiguous word before more informative labels have been assigned.

The problem becomes more demanding when the labels describe detailed syntactic functions rather than ordinary parts of speech. In Combinatory Categorial Grammar (CCG) supertagging, each word receives a structured syntactic category. Categories can be numerous, mutually informative, and sparsely represented in training data.

Bais and Deoskar investigate whether changing the order of these decisions improves accuracy in their study of TagInsert, a dynamically ordered Transformer tagger.¹

The initial result is encouraging. On English CCG Rebank, with 10% of available training examples, TagInsert exceeds a matched left-to-right Transformer by 4.21 percentage points. Yet BERT remains 1.36 points ahead of TagInsert (Section 5.2, Table 2).

That comparison establishes the central model-selection issue: improving the sequence in which labels are generated does not necessarily outperform a model that classifies words using bidirectional input representations.

TagInsert lets earlier decisions come from later words

A conventional autoregressive decoder generates tags in sentence order. Its causal attention mask prevents it from using output tags at positions it has not yet generated.

TagInsert removes this restriction. The decoder initially contains an unknown-tag placeholder, <U>, for every word. It evaluates candidate labels for the unfilled positions, selects the position-tag pair with the highest predicted probability, inserts that tag, and repeats.

Subsequent predictions can condition on already inserted tags from either side of the current word. Unassigned positions remain placeholders: the model does not receive future gold-standard labels during inference.

Training prepares the model for varied partial assignments by sampling random insertion orders. At each training step, it calculates the average negative log-likelihood of the correct tags across all positions still unfilled, rather than supervising only the next position in a fixed sequence (Section 3.2, Equation 1). The architecture also uses pretrained BERT-based encoder representations and word-tag embedding alignment.

The proposed benefit is that confident decisions provide output-side syntactic context for less certain ones. A difficult tag can be predicted after relevant neighboring or distant tags have already been assigned.

Unlike BERT’s conventional word-level classifier, TagInsert explicitly conditions on previously predicted output labels. Unlike a left-to-right decoder, it can choose which label to predict next.

Reduced training data magnifies the advantage over left-to-right decoding

The main benchmark comparisons cover English Penn Treebank part-of-speech tagging and CCG Rebank supertagging, each evaluated at 100%, 50%, 25%, and 10% of training data.

On CCG Rebank, TagInsert’s advantage over the Vanilla Transformer grows from just 0.17 percentage points at full data to 4.21 points at 10% data. The paper reports statistically significant TagInsert–Vanilla differences except in the full-data condition (Section 5.2, Table 2).

Penn Treebank shows a similar direction. At 10% training data, TagInsert scores 96.66%, compared with 95.02% for Vanilla. BERT, however, reaches 97.48% (Section 5.1, Table 1).

These controlled benchmark results are the strongest empirical case for flexible insertion. The same sampled training subsets are used across models, and TagInsert consistently exceeds the matched left-to-right baseline.

But the baseline determines the conclusion. BERT achieves the highest overall accuracy at every English training-data proportion. Even where flexible generation substantially improves an encoder-decoder system, conventional bidirectional classification remains competitive or superior.

The multilingual results reinforce this distinction. On Italian CCG supertagging, TagInsert reaches 89.96%, compared with 81.12% for Vanilla and 90.53% for BERT. On Dutch, the corresponding figures are 91.62%, 84.15%, and 93.01%. The reported TagInsert–BERT differences are not statistically significant in these two languages, although that does not establish equivalence. In German, BERT’s advantage is statistically significant (Section 5.3, Table 4).

The rare-category results reverse the apparent advantage

The expectation that dynamic ordering should particularly help difficult or infrequent labels meets a substantial counterexample.

The researchers group CCG categories according to training frequency. Rare categories appear only 1–9 times in training. Their accuracy can differ dramatically from aggregate accuracy because frequent categories account for most evaluation tokens.

Source: Tables 3 and 5. Accuracy values are three-run means; the reported rare-category bins are small.

The English result is especially consequential. TagInsert’s full-data overall Rebank accuracy exceeds Vanilla’s, yet its rare-category accuracy is nearly zero. An aggregate improvement therefore coexists with a severe failure on one category-frequency segment.

The multilingual pattern points in the opposite direction: TagInsert obtains the highest numerical rare-category accuracy in German, Italian, and Dutch. But the test bins contain only 47, 33, and 34 tokens, respectively. Dutch TagInsert rare-category accuracy also varies considerably across runs, with a reported standard deviation of 12.82 percentage points.

These frequency-stratified analyses reveal heterogeneous performance, not reliable universal long-tail generalization. Differences in dataset size and annotation consistency further restrict direct English–multilingual comparisons.

For applications where infrequent labels correspond to consequential processing decisions, the English reversal is sufficient reason to require separate rare-label evaluation before deployment.

Late insertion is consistent with deferral, not proof of its benefit

The paper investigates whether TagInsert postpones difficult predictions.

Among positions where TagInsert is correct and BERT is wrong, the average normalized insertion position is 94% of sentence length for English POS tagging and 89% for English CCG supertagging. Italian PMB shows a corresponding figure of 91.2% (Sections 5.1–5.3; Figures 3–4).

These descriptive analyses are consistent with an easy-first mechanism: predictions that differentiate TagInsert from BERT often occur after most other tags have been assigned.

However, the measurements are conditional on selected cases where TagInsert succeeds and BERT fails. They do not experimentally establish that delaying those positions caused their correct classification. Nor do they explain why the same strategy performs so poorly on English rare supertags.

The architecture and observed insertion behavior offer a plausible explanation for the benchmark gains, but the mechanism remains incompletely identified.

Training expense changes the adoption decision

In the reported demanding full-data configuration, training one TagInsert epoch takes approximately 10 hours, compared with 20 minutes for Vanilla and 12 minutes for BERT on an NVIDIA A100-SXM4-40GB GPU (Section 3.3; Appendix A).

This is a substantial computational premium. Per-epoch measurements do not by themselves determine total training cost, and the study does not establish production inference latency or throughput. TagInsert also performs repeated insertion steps at inference rather than assigning every label in one classification pass.

There is an additional experimental boundary: TagInsert and Vanilla use different loss computations, so the matched architecture does not isolate generation order perfectly. Comparisons with BERT involve further architectural differences. Alternative insertion policies and downstream production outcomes are not established.

For Cognaptus, the appropriate business inference is therefore a bounded evaluation strategy. A document-processing team should compare TagInsert against both autoregressive and encoder-only alternatives using its actual annotation budget. It should measure overall token accuracy, performance by label frequency, variability across runs, and training and inference costs separately.

Where dependencies among predicted labels are substantial and annotated data are scarce, dynamic insertion offers a credible research direction. Where BERT already delivers higher overall accuracy at materially lower training cost, the case for switching requires a demonstrated task-specific benefit.

TagInsert shows that the order of prediction can improve a sequence model. Its results also show why a better ordering strategy is not, by itself, a better model-selection decision.

Cognaptus: Automate the Present, Incubate the Future.