Domain Adaptation Lost the Benchmark

TL;DR for operators

SinBrief¹ addresses a familiar specialised-AI problem: a team has documents worth summarising but no large collection of human-written document-summary pairs. Instead of making an end-to-end language model solve the entire task, it divides the work between structured candidate generation and a learned scorer.

That decomposition produced the paper’s strongest reported result. On 242 held-out Sinhala legal documents, SinBrief using a base Llama 3.1 scorer reached a SummaC score of 0.752. A version whose Llama 3.1 backbone had been continually pre-trained on the legal corpus reached 0.591. The base model also had higher sentence-scorer validation accuracy: 90.05% versus 88.37%.

For teams building specialised summarisation systems, the useful lesson is not that graph pipelines beat domain adaptation generally. It is that annotation strategy, generation strategy, and model adaptation can be tested as separate decisions. SinBrief offers a lower-annotation route when paired summaries are unavailable, while its negative continual-pre-training result argues for benchmarking an unchanged base scorer before paying for additional adaptation.

The release boundary is substantial. The authors report grammatical problems in generated text, outputs vary because candidate generation is stochastic, complete summaries receive no formal human evaluation, no ablation isolates the value of individual components, and the paper cautions that SummaC may be unreliable for long, complex legal sentences. This is evidence for an architecture worth testing, not evidence for autonomous legal summarisation.

Scarce summaries do not force an end-to-end solution

Suppose an organisation has thousands of specialised documents but cannot afford to commission thousands of corresponding human summaries. The conventional response is to ask how much additional labelled data, fine-tuning, or domain adaptation will be required.

SinBrief changes the unit of the problem. It does not require one model to learn the whole transformation from legal document to summary. It first breaks the source document into semantic regions, constructs possible summary sentences from those regions, and then asks a learned model to rank the candidates.

The paper develops this design on 1,206 Sinhala Acts and Bills from the SinhaLegal corpus. It uses 964 documents for development and 242 for held-out evaluation. The important architectural point is not the corpus size itself. It is that the system can train its ranking component without human-authored summaries.

That makes SinBrief annotation-light rather than unsupervised. Its supervision comes from heuristics and domain knowledge rather than document-summary pairs.

Generation and selection are deliberately separate

After sentence extraction, SinBrief groups related sentences into at most three semantic clusters:

where $N$ is the number of retained sentences.

Within each cluster, the system builds a directed graph of content-bearing Sinhala words and their part-of-speech tags. Connections involving one of 19 predefined legal keywords receive weight 2.0 rather than 1.0. Candidate sentences are then assembled by traversing this graph, with paths constrained to roughly 8–25 words and up to eight candidates generated per cluster.

This is where the system becomes genuinely abstractive: the candidate need not be copied as a complete source sentence. But graph connectivity captures lexical relationships and domain salience more readily than grammatical structure. That design choice later becomes one of the system’s main failure modes.

Candidate selection is learned separately. The paper constructs 38,920 balanced training examples without human sentence-quality labels. Positive examples come from signals such as document openings, closings, legal-keyword occurrence, and selected long sentences. Negative examples are created through corruption operations including shuffling, deletion, replacement, and truncation.

Five scoring backbones are tested: mBert, Llama 3.1 8B, Falcon 7B, Laser, and a continually pre-trained Llama 3.1. The highest-scoring two candidates from each cluster are used to construct a final summary constrained to approximately 300–500 characters.

The system therefore uses domain structure to propose text and neural representations to decide which proposals survive.

The scorer mattered more than additional domain pre-training

The comparison across scorers is the most consequential empirical result.

The continually pre-trained model had received additional exposure to the Sinhala legal corpus using causal-language-model training with LoRA. Yet it performed worse than the unchanged Llama 3.1 backbone on both scorer validation accuracy and downstream SummaC.

That result does not establish that continual pre-training is ineffective for Sinhala legal NLP. There is only one adaptation setup here, without a wider hyperparameter or multi-seed study. It does establish something narrower and operationally useful: domain adaptation did not earn its additional complexity in this experiment.

For an engineering team, that changes sequencing. Benchmark the base representation on the downstream decision first. Adapt only when the adapted model demonstrates task-level improvement.

The abstractiveness claim needs metric-level precision

SinBrief clearly produces lower extractive-fragment Density than the evaluated baselines. Its five configurations range from 2.924 to 3.125, compared with 7.605 to 8.153 for the three extractive baselines. Fine-tuned mT5 and BenSumm sit at 3.989 and 4.205 respectively.

Coverage tells a different story. SinBrief ranges from 0.956 to 0.959, while fine-tuned mT5 scores 0.855 and BenSumm 0.913. Because lower Coverage indicates less source overlap under the paper’s definition, those numbers do not support an unrestricted claim that SinBrief is more abstractive than every baseline on every measure.

This distinction matters because the paper’s accompanying discussion overstates the Coverage result relative to its own table. The defensible conclusion is metric-specific: SinBrief reduces the length and concentration of copied fragments substantially, but its results do not dominate all abstractive baselines across every lexical-overlap measure.

Compression Ratio is also difficult to interpret independently because final summary length is explicitly constrained to roughly 300–500 characters.

What Cognaptus infers for specialised document systems

The paper directly shows a functioning hybrid architecture under one low-resource legal benchmark. The business inference is broader but conditional.

A team without paired summaries could test a similar decomposition before funding a full supervised summarisation programme:

1. use document structure and domain knowledge to constrain candidate generation;
2. create inexpensive weak labels for a candidate-quality model;
3. benchmark several existing representations as scorers;
4. compare any domain adaptation against the untouched base model;
5. reserve expensive human annotation for evaluation and failure cases where it changes the release decision.

The affected decision is therefore not simply which language model to fine-tune. It is where supervision should enter the system.

SinBrief places much of it at the selection stage. That can reduce dependence on human-written reference summaries, but it shifts quality risk into candidate generation and evaluation.

Automatic scores do not clear the deployment boundary

The strongest SummaC result, 0.752, should not be read as verified factual correctness. The authors themselves caution that sentence-level NLI-based evaluation can struggle with long, complex legal sentences and documents.

More consequentially, the native-Sinhala-speaking authors observed grammatical errors, reduced fluency, and limited naturalness in the generated summaries. The graph traversal has no explicit grammatical constraint. Outputs can also vary between runs because candidate generation is stochastic, yet the study does not perform multi-seed evaluation.

There is no formal human evaluation of complete summaries, and no ablation isolates whether legal-keyword weighting, graph construction, or another component produces the reported gains. Sentence-scorer validation accuracy is likewise a binary-classification measure, not a direct test of ranking quality over the graph-generated candidates encountered in production.

For legal-technology use, those gaps keep SinBrief in an assistive role: search support, triage, document navigation, or first-pass summarisation with human review. The architecture reduces one bottleneck—paired-summary annotation—without resolving the separate problem of proving that generated legal summaries are consistently fluent and reliable.

The cheaper experiment comes before the more specialised model

SinBrief’s most transferable contribution is architectural rather than linguistic. It demonstrates that low-resource summarisation can be decomposed so that structured generation supplies candidates and learned models concentrate on selection.

Its strongest scorer was also not the one given additional domain pre-training. That makes a useful experimental ordering explicit: establish the base-model benchmark, test whether system decomposition already supplies enough task structure, and require adaptation to demonstrate incremental value on the actual downstream metric.

For organisations with specialised documents and limited annotation budgets, that can prevent two premature investments: building a large paired-summary dataset before knowing where supervision is needed, and adapting a model before showing that adaptation improves the decision the model actually performs.

The paper does not establish production readiness. It does provide a credible design hypothesis for what to test first.

Cognaptus: Automate the Present, Incubate the Future.