Domain Adaptation Lost the Benchmark
TL;DR for operators SinBrief1 addresses a familiar specialised-AI problem: a team has documents worth summarising but no large collection of human-written document-summary pairs. Instead of making an end-to-end language model solve the entire task, it divides the work between structured candidate generation and a learned scorer. That decomposition produced the paper’s strongest reported result. On 242 held-out Sinhala legal documents, SinBrief using a base Llama 3.1 scorer reached a SummaC score of 0.752. A version whose Llama 3.1 backbone had been continually pre-trained on the legal corpus reached 0.591. The base model also had higher sentence-scorer validation accuracy: 90.05% versus 88.37%. ...