📄 Cleaning English Abstracts of Scientific Publications

Summary

Abstracts are widely used as stand-ins for what a paper is about. Many of them, however, carry non-content material: publisher copyright lines, journal section headings and templates, keywords and JEL codes, trial registrations, funding notes, figure pointers, and citations with full bibliographic detail. This material makes papers from the same journal or publisher look artificially alike, and it wastes space in fixed-length embeddings. The authors manually annotated 9,000 English Scopus abstracts dating from 1970 onward, marking every unwanted stretch with a single label, REM. They trained a spaCy span-recognition model to detect those stretches so they can be cut out.

On a 20% hold-out, token-level F1 is 0.945. The model is cautious: it almost never deletes text from abstracts that contain no clutter. Citations are its weakest category. Cleaning changes SPECTER2 similarity rankings, which matters for anyone doing similarity or embedding work on scholarly text. The model and the dataset are both released on Huggingface.

Key points

• Data and labeling:
  • The data are 9,000 abstracts, labeled through an interactive interface with one entity type, REM, and split 80/20.
  • Sampling started from random 2018 papers and was then enriched with abstracts containing known clutter types, so 2018 is over-represented.
  • Fields follow the top-level ASJC classification.
• Training:
  • The model was trained as NER in spaCy, with up to 100 epochs and early stopping after 20 epochs without improvement.
  • Training stopped at epoch 52, after about 4 hours on four A100 GPUs with an effective batch size of 16.
• Hold-out scores: precision 0.973, recall 0.919, F1 0.945.
• Results on the training sample:
  • Clean abstracts were left untouched 99.93% of the time.
  • Abstracts with clutter were handled correctly in 96.97% of cases.
  • Over-deletion consisted mostly of punctuation and numbers.
• Unseen test sets (100 abstracts each, selected by regex):
  • Citations: 55% fully correct, with 33% still containing missed clutter.
  • Copyright statements: 96% correct.
  • Other clutter: 84% correct.
  • Random abstracts: no over-deletion at all.
• Downstream effect: for 17 papers behind the 2025 Nobel prizes, the cosine ranking of each paper's cited references changed in every case, and the top-ranked reference changed in 4 of the 17.
• Release: mpi-inno-comp/abstract-cleaner and mpi-inno-comp/abstract_dataset, at roughly 0.5 s per abstract.

Interesting ideas

• I1. Cleanup is framed as tagging character spans rather than rewriting the text, so the original string stays intact and every deletion can be traced to its offsets. This makes removals auditable and reversible.
• I2. Errors are reported in two directions: over-deletion ("excess") and leftover clutter ("missing"). Each is measured separately for inputs with clutter and inputs without it, which exposes a model's bias toward caution or toward aggression.
• I3. When no external gold standard exists, the authors build small regex-selected stress sets, one per clutter type, plus a random control set. Each set isolates a single failure mode cheaply.
• I4. The training pool grows from a random seed by adding examples that are known to contain the target phenomenon. This keeps rare categories from being drowned out by easy negatives.
• I5. Error rates are plotted against input length. Damage to short texts is weighted as more costly because each lost token there means a larger share of lost content.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (tokens): a pre-tokenizer pass could tag boilerplate spans such as "[1-4]", "(Fig. 1)" or copyright lines and keep them out of parsing, while retaining their offsets so explanations can still point back to the raw text.
• I2 (learn): repair rules and taggers could be scored the same way, with untouched clean sentences serving as a negative control that must show zero changes.
• I3 (learn): pattern-mined probe sets could each target one construction, such as citations inside sentences or parentheticals, giving per-phenomenon pass rates alongside an unfiltered random baseline.
• I4 (learn): silver data for rare constructions could be enriched the same way, as long as the seed stays random so that negative-control rates remain honest.
• I5 (answer): accuracy on school tests and fable QA could be broken down by passage length, so that losses on short texts, where one wrong frame ruins the answer, are flagged first.