📄 Temporal Effects on Pre-trained Models for Language Processing Tasks

Summary

Language and facts about the world shift over time, so a deployed NLP system may lose accuracy. Earlier studies usually tested on one final time slice, which cannot show whether performance declines. Agarwal and Nenkova separate two questions. Does a model trained once get worse on later data (temporal model deterioration)? Does retraining on fresher labeled data help (temporal domain adaptation)? They study four English tasks whose correct labels do not depend on time: named entity recognition, truecasing, sentiment classification and domain classification. Models are trained on each time split and tested on every later one, using summary scores and significance testing. Systems built on GloVe, GloVe+ELMo, BERT and RoBERTa, including several model sizes, often show no statistically significant decline. The same networks with randomly initialized word vectors degrade sharply. Retraining on recent data still helps, so the authors test two adaptation methods that need no new labels. An extra intermediate pre-training step proves unsuitable, while self-labeling works consistently. The paper offers practical guidance on when to retrain and how to design temporal evaluations.

Key points

• The evaluation trains on every temporal split and tests on all later ones. This avoids relying on a single test year, which can mislead: Twitter's late-2017 character-limit increase made 2018 tweets longer and richer in entities.
• With pre-trained representations, deterioration depends on the task and is often not significant. Adaptation gains from newer gold data are typically significant. This holds whether or not the pre-training data overlaps in time with the task data.
• Self-labeling improves adaptation in all settings. On NER-TTC it beats fine-tuning on new human annotations. For BERT, adding only 25% new self-labeled data to the old gold data already surpasses gold-standard fine-tuning.
• Sentiment results report F1 for the negative class, because about 80% of the examples are positive. Macro-F1 is given in an appendix.
• Hyperparameters are tuned on the oldest split and then reused for all later splits. Code and design details are published at github.com/oagarwal/temporal-effects.