📄 Temporal Text Classification with Large Language Models

Summary

Temporal Text Classification (TTC) estimates the period in which a document was written. It helps organise large historical collections in digital humanities and information retrieval. Earlier work mostly relied on statistical classifiers built on hand-crafted stylistic and n-gram features. This paper is the first systematic test of modern LLMs on the task.

The authors evaluate three commercial systems (Claude 3.5 Sonnet, GPT-4o, Gemini 1.5 Pro) and four open-weight ones (LLaMA, Gemma 2, Mistral, Nemotron) on three corpora: two English (CLMET and the SemEval-2015 DTE data) and one Portuguese (Colonia). Texts are truncated to their first 10, 50 or 100 tokens, and models are compared under zero-shot and 5-shot prompting. The open models are also adapted with LoRA. Results are reported as Macro F1 against a TF-IDF SVM baseline.

Commercial models perform best, especially when given examples, and scores rise with input length. Fine-tuning gives the open models large gains, but they still trail the proprietary ones. A further experiment uses LLM-generated test sets to check how well models reproduce period style. The study is a useful reference for LLM-based text dating, including for languages other than English.

Key points

• Corpora:
  • Colonia: Portuguese, 16th–early 20th c., five century classes.
  • CLMET: British English 1710–1920, three 70-year periods.
  • DTE: SemEval-2015 Task 7, texts from The Spectator.
• Proprietary results: With 5-shot prompting on 100-token inputs, these models often exceed 0.90 Macro F1. Examples include GPT-4o at .96 on Colonia and .91 on DTE, and Claude at .93 on CLMET.
• Open-source results (100 tokens):
  • Zero-shot: about 0.26–0.36.
  • 5-shot: about 0.44–0.53.
  • LoRA fine-tuning: 0.61–0.78, with mean gains of +0.23 to +0.29.
  • Gemma 2 is usually the strongest.
  • Fine-tuned scores are roughly comparable to the SVM baseline (.69–.78).
• Synthetic test sets: These were generated 5-shot by GPT-4o, Claude and Gemini, then used to evaluate only the open models. Model rankings were preserved (e.g., fine-tuned Gemma 2 reached .83 on Colonia and .84 on DTE). Individual scores shifted, which suggests period style is not yet imitated faithfully.
• Model naming inconsistency: The paper is inconsistent about the LLaMA version. It is called LLaMA 3.2 in the abstract and Table 4, but listed as LLaMA 3.1 (8B) in Table 2.