第一届古代汉语分词和词性标注国际评测

Summary

Classical Chinese texts survive in vast numbers, but automatic processing has lagged. There are few large lexicons and annotated corpora, and modern Chinese tools do not handle the older vocabulary and grammar. To give the field a shared benchmark, the authors organised EvaHan2022, the first international shared task on joint word segmentation and POS tagging for Ancient Chinese. It was held at the second LT4HALA workshop at LREC 2022 in Marseille.

Participants received a hand-corrected portion of the Zuozhuan for training and were evaluated on two test sets:

• TestA: the remaining part of the same book.
• TestB: newly annotated passages from other historical works.

Systems were ranked by F1 in two settings. A closed track limited resources to the given data and SikuRoBERTa, while an open track allowed external resources.

Fourteen teams took part. The strongest systems, all built on pre-trained deep models, reached roughly 96% F1 for segmentation and 92% for tagging on TestA, with lower scores on TestB. Fudan University (FDU) ranked first overall. Out-of-vocabulary words remain the main weakness.

Key points

• Data: Traditional-character text from the Zuozhuan, Shiji and Zizhi Tongjian, punctuated, segmented and POS-tagged, then checked twice by human annotators. Annotation follows Nanjing Normal University's pre-Qin Chinese guidelines with 21 POS tags.

• Splits:

  • Training uses the first ten volumes of the Zuozhuan (about 5/6 of the book).
  • TestA covers the last two volumes.
  • TestB is unreleased material from the Shiji (Zhou and Qin annals) and the Zizhi Tongjian (Zhou records).
  • Training data and TestA are distributed through LDC (LDC2017T14).

• Tracks: The closed track allowed only the Zuozhuan training set plus SikuRoBERTa, a RoBERTa model pre-trained on about 500 million characters of the Siku Quanshu. The open track was unrestricted, except that manual correction was banned in both.

• Results (14 teams, 55 runs; segmentation / tagging F1):

  Test set	Closed track	Open track
  TestA	96.16 / 92.05	96.34 / 92.56
  TestB	93.64 / 87.77	95.03 / 89.47

  A tag counted as correct only if the word boundaries were also correct.

• OOV problem:

  • Unseen words make up 6.80% of tokens in TestA and 12.57% in TestB.
  • By word type, the shares are 38.72% and 51.13%.
  • Even the winning FDU system scored only about 86% (segmentation) and 80% (tagging) F1 on these words.
  • A forward-maximum-matching "topline" built from the test vocabulary reached about 98% segmentation F1.