PreCo: A Large-scale Dataset in Preschool Vocabulary for Coreference Resolution
What it is about
PreCo is 38K documents and 12.5M words of English reading-comprehension passages from Chinese middle- and high-school tests, chosen because their vocabulary (about 3,000 common words) resembles a preschooler's and because a small vocabulary raises the overlap between training and test, so that errors reflect coreference itself rather than unseen words. Annotation follows OntoNotes with changes: singletons are marked, generic mentions may corefer directly, non-proper noun modifiers are markable, and an event verb can corefer with a later noun («Sales [grew]… [The growth]…»). The development and test sets were annotated by two annotators and merged by a third; merging mainly raises recall, because single annotators miss distant links out of inattention. Inter-annotator CoNLL agreement is 85.3. A strong neural model reaches 81.5 F1 against an estimated human 87.9; with oracle mentions a model trained on 2.5K documents goes from 77.3 to 81.6, so most of the remaining gap is clustering.
Ideas
- I1. School-register evaluation for Mova's coreference: PreCo's dev/test passages are exactly the register of Mova's courses (simple vocabulary, reading-test stories). If the licence allows evaluation, run
coref eval(SLM trees) and the frame reader's resolver on them and report per-pronoun precision; if not, use the paper's finding (clustering, not vocabulary, is the hard part in simple texts) to prioritise pronoun linking over mention detection for lessons. - I2. Event-to-noun coreference, one of the «What next» items of
coref(event mentions): link a definite deverbal noun to an earlier verb of the same root («grew… the growth», «decided… the decision», «arrived… her arrival»). The root pairs come from Mova's own derivation rules (enmorphology, the suffix chapter), not from NOMLEX (research-only). Count candidate pairs on GUM dev and check them against GUM's verbal mentions before turning the rule on. - I3. Copy the two-pass-and-merge procedure for teacher silver on coreference: two independent teacher passes over a text, merged by rule (union of links, conflicts to a third look), since single passes lose recall on distant links. Measure the recall of each pass against the merge on a few documents.
For Mova
languages/en/35-reference-and-coreference.md §3.4 (abstract and event anaphora), §9; 17-derivation-suffixes.md (deverbal nouns). Code: coref/src/mention.rs and coref/src/sieve.rs (event-noun link), the en derivation tables. Measures: per-pronoun precision on school-register texts; precision of event-noun links on GUM dev.
Library summary
Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/D18-1016.pdf
Summary
Coreference systems are mostly benchmarked on OntoNotes. That dataset has two weaknesses. First, little vocabulary is shared between its training and test portions, so errors caused by rare words get mixed up with the harder problems of coreference. Second, it leaves singleton mentions unannotated, which makes it hard to train or evaluate mention detection on its own.
The authors built PreCo, a new English corpus drawn from reading-comprehension tests for Chinese middle and high school students. Its vocabulary is roughly that of an English-speaking preschooler, yet the texts cover open topics in varied styles. Mentions, including singletons, were labelled mostly under OntoNotes conventions. Each document went through a multi-annotator merge-and-vote procedure.
A state-of-the-art resolver still trails human agreement on PreCo. Experiments with gold mentions show that perfect mention detection would recover only a few F1 points, so mention clustering is the main bottleneck. PreCo's lexical overlap also makes error analysis cleaner, because far fewer errors involve rare words. The dataset is freely released.
Key points
- Scale and sources: PreCo has about 38K documents and 12.5M words, roughly ten times the size of OntoNotes (1.3M words). About two-thirds of the documents come from the RACE dataset. Near-duplicates were removed with a bag-of-words similarity threshold of 0.9.
- Training-test overlap: The training-test OOV rate is 0.8%, compared with 2.1% for OntoNotes. The training-dev rate of low-frequency words is 12.3%, against 34.8% for OntoNotes.
- Annotation process: 80 trained annotators took part. Three people labelled each document independently, three further annotators merged the labels pairwise, and an ensemble algorithm produced the final version.
- Model vs. human: The Peters et al. (2018) system reaches 81.5 F1 on PreCo, while human performance is 87.9.
- Mention detection vs. clustering: With EE2E-Coref trained on 2.5K documents, F1 is 77.3 using predicted mentions. It rises to 81.6 with all gold mentions, including singletons, and to 89.2 with gold non-singleton mentions only. This suggests earlier estimates overstated how much better mention detection could help.
Cite
@inproceedings{chen2018preco,
title = {{PreCo: A Large-scale Dataset in Preschool Vocabulary for Coreference Resolution}},
author = {Chen, Hong and Fan, Zhenhua and Lu, Hao and Yuille, Alan and Rong, Shu},
booktitle = {Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing},
year = {2018},
doi = {10.18653/v1/d18-1016},
eprint = {1810.09807},
archivePrefix = {arXiv},
url = {https://doi.org/10.18653/v1/d18-1016}
}References
- Steven Bird, Edward Loper, and Ewan Klein. 2009. Natural Language Processing with Python. O’Reilly Media Inc. find
- OntoNotes Guidelines. 2007. Ontonotes english coreference guidelines. find
- M. E. Peters, M. Neumann, M. Iyyer, M. Gardner, C. Clark, K. Lee, and L. Zettlemoyer. 2018. Deep contextualized word representations. ArXiv e-prints. find
- Sameer Pradhan, Alessandro Moschitti, Nianwen Xue, Olga Uryupina, and Yuchen Zhang. 2012. CoNLL-2012 shared task: Modeling multilingual unrestricted coreference in OntoNotes. In Proceedings of the Sixteenth Conference on Computational Natural Language Learning (CoNLL 2012), Jeju, Korea. find our card
- Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. Squad: 100,000+ questions for machine comprehension of text. arXiv preprint arXiv:1606.05250. ↗ our card
- Nafise Sadat Moosavi and Michael Strube. 2017. Lexical features in coreference resolution: To be used with caution. find our card
- Marc Vilain, John Burger, John Aberdeen, Dennis Connolly, and Lynette Hirschman. 1995. A model-theoretic coreference scoring scheme. In Proceedings of the 6th Conference on Message Understanding, MUC6 ’95, pages 45–52, Stroudsburg, PA, USA. Association for Computational Linguistics. find our card
- Ralph Weischedel, Eduard Hovy, Mitchell Marcus, Martha Palmer, Robert Belvin, Sameer Pradhan, Lance Ramshaw, and Nianwen Xue. 2011. Ontonotes: A large training corpus for enhanced processing. Handbook of Natural Language Processing and Machine Translation. Springer. find
- Wikipedia. 2018. Vocabulary development. [Online; accessed 18-April-2018]. find
- Sam Wiseman, Alexander M. Rush, and Stuart M. Shieber. 2016. Learning global features for coreference resolution. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 994–1004. Association for Computational Linguistics. find our card