Elamur Research⌕
☰
home / papers / 2018 / English Out-of-Vocabulary Lexical Evaluation Task

English Out-of-Vocabulary Lexical Evaluation Task

Paper · 2018 · Han Wang, Ye Wang, Xinxiang Zhang et al.

Summary

The paper proposes a benchmark for words a model has never seen: out-of-vocabulary (OOV) items that occur only in test material, so their meaning must come from surrounding text. The authors argue that earlier unknown-noun supersense taggers relied on WordNet, hand rules and features such as POS tags or chunking, so they fail on words that WordNet lacks. They built a dataset from Wikipedia dumps. Each item pairs a rare word with a single descriptive context. They kept five readable domains (Greek mythology, locations, animals, plants, technology) and dropped opaque chemistry and medical entries. The benchmark has two tasks. Task 1 assigns the OOV word to its Wikipedia category and is scored by rank. Task 2 predicts human-annotated attributes and is scored by top-K hits. As unsupervised baselines they use Word2Vec, a pretrained FastText model and Word2GM, and they compare variants with and without word sense disambiguation (WSD). The extracted text omits the results section, so no baseline scores are reported here. The value of the work is a public, knowledge-free test of how well a system reads new words from context.

Key points

  • Data source and ground truth. Items come from Wikipedia dumps, and Wikipedia's category tree supplies the labels. Most selected OOV words are absent from WordNet.
  • Selection. Very frequent and very rare words were filtered out. Each OOV word may appear only in its given context.
  • Annotation. Five fluent, non-native English graduate students served as annotators. Each context went to three random annotators, and only the attributes they shared were kept, up to five per word.
  • Training corpus. The baselines were trained on about 37M tokens with about 340K unique types. A minimum count of 5 left a vocabulary of about 75K.
  • Models. They trained a 50-dimensional skip-gram Word2Vec and compared it with pretrained FastText vectors (16B tokens, 300 dimensions, 1M words) and with Word2GM, which uses Gaussian mixture embeddings.
  • Availability. The dataset is released at github.com/hwangtamu/OOVLexical.

Interesting ideas

  • I1. Definitional test items: each test word is paired with a single sentence that defines it, so the only way to understand the word is to read that context; this cleanly isolates the "learn a word from one description" skill.
  • I2. Keep only agreed labels: three independent annotators label each item, and only the attributes they agree on survive, which trades recall for reliable gold labels.
  • I3. Drop unreadable domains: domains that even human annotators could not understand (chemistry, medicine) were removed, so the benchmark measures context reading rather than specialist knowledge.
  • I4. Two metrics for two label types: closed category labels are scored by rank, while open attribute sets are scored by whether any top-K prediction hits; the gold attributes may even be words absent from the context (adware, spyware for a malware term).
  • I5. Category tree as free gold: Wikipedia's category hierarchy provides class labels for words that WordNet does not cover.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (world): copular definitions such as "X is a genus of … plants in the pea family" could be parsed in the UD tree to create a typed entity with attributes. This would give unknown nouns an explainable ontology entry without any embeddings.
  • I2 (learn): this mirrors keeping silver labels only when several witness parsers or LLM teacher tiers agree, and the same intersection rule could be applied to frame and attribute annotations.
  • I3 (learn): evaluation sets could be filtered by a human-readability check, so failures point to real reading errors rather than missing jargon.
  • I4 (answer): reader tests on fables could combine a ranked closed-choice score with an open top-K hit score for inferred properties, rewarding inferences that go beyond surface words.
  • I5 (tokens): holding out words seen only in test material and checking the tagger's and lemmatizer's guesses against Wikidata/Wikipedia classes would give a systematic unknown-word evaluation.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Texas A&M UniversityCollege Station · United States13 works · 1,683 citations🎓Southern Methodist UniversityDallas · United States3 works · 63 citations🎓Wuhan University of TechnologyWuhan · China2 works · 84 citations

📚 Shelves 4 Computer Science · 9,086 works Artificial Intelligence · 8,889 works Arts and Humanities · 9,296 works Language and Linguistics · 9,206 works

🧭 Topics 2 English · 1,373 works Evaluation & benchmarks · 685 works

Cite

Cite as Han Wang, Ye Wang, Xinxiang Zhang, Mi Lu, Yoonsuck Choe and Jingjing Cao. 2018. English Out-of-Vocabulary Lexical Evaluation Task. arXiv:1804.04242.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1804.04242 · arXiv 1804.04242

@misc{wang2018english,
  title = {{English Out-of-Vocabulary Lexical Evaluation Task}},
  author = {Han Wang, Ye Wang, Xinxiang Zhang, Mi Lu, Yoonsuck Choe and Jingjing Cao},
  year = {2018},
  doi = {10.48550/arxiv.1804.04242},
  eprint = {1804.04242},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1804.04242}
}

References

29 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ziming Zhang and Venkatesh Saligrama, “Zero-shot learning via semantic similarity embedding,” in Proceedings of the IEEE International Conference on Computer Vision, 2015, pp. 4166–4174. find
  2. Ye Wang and Mi Lu, “A self-adaptive algorithm to defeat text-based CAPTCHA,” in Industrial Technology (ICIT), 2016 IEEE International Conference on. IEEE, 2016, pp. 720–725. find
  3. Ye Wang, Yuanjiang Huang, Wu Zheng, Zhi Zhou, Debin Liu, and Mi Lu, “Combining convolutional neural network and self-adaptive algorithm to defeat synthetic multi-digit text-based CAPTCHA,” in Industrial Technology (ICIT), 2017 IEEE International Conference on. IEEE, 2017, pp. 980–985. find
  4. Advaith Siddharthan, “Syntactic simplification and text cohesion,” Research on Language & Computation, vol. 4, no. 1, pp. 77–109, 2006. ↗
  5. Yue Zhang and Xinxiang Zhang, “effective real-scenario video copy detection,” in 2016 23rd International Conference on Pattern Recognition (ICPR). IEEE, 2016, pp. 3951–3956. find
  6. Zhengcong Yin, Chong Zhang, Daniel W Goldberg, and Sathya Prasad, “An nlp-based question answeringframework for spatio-temporal analysis and visualization,” in ICGDA ’19: Proceedings of the International Conference on Geoinformatics and Data Analysis, New York, NY, USA, 2019, ACM. find
  7. Ido Dagan, Oren Glickman, and Bernardo Magnini, “The pascal recognising textual entailment challenge,” in Machine learning challenges. evaluating predictive uncertainty, visual object classification, and recognising tectual entailment, pp. 177–190. Springer, 2006. find
  8. James R Curran, “Supersense tagging of unknown nouns using semantic similarity,” in Proceedings of the 43rd Annual Meeting on Association for Computational Linguistics. Association for Computational Linguistics, 2005, pp. 26–33. our card
  9. Massimiliano Ciaramita and Mark Johnson, “Supersense tagging of unknown nouns in wordnet,” in Proceedings of the 2003 conference on Empirical methods in natural language processing. Association for Computational Linguistics, 2003, pp. 168–175. our card
  10. Or Biran, Samuel Brody, and Noémie Elhadad, “Putting it simply: a context-aware approach to lexical simplification,” in Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies: short papers-Volume 2. Association for Computational Linguistics, 2011, pp. 496–501. ↗ our card
  11. Lucia Specia, Sujay Kumar Jauhar, and Rada Mihalcea, “Semeval-2012 task 1: English lexical simplification,” in Proceedings of the First Joint Conference on Lexical and Computational Semantics-Volume 1: Proceedings of the main conference and the shared task, and Volume 2: Proceedings of the Sixth International Workshop on Semantic Evaluation. Association for Computational Linguistics, 2012, pp. 347– 355. our card
  12. Diana McCarthy and Roberto Navigli, “Semeval-2007 task 10: English lexical substitution task,” in Proceedings of the 4th International Workshop on Semantic Evaluations. Association for Computational Linguistics, 2007, pp. 48–53. our card
  13. Christopher D Manning, Prabhakar Raghavan, and Hinrich Schütze, “Boolean retrieval,” Introduction to information retrieval, pp. 1–18, 2008. find
  14. DE Rumelhart, “David e. rumelhart, geoffrey e. hinton, and ronald j. williams,” Nature, vol. 323, pp. 533–536, 1986. find
  15. Yoshua Bengio, Réjean Ducharme, Pascal Vincent, and Christian Jauvin, “A neural probabilistic language model,” Journal of machine learning research, vol. 3, no. Feb, pp. 1137–1155, 2003. our card
  16. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean, “Distributed representations of words and phrases and their compositionality,” in Advances in neural information processing systems, 2013, pp. 3111–3119. ↗ our card
  17. Ye Wang, Zhi Zhou, Shan Jin, Debin Liu, and Mi Lu, “Comparisons and selections of features and classifiers for short text classification,” in IOP Conference Series: Materials Science and Engineering. IOP Publishing, 2017, vol. 261, p. 012018. find
  18. Joseph Reisinger and Raymond J Mooney, “Multi-prototype vector-space models of word meaning,” in Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics. Association for Computational Linguistics, 2010, pp. 109–117. our card
  19. Eric H Huang, Richard Socher, Christopher D Manning, and Andrew Y Ng, “Improving word representations via global context and multiple word prototypes,” in Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics: Long Papers-Volume 1. Association for Computational Linguistics, 2012, pp. 873–882. our card
  20. Fei Tian, Hanjun Dai, Jiang Bian, Bin Gao, Rui Zhang, Enhong Chen, and Tie-Yan Liu, “A probabilistic model for learning multi-prototype word embeddings.,” in COLING, 2014, pp. 151–160. find
  21. Luke Vilnis and Andrew McCallum, “Word representations via gaussian embedding,” arXiv preprint arXiv:1412.6623, 2014. ↗
  22. Ye Wang, Han Wang, Xinxiang Zhang, Theodora Chaspari, Yoonsuck Choe, and Mi Lu, “An attention-aware bidirectional multi-residual recurrent neural network (abmrnn): A study about better short-term text classification,” in ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2019, pp. 3582–3586. find
  23. Zhengcong Yin, Andong Ma, and Daniel W Goldberg, “A deep learning approach for rooftop geocoding,” Transactions in GIS, 2019. ↗
  24. Ye Wang and Mi Lu, “An Optimized System to Solve Text-based CAPTCHA,” arXiv preprint arXiv:1806.07202, 2018. ↗
  25. Hao Wu, Xinxiang Zhang, Brett Story, and Dinesh Rajan, “Accurate vehicle detection using multi-camera data fusion and machine learning,” in ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2019, pp. 3767–3771. find
  26. Chong Zhang, Zhengcong Yin, Peng Gao, and Sathya Prasad, “A visual analytics approach to explorationof hotels in overlaid drive-time polygons of attractions,” in International Symposium On Web and Wireless Geographical Information Systems. 2019, Springer International Publishing. find
  27. Ben Athiwaratkun and Andrew Gordon Wilson, “Multimodal word distributions,” in Conference of the Association for Computational Linguistics (ACL), 2017. find
  28. Yoav Goldberg and Omer Levy, “Word2vec explained: Deriving mikolov et al.’s negative-sampling word-embedding method,” 2014. find
  29. Tony Jebara, Risi Kondor, and Andrew Howard, “Probability product kernels,” Journal of Machine Learning Research, vol. 5, no. Jul, pp. 819–844, 2004. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/wang-2018-english-out-vocabulary-lexical-evaluation/