Elamur Research⌕
☰
home / papers / 2023 / CQE: A Comprehensive Quantity Extractor

CQE: A Comprehensive Quantity Extractor

Summary

The authors address the task of automatically extracting quantitative information from text. Until now, it has rarely been studied in its own right: it usually serves only as an auxiliary step in search systems or in textual entailment tasks, and nobody has separately evaluated the quality of the extractor itself. The authors therefore propose CQE. This system represents each quantity as a quadruple: value, unit, change, and associated concept. Values are normalized, and units can be physical quantities and currencies as well as arbitrary nouns. Change means a bound or a trend, such as an increase or a decrease. The concept is what the number refers to. The method relies on rules built on dependency parsing and POS tags (SpaCy), a custom tokenizer, and a unit dictionary. Separate classifiers disambiguate ambiguous units. For evaluation, the authors create a new corpus, NewsQuant, built from news texts. CQE outperforms open-source libraries and GPT-3 at detecting values, units, and changes. It is also the first system to identify concepts. The work gives researchers an open baseline tool and the first full-fledged benchmark for this task.

Key points

  • Values, units, and changes are detected by 61 rules built on the dependency tree and POS tags. Change has six categories: =, ∼, >, < (following the scheme of Roy et al.) and two new ones, up and down.
  • NewsQuant contains news from the economy, sports, technology, cars, science, and companies sections. The comparison was also carried out on the R-Txt dataset with four quantity types. There, CQE falls behind only on temperature, and this difference is not statistically significant.
  • Baseline systems for comparison: Grobid-quantities, Illinois Quantifier, Recognizers-Text, Quantulum3 (Q3), and GPT-3. GPT-3 hallucinates a lot and normalizes units poorly.
  • Ambiguous units (e.g., "pound" as a weight or as a currency) are disambiguated by 18 classifiers trained on data generated by ChatGPT. The weighted F1 is 88.1 versus 54.46 for Q3.
  • Finding concepts is noticeably harder than finding value–unit pairs. The method is designed primarily for high recall, so the scores under strict and relaxed match differ substantially. Code and data are open: github.com/vivkaz/CQE.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Part-of-speech tagging

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2023.emnlp-main.793 · arXiv 2305.08853

@inproceedings{almasian2023cqe,
  title = {{CQE: A Comprehensive Quantity Extractor}},
  author = {Almasian, Satya and Kazakova, Vivian and Göldner, Philipp and Gertz, Michael},
  booktitle = {Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing},
  year = {2023},
  doi = {10.18653/v1/2023.emnlp-main.793},
  eprint = {2305.08853},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/2023.emnlp-main.793}
}

References

22 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Satya Almasian, Milena Bruseva, and Michael Gertz. 2022. Qfinder: A framework for quantity-centric ranking. In SIGIR ’22: The 45th International ACM SIGIR Conference on Research and Development in Information Retrieval, Madrid, Spain, July 11 - 15, 2022, pages 3272–3277. ACM. find
  2. Somnath Banerjee, Soumen Chakrabarti, and Ganesh Ramakrishnan. 2009. Learning to Rank for Quantity Consensus Queries. In Proceedings of the 32nd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2009, Boston, MA, USA, July 19-23, 2009, pages 243–25… find
  3. Soumia Lilia Berrahou, Patrice Buche, Juliette Dibie, and Mathieu Roche. 2017. Xart: Discovery of correlated arguments of n-ary relations in text. Expert Syst. Appl., 73:115–124. find
  4. Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter,… find
  5. Sanxing Chen, Yongqiang Chen, and Börje F Karlsson. 2023. Dataset and baseline system for multi-lingual extraction and normalization of temporal and numerical expressions. arXiv preprint arXiv:2303.18103. — [arXiv:2303.18103](https://arxiv.org/abs/2303.18103) find
  6. Jacob Cohen. 1960. A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20(1):37–46. find
  7. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,… find
  8. Luca Foppiano, Laurent Romary, Masashi Ishii, and Mikiko Tanifuji. 2019. Automatic identification and normalisation of physical measurements in scientific literature. In Proceedings of the ACM Symposium on Document Engineering 2019, Berlin, Germany, September 23-26, 2019, pages 24:1–24:4. ACM. find
  9. Kenneth D. Forbus. 1984. Qualitative process theory. Artif. Intell., 24(1-3):85–168. find
  10. Vinh Thinh Ho, Yusra Ibrahim, Koninika Pal, Klaus Berberich, and Gerhard Weikum. 2019. Qsearch: Answering quantity queries from text. In The Semantic Web - ISWC 2019 - 18th International Semantic Web Conference, Auckland, New Zealand, October 26-30, 2019, Proceedings, Part I, volume 11778 of Lecture… find
  11. Wenhao Huang, Zijia Lin, Chris McConnell, and Börje F. Karlsson. 2017. Recognizers-Text: Recognition and Resolution of Numbers, Units, and Date/time Entities Expressed Across Multiple Languages. find
  12. Philipp Koehn. 2004. Statistical significance tests for machine translation evaluation. In Proceedings of the 2004 Conference on Empirical Methods in Natural Language Processing , EMNLP 2004, A meeting of SIGDAT, a Special Interest Group of the ACL, held in conjunction with ACL 2004, 25-26 July 2004… find
  13. Klaus Krippendorff. 2004. Content Analysis: An Introduction to Its Methodology (2nd ed.). Sage publications. find
  14. Maxime Lefrançois and Antoine Zimmermann. 2018. The unified code for units of measure in RDF: cdt: ucum and other UCUM datatypes. In The Semantic Web: ESWC 2018 Satellite Events - ESWC 2018 Satellite Events, Heraklion, Crete, Greece, June 3-7, 2018, Revised Selected Papers, volume 11155 of Lecture N… find
  15. Tongliang Li, Lei Fang, Jian-Guang Lou, Zhoujun Li, and Dongmei Zhang. 2021. AnaSearch: Extract, Retrieve and Visualize Structured Results from Unstructured Text for Analytical Queries. In WSDM ’21, The Fourteenth ACM International Conference on Web Search and Data Mining, Virtual Event, Israel, Mar… find
  16. Arun S. Maiya, Dale Visser, and Andrew Wan. 2015. Mining Measured Information from Text. In Proceedings of the 38th International SIGIR Conference on Research and Development in Information Retrieval, pages 899–902. ACM. find
  17. Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. GloVe: Global vectors for word representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1532–1543, Doha, Qatar. Association for Computational Linguistics. find
  18. Stefan Riezler and John T. Maxwell. 2005. On some pitfalls in automatic evaluation and significance testing for MT. In Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization, pages 57–64, Ann Arbor, Michigan. Association for Comput… find
  19. Hajo Rijgersberg, Mark van Assem, and Jan L. Top. 2013. Ontology of units of measure and related concepts. Semantic Web, 4(1):3–13. find
  20. Subhro Roy, Tim Vieira, and Dan Roth. 2015. Reasoning about quantities in natural language. Transactions of the Association for Computational Linguistics, 3:1–13. find
  21. Sunita Sarawagi and Soumen Chakrabarti. 2014. Open-domain Quantity Queries on Web Tables: Annotation, Response, and Consensus Models. In The 20th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD ’14, New York, NY, USA - August 24 - 27, 2014, pages 711–720. ACM. find
  22. Naushad UzZaman, Hector Llorens, Leon Derczynski, James Allen, Marc Verhagen, and James Pustejovsky. 2013. Semeval-2013 task 1: Tempeval-3: Evaluating Time Expressions, Events, and Temporal Relations. In Second Joint Conference on Lexical and Computational Semantics (* SEM), Volume 2: Proceedings of… find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/papers/almasian-2023-cqe-comprehensive-quantity-extractor/