Elamur Research⌕
☰
home / papers / 2020 / WIKIR: A Python Toolkit for Building a Large-scale Wikipedia-based English Information Retrieval Dataset

WIKIR: A Python Toolkit for Building a Large-scale Wikipedia-based English Information Retrieval Dataset

Paper · 2020 · Jibril Frej, Didier Schwab and Jean-Pierre Chevallet

Summary

Neural ranking models for ad-hoc information retrieval need many more labelled queries than public English collections such as Robust04 or ClueWeb09 provide. As a result, recent models have often been trained on private search-engine logs, which makes the work hard to reproduce. The authors respond with WIKIR, an open-source Python toolkit that turns an English Wikipedia dump into a complete IR dataset of documents, queries and relevance judgements, with no manual annotation.

They first give a general recipe. Any document collection can be used if three conditions hold:

  • every document has a topic;
  • topics can be detected automatically;
  • each topic can be given a short description.

For Wikipedia, an article's topics are its own subject plus the subjects of articles whose opening sentence links to it. Queries are built from article titles or first sentences. With the toolkit they release wikIR78k and wikIRS78k, each holding 78,628 queries and 3,060,191 query–relevant-document pairs, along with scripts for training and evaluating deep text-matching models. The result is a free, reproducible, large benchmark for neural IR.

Key points

  • Public English IR collections listed by the authors have at most 1,692 labelled queries (MQ2007). Yahoo! LETOR is larger but ships only feature vectors, not raw text. The Chinese Sogou-QCL (≈500k queries) is described as the only large public alternative.
  • Relevance is graded on three levels:
    • 2: the query and the document come from the same article.
    • 1: the document's article links to the query's article in its first sentence (e.g. Autism → Developmental disorder).
    • 0: otherwise.
  • Pipeline: WikiExtractor with links preserved. The title line is removed from each document, link targets are stripped while the anchor text is kept, non-alphanumeric characters are deleted, and text is lowercased by default.
  • Queries are kept only if they have at least 5 relevant documents (default) and at most 10 words.
  • Queries and qrels are split randomly into train, validation and test sets. The document pool is shared across all splits.

Interesting ideas

  • I1. Stating three checkable properties (Existence, Identifiability, Describability) turns "can this resource become labelled data for free?" into a clear test that can be applied to any collection.
  • I2. A link in an article's opening sentence signals a central topic, because Wikipedia first sentences usually define the subject ("Autism is a developmental disorder…").
  • I3. The title is deleted from the matching document so the system cannot win by spotting the query at the start of the text; this is a deliberate block on a trivial shortcut.
  • I4. Relevance is graded by structural distance (same article versus linked article) instead of a binary yes/no, which gives richer supervision at no extra cost.
  • I5. Only queries are split into train and test, while the document collection stays shared, which matches how retrieval works in real use.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Use the same checklist to decide which corpora can yield silver supervision automatically, for example glossed dictionaries or linked encyclopedic text.
  • I2 (world): Mining first-sentence links in Wikipedia could add genus/category edges to the ontology, alongside WordNet and Wikidata, with the sentence itself kept as an explainable source.
  • I3 (learn): When building evaluation items, strip surface copies of the answer from the input and keep this as a negative-control check, so that string matching cannot pass for understanding.
  • I4 (answer): QA on fables could score supporting sentences in tiers, such as directly stating the fact versus mentioning a linked entity, which makes evidence retrieval easier to explain.
  • I5 (learn): Evaluate readers by holding out questions over a common set of texts, so that generalisation to new questions is measured separately from memorising texts.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🔬Centre National de la Recherche ScientifiqueParis · France104 works · 60,877 citations🎓Université Grenoble AlpesSaint-Martin-d'Hères · France7 works · 6,195 citations🔬Laboratoire d'Informatique de GrenobleGrenoble · France8 works · 522 citations🎓Institut polytechnique de GrenobleGrenoble · France4 works · 163 citations

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 9,086 worksArtificial Intelligence · 8,889 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 worksInformation Systems · 523 works

🧭 Topics 2Research subjects and languages the work is about.English · 1,373 worksInformation retrieval & search · 145 works

Cite

Cite as Jibril Frej, Didier Schwab and Jean-Pierre Chevallet. 2020. WIKIR: A Python Toolkit for Building a Large-scale Wikipedia-based English Information Retrieval Dataset. Proceedings of the Twelfth Language Resources and Evaluation Conference.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{frej2020wikir,
  title = {{WIKIR: A Python Toolkit for Building a Large-scale Wikipedia-based English Information Retrieval Dataset}},
  author = {Jibril Frej, Didier Schwab and Jean-Pierre Chevallet},
  booktitle = {Proceedings of the Twelfth Language Resources and Evaluation Conference},
  year = {2020},
  url = {https://aclanthology.org/2020.lrec-1.237/}
}

References

30 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Baeza-Yates, R. A. and Ribeiro-Neto, B. A. (1999). Modern Information Retrieval. ACM Press / Addison-Wesley. find
  2. Chapelle, O. and Chang, Y. (2011). Yahoo! learning to rank challenge overview. In Proceedings of the Yahoo! Learning to Rank Challenge, held at ICML 2010, Haifa, Israel, June 25, 2010, pages 1–24. find
  3. Dai, Z., Xiong, C., Callan, J., and Liu, Z. (2018). Convolutional neural networks for soft-matching n-grams in ad-hoc search. In Proceedings of the Eleventh ACM International Conference on Web Search and Data Mining, WSDM ’18, pages 126–134, New York, NY, USA. ACM. find
  4. Dehghani, M., Zamani, H., Severyn, A., Kamps, J., and Croft, W. B. (2017). Neural ranking models with weak supervision. In Proceedings of The 40th International ACM SIGIR Conference on Research and Development in Information Retrieval. ↗
  5. Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. (2019). BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota, June. Association for Computational Linguistics. ↗
  6. Fan, Y., Guo, J., Lan, Y., Xu, J., Zhai, C., and Cheng, X. (2018). Modeling diverse relevance patterns in ad-hoc retrieval. In The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval, SIGIR 2018, Ann Arbor, MI, USA, July 08-12, 2018, pages 375–384. find
  7. Fuhr, N. (2018). Some common mistakes in ir evaluation, and how they can be avoided. SIGIR Forum, 51(3):32– 41, February. ↗
  8. Guo, J., Fan, Y., Ai, Q., and Croft, W. B. (2016). A deep relevance matching model for ad-hoc retrieval. In Proceedings of the 25th ACM International on Conference on Information and Knowledge Management, CIKM ’16, pages 55–64, New York, NY, USA. ACM. find
  9. Guo, J., Fan, Y., Pang, L., Yang, L., Ai, Q., Zamani, H., Wu, C., Croft, W. B., and Cheng, X. (2019a). A deep look into neural ranking models for information retrieval. CoRR, abs/1903.06902. https://www.ncbi.nlm.nih.gov/pmc/LIG, Université Grenoble-Alpes 12 School of Engineering and Physical Sciences, Heriot-Watt University, Edinburgh Guo, J., Yixing, F., Xiang, J., and Xueqi, C. (2019b). Matchzoo: A learning, practicing, and developing system for neural text matching. In Proceedings of the 42Nd International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR’19, pages 1297–1300, New York, NY, USA. ACM. ↗
  10. Hu, B., Lu, Z., Li, H., and Chen, Q. (2014). Convolutional neural network architectures for matching natural language sentences. In Advances in Neural Information Processing Systems 27: Annual Conference on Neural Information Processing Systems 2014, December 8-13 2014, Montreal, Quebec, Canada, pages 2042–2050. ↗
  11. Järvelin, K. and Kekäläinen, J. (2002). Cumulated gain-based evaluation of IR techniques. ACM Trans. Inf. Syst., 20(4):422–446. ↗
  12. Kingma, D. P. and Ba, J. (2015). Adam: A method for stochastic optimization. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings. ↗
  13. Loper, E. and Bird, S. (2002). NLTK: the natural language toolkit. CoRR, cs.CL/0205028. ↗
  14. Mitra, B., Diaz, F., and Craswell, N. (2017). Learning to match using local and distributed representations of text for web search. In Proceedings of the 26th International Conference on World Wide Web, WWW ’17, pages 1291–1299, Republic and Canton of Geneva, Switzerland. International World Wide Web Conferences Steering Committee. find
  15. Mizzaro, S. (1997). Relevance: The whole history. JASIS, 48(9):810–832. ↗
  16. Pang, L., Lan, Y., Guo, J., Xu, J., Wan, S., and Cheng, X. (2016). Text matching as image recognition. In Thirtieth AAAI Conference on Artificial Intelligence. find
  17. Pang, L., Lan, Y., Guo, J., Xu, J., Xu, J., and Cheng, X. (2017). Deeprank: A new deep architecture for relevance ranking in information retrieval. In Proceedings of the 2017 ACM on Conference on Information and Knowledge Management, CIKM ’17, pages 257–266, New York, NY, USA. ACM. find
  18. Pennington, J., Socher, R., and Manning, C. (2014). Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 1532– 1543.
  19. Porter, M. F. (2001). Snowball: A language for stemming algorithms. find
  20. Robertson, S. E. and Walker, S. (1994). Some simple effective approximations to the 2-poisson model for probabilistic weighted retrieval. In Proceedings of the 17th Annual International ACM-SIGIR Conference on Research and Development in Information Retrieval. Dublin, Ireland, 3-6 July 1994 (Special Issue of the SI-GIR Forum), pages 232–241. ↗
  21. Sanderson, M. (2010). Christopher d. manning, prabhakar raghavan, hinrich schütze, Introduction to Information Retrieval, cambridge university press 2008. ISBN-13 978-0-521-86571-5, xxi + 482 pages. Natural Language Engineering, 16(1):100–103. find
  22. Sasaki, S., Sun, S., Schamoni, S., Duh, K., and Inui, K. (2018). Cross-lingual learning-to-rank with shared representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 458–463, New Orleans, Louisiana, June. Association for Computational Linguistics. find
  23. Schamoni, S., Hieber, F., Sokolov, A., and Riezler, S. (2014). Learning translational and knowledge-based similarities from relevance rankings for cross-language retrieval. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 488–494. find
  24. Schütze, H., Manning, C. D., and Raghavan, P. (2008). Introduction to information retrieval. In Proceedings of the international communication of association for computing machinery conference, page 260. ↗
  25. Urbano, J., Marrero, M., and Martı́n, D. (2013). A comparison of the optimality of statistical significance tests for information retrieval evaluation. In The 36th International ACM SIGIR conference on research and development in Information Retrieval, SIGIR ’13, Dublin, Ireland - July 28 - August 01, 2013, pages 925–928. find
  26. Xiong, C., Dai, Z., Callan, J., Liu, Z., and Power, R. (2017). End-to-end neural ad-hoc ranking with kernel pooling. In Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR ’17, pages 55–64, New York, NY, USA. ACM. find
  27. Yang, W., Lu, K., Yang, P., and Lin, J. (2019a). Critically examining the ”neural hype”: Weak baselines and the additivity of effectiveness gains from neural ranking models. In Proceedings of the 42nd International ACM SI-GIR Conference on Research and Development in Information Retrieval, SIGIR 2019, Paris, France, July 21-25, 2019, pages 1129–1132. ↗
  28. Yang, Z., Dai, Z., Yang, Y., Carbonell, J. G., Salakhutdinov, R., and Le, Q. V. (2019b). Xlnet: Generalized autoregressive pretraining for language understanding. CoRR, abs/1906.08237. ↗
  29. Zamani, H., Dehghani, M., Croft, W. B., Learned-Miller, E., and Kamps, J. (2018). From neural re-ranking to neural ranking: Learning a sparse representation for inverted indexing. In Proceedings of the 27th ACM International Conference on Information and Knowledge Management, pages 497–506. ACM. find
  30. Zheng, Y., Fan, Z., Liu, Y., Luo, C., Zhang, M., and Ma, S. (2018). Sogou-qcl: A new dataset with click relevance label. In The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval, SIGIR ’18, pages 1117–1120, New York, NY, USA. ACM. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/frej-2020-wikir-python-toolkit-building-large/