Elamur Research⌕
☰
home / papers / 2023 / IDRISI-D: Arabic and English Datasets and Benchmarks for Location Mention Disambiguation over Disaster Microblogs

IDRISI-D: Arabic and English Datasets and Benchmarks for Location Mention Disambiguation over Disaster Microblogs

Paper · 2023 · Reem Suwaileh, Tamer Elsayed and Muhammad Imran

Summary

Disaster-related posts on X often mention places, but users rarely enable geotagging. Response teams therefore need tools that link a textual location mention to the correct entry in a gazetteer. This task, Location Mention Disambiguation (LMD), has very few public microblog datasets in English and none in Arabic. The authors build IDRISI-D on top of their earlier location-recognition datasets, IDRISI-RE and IDRISI-RA. The result is two human-annotated resources: IDRISI-DE (English, around 5.5K posts and roughly 9.6–9.7K mentions) and IDRISI-DA (Arabic, 2,869 posts and 3,893 mentions). Annotators also recorded which contextual signals helped them resolve each mention, such as hashtags, replies, URLs or other mentions. The authors argue that binary-style accuracy and F-scores fit poorly with what is really a ranking problem. They propose a lenient hierarchical evaluation that scores systems with MRR@r at several granularity levels, from country down to street. In the benchmarks, plain exact matching with Nominatim is a strong baseline, and a BERT-based model (BERT_LMD) is promising. The datasets and the scoring script are released under CC BY 4.0.

Key points

  • IDRISI-DA is presented as the first public Arabic LMD dataset, with 2,869 posts, 3,893 mentions and 763 unique mentions.
  • IDRISI-DE is described as the largest human-labeled English LMD microblog set. The paper gives slightly inconsistent figures: 5,591 or 5,461 posts, and 9,685 or 9,586 mentions with 1,601 unique.
  • The source data come from IDRISI-RE (sampled from HumAID, 19 events) and IDRISI-RA (sampled from Kawarith, Arab-region events). For English, 8,224 posts with 11,023 mentions were sampled so that location-type proportions were kept while fine-grained types were still covered.
  • The task allows abstention: a system must declare a mention unresolvable when no gazetteer entry fits.
  • Documented difficulties include:
    • nicknames ("Detroit of India");
    • abbreviations ("T. Nagar");
    • misspellings and shortcuts ("st", "@" for "at");
    • ignored capitalization;
    • dialectal Arabic forms.

Interesting ideas

  • I1. The scoring is hierarchical and lenient: a prediction is credited at whatever administrative level it gets right (country, city, street), and MRR@r is applied to ranked candidates, which replaces arbitrary distance cut-offs such as 161 km.
  • I2. Annotators labeled which signals actually helped them disambiguate each mention, and this tells developers which features are worth modelling.
  • I3. The task explicitly requires abstaining when a mention has no matching toponym, rather than forcing a link.
  • I4. The catalogue of user-generated noise (shortcuts, abbreviations, dropped capitals, dialect forms) gives a concrete checklist of normalization problems.
  • I5. The strong showing of simple exact matching (Nominatim) against learned models shows the value of a transparent lookup baseline.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Evaluation could grant partial credit when a resolved entity is correct at a coarser node of the ontology, which shows where resolution fails instead of giving a single pass/fail.
  • I2 (world): When entity linking builds the world of a text, each link could record the evidence that justified it (neighbouring mention, event context), which keeps the step explainable.
  • I3 (world): Entity grounding against Wikidata/WordNet should treat "unlinkable" as a valid result, so that temporary or invented referents do not get attached to wrong knowledge-graph nodes.
  • I4 (tokens): The tokenizer and lemmatizer could expand forms like "st", "@" for "at" and missing capitalization before tagging, so that proper-noun and location detection survive informal text.
  • I5 (learn): Every learned resolver could be reported alongside a plain dictionary-lookup control, acting as a negative control that shows whether learning adds anything.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Hamad bin Khalifa UniversityDoha · Qatar141 works · 14,175 citations🎓Qatar UniversityDoha · Qatar48 works · 1,026 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksSocial Sciences · 1,772 worksGeography, Planning and Development · 6 works

🧭 Topics 3Research subjects and languages the work is about.English · 2,263 worksSemitic languages · 2,908 worksEvaluation & benchmarks · 1,220 works

Cite

Cite as Reem Suwaileh, Tamer Elsayed and Muhammad Imran. 2023. IDRISI-D: Arabic and English Datasets and Benchmarks for Location Mention Disambiguation over Disaster Microblogs. Proceedings of ArabicNLP 2023.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2023.arabicnlp-1.14

@inproceedings{suwaileh2023idrisi,
  title = {{IDRISI-D: Arabic and English Datasets and Benchmarks for Location Mention Disambiguation over Disaster Microblogs}},
  author = {Reem Suwaileh, Tamer Elsayed and Muhammad Imran},
  booktitle = {Proceedings of ArabicNLP 2023},
  year = {2023},
  doi = {10.18653/v1/2023.arabicnlp-1.14},
  url = {https://doi.org/10.18653/v1/2023.arabicnlp-1.14}
}

References

31 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Muhammad Abdul-Mageed, AbdelRahim Elmadany, and El Moatez Billah Nagoudi. 2021. ARBERT & MARBERT: Deep Bidirectional Transformers for Arabic. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). Association for Computational Linguistics, Online, 7088–7105. https://doi.org/10.18653/ v1/2021.acl-long.551 ↗
  2. Firoj Alam, Umair Qazi, Muhammad Imran, and Ferda Ofli. 2021. HumAID: Human-Annotated Disaster Incidents Data from Twitter. In 15th International Conference on Web and Social Media (ICWSM). AAAI Press, Palo Alto, California, USA, 933–942. find
  3. Alaa Alharbi and Mark Lee. 2021. Kawarith: an Arabic Twitter Corpus for Crisis Events. In Proceedings of the Sixth Arabic Natural Language Processing Workshop. Association for Computational Linguistics, Kyiv, Ukraine (Virtual), 42–52. https://aclanthology.org/2021.wanlp-1.5 ↗
  4. Jacob Cohen. 1960. A coefficient of agreement for nominal scales. Educational and psychological measurement 20, 1 (1960), 37–46. ↗
  5. Joao Porto De Albuquerque, Benjamin Herfort, Alexander Brenning, and Alexander Zipf. 2015. A geographic approach for combining social media and authoritative data towards identifying useful information for disaster management. International journal of geographical information science 29, 4 (2015), 667–689. ↗
  6. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Vol. 1. Association for Computational Linguistics (ACL), Minneapolis, Minnesota, 4171– 4186. https://doi.org/10.18653/v1/N19-1423 ↗
  7. Miguel Á García-Cumbreras, José M Perea-Ortega, Manuel García-Vega, and L Alfonso Ureña-López. 2009. Information retrieval with geographical references. Relevant documents filtering vs. query expansion. Information Processing & Management 45, 5 (2009), 605–614. find
  8. Bo Han, Paul Cook, and Timothy Baldwin. 2013. Lexical Normalization for Social Media Text. ACM Transactions on Intelligent Systems and Technology 4, 1 (Feb. 2013), 1–27. https://doi.org/10.1145/ 2414425.2414430 ↗
  9. Yingjie Hu and Jimin Wang. 2021. How Do People Describe Locations During a Natural Disaster: An Analysis of Tweets from Hurricane Harvey. In 11th International Conference on Geographic Information Science (GIScience 2021) - Part I (Leibniz International Proceedings in Informatics (LIPIcs), Vol. 177), Krzysztof Janowicz and Judith A. Verstegen (Eds.). Schloss Dagstuhl–Leibniz-Zentrum für Informatik, Dagstuhl, Germany, 6:1–6:16. https://doi.org/ 10.4230/LIPIcs.GIScience.2021.I.6 ↗
  10. Zongcheng Ji, Aixin Sun, Gao Cong, and Jialong Han. 2016. Joint Recognition and Linking of Fine-Grained Locations from Tweets. In Proceedings of the 25th International Conference on World Wide Web (Montréal, Québec, Canada) (WWW ’16). International World Wide Web Conferences Steering Committee, Republic and Canton of Geneva, CHE, 1271–1281. https://doi.org/10.1145/2872427.2883067 ↗
  11. Morteza Karimzadeh. 2016. Performance Evaluation Measures for Toponym Resolution. In Proceedings of the 10th Workshop on Geographic Information Retrieval (Burlingame, California) (GIR ’16). Association for Computing Machinery, New York, NY, USA, Article 8, 2 pages. https://doi.org/10. 1145/3003464.3003472 ↗
  12. Jessica Kropczynski, Rob Grace, Julien Coche, Shane Halse, Eric Obeysekare, Aurelie Montarnal, Frederick Benaben, and Andrea Tapia. 2018. Identifying Actionable Information on Social Media for Emergency Dispatch. In ISCRAM Asia Pacific 2018: Innovating for Resilience – 1st International Conference on Information Systems for Crisis Response and Management Asia Pacific. ISCRAM Digital Library, Wellington, New Zealand, p.428–438. https://hal-mines-albi.archives-ouvertes.fr/hal-01987793 ↗
  13. Guoliang Li, Jun Hu, Jianhua Feng, and Kian-lee Tan. 2014. Effective location identification from microblogs. In 2014 IEEE 30th International Conference on Data Engineering. Institute of Electrical and Electronics Engineers (IEEE), Chicago, IL, USA, 880–891. https://doi.org/10.1109/ICDE. 2014.6816708 ↗
  14. Haonan Li, Minghan Wang, Timothy Baldwin, Martin Tomko, and Maria Vasardani. 2019. UniMelb at SemEval-2019 Task 12: Multi-model combination for toponym resolution. In Proceedings of the 13th International Workshop on Semantic Evaluation. Association for Computational Linguistics, Minneapolis, Minnesota, USA, 1313–1318. https://doi.org/10.18653/v1/S19-2231 ↗
  15. Xiangyang Luo, Yaqiong Qiao, Chenliang Li, Jiangtao Ma, and Yimin Liu. 2020. An overview of microblog user geolocation methods. Information Processing & Management 57, 6 (2020), 102375. ↗
  16. Stuart E. Middleton, Giorgos Kordopatis-Zilos, Symeon Papadopoulos, and Yiannis Kompatsiaris. 2018. Location Extraction from Social Media: Geoparsing, Location Disambiguation, and Geotagging. ACM Transactions on Information Systems 36, 4 (June 2018), 1–27. ↗
  17. Ahmed Mourad, Falk Scholer, Walid Magdy, and Mark Sanderson. 2019. A practical guide for the effective evaluation of twitter user geolocation. ACM Transactions on Social Computing 2, 3 (2019), 1–23. ↗
  18. Nominatim. 2023. Nominatim Documentation. https://nominatim.org/release-docs/develop/ Ozer Ozdikis, Heri Ramampiaro, and Kjetil Nørvåg. 2019. Locality-adapted kernel densities of term co-occurrences for location prediction of tweets. Information Processing & Management 56, 4 (2019), 1280–1299. ↗
  19. Daniel J Solove. 2012. Introduction: Privacy self-management and the consent dilemma. Harv. L. Rev. 126 (2012), 1880. find
  20. Reem Suwaileh, Tamer Elsayed, and Muhammad Imran. 2022. Role of Geolocation Prediction in Disaster Management. Springer Nature Singapore, Singapore, 1–33. https://doi.org/10.1007/ 978-981-16-8800-3_176-1 ↗
  21. Reem Suwaileh, Tamer Elsayed, and Muhammad Imran. 2023a. IDRISI-RE: A generalizable dataset with benchmarks for location mention recognition on disaster tweets. Information Processing & Management 60, 3 (2023), 103340. https://doi.org/10. 1016/j.ipm.2023.103340 ↗
  22. Reem Suwaileh, Muhammad Imran, and Tamer Elsayed. 2023b. IDRISI-RA: The First Arabic Location Mention Recognition Dataset of Disaster Tweets. In Proceedings of the 61th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, Toronto, Canada, 16298–16317. https://doi.org/10.18653/v1/2023.acl-long.901 ↗
  23. Jan Oliver Wallgrün, Morteza Karimzadeh, Alan M. MacEachren, and Scott Pezanowski. 2018. GeoCorpora: building a corpus to test and train microblog geoparsers. International Journal of Geographical Information Science 32, 1 (2018), 1–29. ↗
  24. Jimin Wang and Yingjie Hu. 2019a. Are We There yet? Evaluating State-of-the-Art Neural Network Based Geoparsers Using EUPEG as a Benchmarking Platform. In Proceedings of the 3rd ACM SIGSPATIAL International Workshop on Geospatial Humanities (Chicago, Illinois) (GeoHumanities ’19). Association for Computing Machinery, New York, NY, USA, Article 2, 6 pages. https://doi.org/10.1145/ 3356991.3365470 ↗
  25. Jimin Wang and Yingjie Hu. 2019b. Enhancing spatial and textual analysis with EUPEG: An extensible and unified platform for evaluating geoparsers. Transactions in GIS 23, 6 (2019), 1393–1419. https://doi.org/10.1111/tgis.12579 ↗
  26. Xiaobin Wang, Chunping Ma, Huafei Zheng, Chu Liu, Pengjun Xie, Linlin Li, and Luo Si. 2019. DM_NLP at SemEval-2018 Task 12: A Pipeline System for Toponym Resolution. In Proceedings of the 13th International Workshop on Semantic Evaluation. Association for Computational Linguistics, Minneapolis, Minnesota, USA, 917–923. https://doi.org/10. 18653/v1/S19-2156 ↗
  27. Junhang Wu, Ruimin Hu, Dengshi Li, Lingfei Ren, Wenyi Hu, and Yilin Xiao. 2022. Where have you been: Dual spatiotemporal-aware user mobility modeling for missing check-in POI identification. Information Processing & Management 59, 5 (2022), 103030. ↗
  28. Canwen Xu, Jiaxin Pei, Jing Li, Chenliang Li, Xiangyang Luo, and Donghong Ji. 2019. DLocRL: A Deep Learning Pipeline for Fine-Grained Location Recognition and Linking in Tweets. In The World Wide Web Conference, WWW 2019. ACM, San Francisco, CA, USA, 3391–3397. https://doi.org/10.1145/3308558.3313491 ↗
  29. Vikas Yadav, Egoitz Laparra, Ti-Tai Wang, Mihai Surdeanu, and Steven Bethard. 2019. University of Arizona at SemEval-2019 Task 12: Deep-Affix Named Entity Recognition of Geolocation Entities. In Proceedings of the 13th International Workshop on Semantic Evaluation. Association for Computational Linguistics, Minneapolis, Minnesota, USA, 1319– 1323. https://doi.org/10.18653/v1/S19-2232 ↗
  30. Wei Zhang and Judith Gelernter. 2014. Geocoding location expressions in Twitter messages: A preference learning method. Journal of Spatial Information Science 2014, 9 (2014), 37–70. ↗
  31. Xin Zheng, Jialong Han, and Aixin Sun. 2018. A Survey of Location Prediction on Twitter. IEEE Transactions on Knowledge and Data Engineering 30, 9 (sep 2018), 1652–1671. https://doi.org/10.1109/ TKDE.2018.2807840 arXiv:1705.03172v2 ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/suwaileh-2023-idrisi-d-arabic-english-datasets/