Elamur Research⌕
☰
home / papers / 2022 / SMASH at Qur'an QA 2022: Creating Better Faithful Data Splits for Low-resourced Question Answering Scenarios

SMASH at Qur'an QA 2022: Creating Better Faithful Data Splits for Low-resourced Question Answering Scenarios

Paper · 2022 · Amr Keleg and Walid Magdy

Summary

The Qur'an QA 2022 shared task asks systems to pull answer spans out of Qur'anic passages written in Classical Arabic (CA), in response to questions posed in Modern Standard Arabic (MSA). The training data is small: 1,337 question/passage/answer triples. The SMASH team (University of Edinburgh) fine-tuned an Arabic BERT span extractor, then looked at its scores per question type. That breakdown exposed a serious flaw in the official development set. Many dev questions are paraphrases of training questions that share the passage and the exact answer span, so a model that memorises can look strong. In addition, one dev question type never appears in training.

To fix this, the authors merged the official splits and re-divided them into four disjoint subsets, separated by leakage and question rarity. This produced a more trustworthy evaluation split. On it, simply fine-tuning a language model pretrained on CA performed best. The overall picture suggests the dataset is too small for large transformers to generalise. The official submission scored a pRR of 0.4004, and the code is public.

Key points

  • Questions were typed by the first of twelve hand-listed Arabic interrogative words. Questions with none of them were labelled "NA", and the ambiguous word "من" was dropped when another interrogative was also present.
  • The polar ("هل") and "what" ("ماذا") types make up 60% of the dev set, and most of these are paraphrases of training items with the same passage and answer.
  • 17.5% of dev questions are "when" ("متى") questions, a type that is absent from training.
  • The four new subsets are:
    • D(1) in+leakage: 119 train / 181 dev.
    • D(2) in+no leakage: 209 / 32.
    • D(3) ood+hard: 0 / 60, used only for evaluation.
    • D(4) ood+easy: 189 / 29.
  • The model was a BERT span extractor fed the passage followed by the question. D(2) and D(4) were split randomly at the original 86.7/13.3 ratio.

Interesting ideas

  • I1. Breaking scores down by question type, instead of trusting one aggregate number, revealed that apparent "reasoning" was really memorisation of repeated answers.
  • I2. Leakage can be caught by flagging items that repeat a (passage, answer) or (question, answer) pair across splits, which surfaces paraphrased duplicates without needing a paraphrase model.
  • I3. Leaked items can be kept but quarantined: only one representative stays in training, and the duplicates go to a separate dev bucket, which measures memorisation explicitly.
  • I4. Items whose question occurs only rarely are held out entirely as a hard out-of-distribution set, which probes reasoning on unfamiliar question forms.
  • I5. A question type that is missing from training (here "when") cannot fairly be expected in dev, so split design should check that type coverage matches.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (answer): Report fable/tale QA and school-test accuracy per question kind (who/what/when/why/yes-no), so that a strong overall score cannot hide weak categories.
  • I2 (learn): Before trusting silver-data evaluations, deduplicate on (text, gold-answer) and (question, gold-answer) keys so that held-out sets truly test generalisation.
  • I3 (learn): Maintain a "seen-variant" control set next to the clean held-out set, so that the gap between the two shows how much a learned repair rule or parser merely memorises.
  • I4 (answer): Build a held-out bucket of rare question templates for the QA readers, to check that answers come from the frame/world graph rather than from familiar surface patterns.
  • I5 (world): When testing temporal questions about event order, make sure the training/evaluation material actually covers time-type queries, or report them separately as zero-shot.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksInformation Systems · 1,437 works

🧭 Topics 3Research subjects and languages the work is about.Semitic languages · 2,908 worksQuestion answering · 441 worksLow-resource languages · 267 works

Cite

Cite as Amr Keleg and Walid Magdy. 2022. SMASH at Qur'an QA 2022: Creating Better Faithful Data Splits for Low-resourced Question Answering Scenarios. Proceedinsg of the 5th Workshop on Open-Source Arabic Corpora and Processing Tools with Shared Tasks on Qur'an QA and Fine-Grained Hate Speech Detection.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{keleg2022smash,
  title = {{SMASH at Qur'an QA 2022: Creating Better Faithful Data Splits for Low-resourced Question Answering Scenarios}},
  author = {Amr Keleg and Walid Magdy},
  booktitle = {Proceedinsg of the 5th Workshop on Open-Source Arabic Corpora and Processing Tools with Shared Tasks on Qur'an QA and Fine-Grained Hate Speech Detection},
  year = {2022},
  url = {https://aclanthology.org/2022.osact-1.17/}
}

References

13 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Abdelali, A., Darwish, K., Durrani, N., and Mubarak, H. (2016). Farasa: A fast and furious segmenter for Arabic. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Demonstrations, pages 11–16, San Diego, California, June. Association for Computational Linguistics.
  2. Chen, D. and Yih, W.-t. (2020). Open-domain question answering. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: Tutorial Abstracts, pages 34–37, Online, July. Association for Computational Linguistics.
  3. Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. (2019). BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota, June. Association for Computational Linguistics. ↗
  4. Farha, I. A. and Magdy, W. (2021). Benchmarking transformer-based language models for arabic sentiment and sarcasm detection. In Proceedings of the sixth Arabic natural language processing workshop, pages 21–31.
  5. Inoue, G., Alhafni, B., Baimukan, N., Bouamor, H., and Habash, N. (2021). The interplay of variant, size, and task type in Arabic pre-trained language models. In Proceedings of the Sixth Arabic Natural Language Processing Workshop, pages 92–104, Kyiv, Ukraine (Virtual), April. Association for Computational Linguistics.
  6. Lewis, P., Stenetorp, P., and Riedel, S. (2021). Question and answer test-train overlap in open-domain question answering datasets. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 1000–1008, Online, April. Association for Computational Linguistics. find
  7. Malhas, R. and Elsayed, T. (2020). AyaTEC. ACM Transactions on Asian and Low-Resource Language Information Processing (TALLIP), 19:1 – 21. find
  8. Malhas, R., Mansour, W., and Elsayed, T. (2022). find
  9. Qur’an QA 2022: Overview of the first shared task on question answering over the holy Qur’an. In Proceedings of the 5th Workshop on Open-Source Arabic Corpora and Processing Tools (OSACT5) at the 13th Language Resources and Evaluation Conference (LREC 2022).
  10. Nigst, L., Romanov, M., Savant, S. B., Seydi, M., and Verkinderen, P. (2020). OpenITI: a Machine-Readable Corpus of Islamicate Texts, June. find
  11. Rajpurkar, P., Zhang, J., Lopyrev, K., and Liang, P. (2016). SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2383–2392, Austin, Texas, November. Association for Computational Linguistics. ↗
  12. Reimers, N. and Gurevych, I. (2017). Reporting score distributions makes a difference: Performance study of LSTM-networks for sequence tagging. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 338–348, Copenhagen, Denmark, September. Association for Computational Linguistics.
  13. Rogers, A., Kovaleva, O., Downey, M., and Rumshisky, A. (2020). Getting closer to AI complete question answering: A set of prerequisite real tasks. Proceedings of the AAAI Conference on Artificial Intelligence, 34(05):8722–8731, Apr. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/keleg-2022-smash-qur-qa-2022-creating/