Elamur Research⌕
☰
home / papers / 2022 / TCE at Qur'an QA 2022: Arabic Language Question Answering Over Holy Qur'an Using a Post-Processed Ensemble of BERT-based Models

TCE at Qur'an QA 2022: Arabic Language Question Answering Over Holy Qur'an Using a Post-Processed Ensemble of BERT-based Models

Paper · 2022 · Mohamemd Elkomy and Amany M. Sarhan

Summary

The paper describes a system built for the OSACT5 Qur'an QA 2022 shared task: extractive question answering in which a question in Modern Standard Arabic must be answered by a span from a Qur'anic passage in Classical Arabic. The data is QRCD, a SQuAD-style collection that is very small and in which the same few questions recur with many different passages. This makes it hard for a model to learn passage-dependent answers. The authors fine-tuned several Arabic BERT variants, including AraBERT large and base and ARBERT, and trained many copies of each. They merged the copies' ranked span predictions through span voting. They then post-processed the merged output to fit the ranking metric, partial Reciprocal Rank (pRR), for example by discarding spans that carry no useful content. Their best submission reached 56.6% pRR on the official test set. Development scores varied strongly from run to run, and development and test results differed noticeably. This suggests that with data this scarce, ensembling and metric-aware cleanup matter about as much as the choice of model.

Key points

  • QRCD contains 1,093 question-passage pairs that expand to 1,337 question-passage-answer triplets. The split is train 710 pairs / 118 unique questions, dev 109 / 17, and test 274 / 34. Only 238 test items were scored, because 36 were too close to the public splits.
  • The evaluation is a ranking: a system returns up to 5 answers. pRR gives partial credit for inexact overlap and discounts that credit by the rank of the first match. EM and F1@1 are reported alongside it.
  • Each BERT model scores spans by summing start and end logits and produces 20 ranked candidates. These candidates are then merged across models.
  • The final ensemble used 16 bert-large-arabertv02 models, 18 bert-base-arabertv02 models and 17 ARBERT models.
  • On the test set, removing uninformative spans gave EM 26.8, F1 50.0 and pRR 56.6. Keeping those spans gave the same EM 26.8, with F1 48.5 and pRR 55.7.

Interesting ideas

  • I1. Many fine-tuned copies of several backbones each vote on candidate spans, which smooths out the large seed-to-seed instability seen on tiny data.
  • I2. Each model over-generates candidates (20 per model) and the final list is cut to the 5 the metric allows, so ranking happens after pooling.
  • I3. A post-processing filter removes answer spans that carry no content and raised pRR by about a point at no training cost.
  • I4. The authors point out that a dataset with few unique questions repeated over many passages forces the model to tie its answer to context rather than to the question alone.
  • I5. The authors blame the dev-test gap on the tiny validation split and on the filtered test items, and they propose extensive cross-validation as a remedy.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): when training gparse or other components on small silver sets, an ensemble of seeds could vote, and agreement between seeds could serve as a confidence signal for which data to keep.
  • I2 (answer): a reader could produce several candidate answers from the world graph, pool them across reasoning paths, and rank them explicitly, recording why each one survived.
  • I3 (answer): answer extraction could include an explainable rule that rejects spans made only of function words or stopwords before ranking.
  • I4 (learn): evaluation sets for fable QA could deliberately reuse one question across different stories, which works as a negative control against answering from question priors.
  • I5 (learn): small treebank or QA evaluations should report variance across folds, so that improvements are not over-trusted.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Tanta UniversityTanta · Egypt7 works · 278 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksInformation Systems · 1,437 works

🧭 Topics 2Research subjects and languages the work is about.Semitic languages · 2,908 worksQuestion answering · 441 works

Cite

Cite as Mohamemd Elkomy and Amany M. Sarhan. 2022. TCE at Qur'an QA 2022: Arabic Language Question Answering Over Holy Qur'an Using a Post-Processed Ensemble of BERT-based Models. Proceedinsg of the 5th Workshop on Open-Source Arabic Corpora and Processing Tools with Shared Tasks on Qur'an QA and Fine-Grained Hate Speech Detection.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{elkomy2022tce,
  title = {{TCE at Qur'an QA 2022: Arabic Language Question Answering Over Holy Qur'an Using a Post-Processed Ensemble of BERT-based Models}},
  author = {Mohamemd Elkomy and Amany M. Sarhan},
  booktitle = {Proceedinsg of the 5th Workshop on Open-Source Arabic Corpora and Processing Tools with Shared Tasks on Qur'an QA and Fine-Grained Hate Speech Detection},
  year = {2022},
  url = {https://aclanthology.org/2022.osact-1.19/}
}

References

13 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Abdelali, A., Hassan, S., Mubarak, H., Darwish, K., and Samih, Y. (2021). Pre-training bert on arabic tweets: Practical considerations. ↗
  2. Abdelnasser, H., Ragab, M., Mohamed, R., Mohamed, A., Farouk, B., El-Makky, N., and Torki, M. (2014). Al-bayan: An Arabic question answering system for the holy quran. In Proceedings of the EMNLP 2014 Workshop on Arabic Natural Language Processing (ANLP), pages 57–64, Doha, Qatar, October. Association for Computational Linguistics.
  3. Abdul-Mageed, M., Elmadany, A., and Nagoudi, E. M. B. (2021). ARBERT & MARBERT: Deep bidirectional transformers for Arabic. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 7088–7105, Online, August. Association for Computational Linguistics. ↗
  4. Alwaneen, T. H., Azmi, A. M., Aboalsamh, H. A., Cambria, E., and Hussain, A. (2022). Arabic question answering system: a survey. Artificial Intelligence Review, 55(1):207–253, Jan. ↗
  5. Antoun, W., Baly, F., and Hajj, H. (2020). AraBERT: Transformer-based model for Arabic language understanding. In Proceedings of the 4th Workshop on Open-Source Arabic Corpora and Processing Tools, with a Shared Task on Offensive Language Detection, pages 9–15, Marseille, France, May. European Language Resource Association. ↗
  6. Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. (2019). BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171– 4186, Minneapolis, Minnesota, June. Association for Computational Linguistics. ↗
  7. Malhas, R. and Elsayed, T. (2020). Ayatec: Building a reusable verse-based test collection for arabic question answering on the holy qur’an. ACM Trans. Asian Low-Resour. Lang. Inf. Process., 19(6), oct. find
  8. Malhas, R., Mansour, W., and Elsayed, T. (2022a). find
  9. Qur’an QA 2022: Overview of the first shared task on question answering over the holy qur’an. In Proceedings of the 5th Workshop on Open-Source Arabic Corpora and Processing Tools (OSACT5) at the 13th Language Resources and Evaluation Conference (LREC 2022).
  10. Malhas, R., Mansour, W., and Elsayed, T. (2022b). Qur’an qa 2022 shared task. https://gitlab.com/bigirqu/quranqa. ↗
  11. Mozannar, H., Maamary, E., El Hajal, K., and Hajj, H. (2019). Neural Arabic question answering. In Proceedings of the Fourth Arabic Natural Language Processing Workshop, pages 108–118, Florence, Italy, August. Association for Computational Linguistics.
  12. Rajpurkar, P., Zhang, J., Lopyrev, K., and Liang, P. (2016). SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2383–2392, Austin, Texas, November. Association for Computational Linguistics. ↗
  13. Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., and Brew, J. (2019). Huggingface’s transformers: State-of-the-art natural language processing. CoRR, abs/1910.03771. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/elkomy-2022-tce-qur-qa-2022-arabic/