Elamur Research⌕
☰
home / papers / 2022 / niksss at Qur'an QA 2022: A Heavily Optimized BERT Based Model for Answering Questions from the Holy Qu'ran

niksss at Qur'an QA 2022: A Heavily Optimized BERT Based Model for Answering Questions from the Holy Qu'ran

Paper · 2022 · Nikhil Singh

Summary

This system paper from team niksss addresses the Qur'an QA 2022 shared task. The task is extractive reading comprehension: given a Qur'anic passage and a question in Modern Standard Arabic, return answer spans that are exact substrings of the passage. The data is QRCD, split into train, development and test portions.

The author compared three approaches, all built on AraBERT or multilingual generators:

  • SEC: sentence embeddings with clustering and nearest-neighbour retrieval.
  • S2S: span generation with mT5 and mBART.
  • FT-AraBERT: a standard fine-tuned span-prediction model that scores start and end positions, tuned with a Weights & Biases hyperparameter sweep.

The fine-tuned AraBERT was submitted. On the test set it reached a pRR of 0.1913, with low Exact Match and F1@1.

Error analysis shows that factoid questions were handled reasonably well, while non-factoid ones mostly failed. The author attributes this to the tiny training set, roughly a thousand examples, being too small to adapt a large MLM-pretrained model to QA. The paper is a useful negative baseline for closed-domain, low-resource QA.

Key points

  • Dataset: QRCD contains 1,093 question-passage pairs and 1,337 question-passage-answer triplets. The split is 861 train (65%), 128 dev (10%) and 348 test (25%).

  • Dev-set results:

    MethodpRREMF1
    SEC0.2360.02620.112
    S2Snot reported0.306not reported
    FT-AraBERT0.2500.08340.139

  • S2S limitation: the mT5/mBART models had the best Exact Match, but they could not be made to output more than one answer, which the ranked metric requires.

  • Submitted configuration: Simple Transformers, learning rate 3e-4, max sequence length 128, 5 epochs, batch size 32.

  • Final test scores: pRR 0.1913, EM 4.2%, F1@1 9.1%. The official metric, Partial Reciprocal Rank, gives credit for partial overlap with any gold answer.

Interesting ideas

  • I1. The generative models matched gold answers best, yet they were unusable because the metric needs a ranked list of several spans; the output format of a system must be designed around how it is scored.
  • I2. The clustering approach first groups passage sentences and retrieves the cluster nearest the question, a cheap way to narrow where an answer could be.
  • I3. Splitting errors into factoid and non-factoid questions exposed a failure that the overall score concealed.
  • I4. The hyperparameter sweep tracked counts of correct and incorrect answers alongside training and evaluation loss, rather than trusting loss alone.
  • I5. Partial Reciprocal Rank rewards partially overlapping spans and also takes the rank of the matched answer into account, which gives a more forgiving signal than exact match.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (answer): readers should return a ranked list of candidate answers, each pinned to an exact text span, rather than a single generated string.
  • I2 (answer): a coarse sentence-selection pass could pick candidate clauses before the expensive frame and world-model reasoning runs on them.
  • I3 (learn): evaluation on fable and school-test QA should report results by question type (who/what/when versus why/how), so weak reasoning categories stay visible.
  • I4 (learn): training dashboards for gparse or the readers could log task-level correct and incorrect counts next to loss, catching runs whose loss improves while answers do not.
  • I5 (learn): a pRR-style score could grade span answers in QA self-play, so near-miss spans earn partial credit instead of counting as total failures.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 2The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 works

🧭 Topics 2Research subjects and languages the work is about.Semitic languages · 3,066 worksQuestion answering · 487 works

Cite

Cite as Nikhil Singh. 2022. niksss at Qur'an QA 2022: A Heavily Optimized BERT Based Model for Answering Questions from the Holy Qu'ran. Proceedinsg of the 5th Workshop on Open-Source Arabic Corpora and Processing Tools with Shared Tasks on Qur'an QA and Fine-Grained Hate Speech Detection.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{singh2022niksss,
  title = {{niksss at Qur'an QA 2022: A Heavily Optimized BERT Based Model for Answering Questions from the Holy Qu'ran}},
  author = {Nikhil Singh},
  booktitle = {Proceedinsg of the 5th Workshop on Open-Source Arabic Corpora and Processing Tools with Shared Tasks on Qur'an QA and Fine-Grained Hate Speech Detection},
  year = {2022},
  url = {https://aclanthology.org/2022.osact-1.15/}
}

References

13 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Abdelnasser, H., Ragab, M., Mohamed, R., Mohamed, A., Farouk, B., El-Makky, N., and Torki, M. (2014). Al-bayan: An Arabic question answering system for the holy quran. In Proceedings of the EMNLP 2014 Workshop on Arabic Natural Language Processing (ANLP), pages 57–64, Doha, Qatar, October. Association for Computational Linguistics.
  2. Alqahtani, M. and Atwell, E. (2016). Arabic quranic search tool based on ontology, June. © 2016, Springer International Publishing. This is an author produced version of a paper published in 21st International Conference on Applications of Natural Language to Information Systems, NLDB 2016 Proceedings, Natural Language Processing and Information Systems, Lecture Notes in Computer Science.. Uploaded in accordance with the publisher’s self-archiving policy. The final publication is available at Springer via http://dx.doi.org/10.1007/978-3-319-41754-7 52. ↗
  3. Antoun, W., Baly, F., and Hajj, H. (2020). Arabert: Transformer-based model for arabic language understanding. arXiv preprint arXiv:2003.00104. ↗
  4. Fix, E. and Hodges, J. L. (1989). Discriminatory analysis. nonparametric discrimination: Consistency properties. International Statistical Review / Revue Internationale de Statistique, 57(3):238–247. ↗
  5. Gusmita, R. H., Durachman, Y., Harun, S., Firmansyah, A. F., Sukmana, H. T., and Suhaimi, A. (2014). A rule-based question answering system on relevant documents of indonesian quran translation. 2014 International Conference on Cyber and IT Service Management (CITSM), pages 104–107. find
  6. Hakkoum, A. and Raghay, S. (2015). Advanced search in the qur’an using semantic modeling. 2015 IEEE/ACS 12th International Conference of Computer Systems and Applications (AICCSA), pages 1– 4. find
  7. Hakkoum, A. and Raghay, S. (2016). Semantic qa system on the qur’an. Arabian Journal for Science and Engineering, 41:5205–5214, 06. find
  8. Liu, Y., Gu, J., Goyal, N., Li, X., Edunov, S., Ghazvininejad, M., Lewis, M., and Zettlemoyer, L. (2020). Multilingual denoising pre-training for neural machine translation. Transactions of the Association for Computational Linguistics, 8:726–742. ↗
  9. Lloyd, S. P. (1982). Least squares quantization in pcm. IEEE Trans. Inf. Theory, 28:129–136. ↗
  10. Malhas, R. and Elsayed, T. (2020). ¡i¿ayatec¡/i¿: Building a reusable verse-based test collection for arabic question answering on the holy qur’an. ACM Trans. Asian Low-Resour. Lang. Inf. Process., 19(6), oct. find
  11. Malhas, R., Mansour, W., and Elsayed, T. (2022). find
  12. Qur’an QA 2022: Overview of the first shared task on question answering over the holy qur’an. In Proceedings of the 5th Workshop on Open-Source Arabic Corpora and Processing Tools (OSACT5) at the 13th Language Resources and Evaluation Conference (LREC 2022).
  13. Xue, L., Constant, N., Roberts, A., Kale, M., Al-Rfou, R., Siddhant, A., Barua, A., and Raffel, C. (2020). mt5: A massively multilingual pre-trained text-to-text transformer. arXiv preprint arXiv:2010.11934. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/singh-2022-niksss-qur-qa-2022-heavily/