Elamur Research⌕
☰
home / papers / 2020 / A Benchmark Arabic Dataset for Commonsense Explanation

A Benchmark Arabic Dataset for Commonsense Explanation

Summary

Arabic NLP has had few resources for testing whether machines can reason with everyday knowledge, and none for explaining why a statement is absurd. The authors release a benchmark for Arabic Commonsense Explanation (CSE). It is derived from the English data of the SemEval-2020 Commonsense Validation and Explanation (ComVE) task and keeps only the explanation subtask. Each item pairs a sentence that contradicts ordinary world knowledge with three candidate reasons, and a system must pick the reason that actually accounts for the absurdity. The resource contains 12k items split into train, validation and test sets. It is published on GitHub under CC-BY-SA 4.0 together with an evaluation script. As baselines, the authors train three transfer-learning language models (BERT, USE and ULMFiT) and report accuracy. All three barely beat chance: BERT reaches 34.20% against 32.5% for random guessing, while BERT on the original English data had reached 45.6%. The result shows the task is far from solved for Arabic and gives later work a fixed point of comparison.

Key points

  • Each example has four fields: one false sentence plus three explanation options, with a label marking the correct option (A/B/C).
  • The splits are 10,000 train, 1,000 validation and 1,000 test items, built on the SemEval-2020 Task 4 (ComVE) English data.
  • Baseline accuracy on test: Random 32.5%, USE 33%, ULMFiT 32.8%, BERT 34.20%.
  • For comparison, BERT scored 45.6% when run on the English source data that this benchmark was adapted from (Wang et al., 2019).
  • The dataset and evaluation code are released at github.com/msmadi under CC-BY-SA 4.0, and the work is presented as the first public Arabic dataset of this kind.

Interesting ideas

  • I1. Wrong options mention the same entities as the false sentence and can even be true (e.g. yachts are usually white, beds beige), so a model must judge explanatory relevance rather than topical overlap.
  • I2. The examples (grandfather put in a cot, yacht put in a bed) are absurd because a role filler breaks a size or category restriction of the verb's slots.
  • I3. A random baseline is reported next to the trained models, which exposes that a 34.2% score is essentially chance-level.
  • I4. Running the same model family on an English benchmark and its ported version exposes a cross-language gap (45.6% vs 34.2%).
  • I5. Fixed public splits plus a shipped evaluation script make results directly comparable across later papers.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (world): Build tests where the world model must name the violated constraint (yacht bigger than bed) and reject true but irrelevant facts about the same entities.
  • I2 (frames): Check VerbNet selectional restrictions on Theme and Destination roles in placement frames, so the frame layer itself can flag an implausible "who put what where".
  • I3 (learn): Always pair evaluations with a random or shuffled-input control, so near-chance gains are not mistaken for real learning.
  • I4 (back): Parallel item sets could test whether translation through the meta-world keeps the impossibility that makes a sentence false.
  • I5 (answer): Freeze test sets for school-course and fable QA readers and publish one scoring script, so reader versions are compared on identical terms.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics Evaluation & benchmarks Reasoning Semitic languages

 Institutions

🎓Jordan University of Science and TechnologyIrbid · Jordan9 works · 1,785 citations

Cite

Cite as Saja AL-Tawalbeh and Mohammad AL-Smadi. 2020. A Benchmark Arabic Dataset for Commonsense Explanation. arXiv:2012.10251.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2012.10251 · arXiv 2012.10251

@misc{altawalbeh2020benchmark,
  title = {{A Benchmark Arabic Dataset for Commonsense Explanation}},
  author = {Saja AL-Tawalbeh and Mohammad AL-Smadi},
  year = {2020},
  doi = {10.48550/arxiv.2012.10251},
  eprint = {2012.10251},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2012.10251}
}

References

18 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Daniel Cer, Yinfei Yang, Sheng-yi Kong, Nan Hua, Nicole Limtiaco, Rhomni St John, Noah Constant, Mario Guajardo-Cespedes, Steve Yuan, Chris Tar, et al. 2018. Universal sentence encoder. arXiv preprint arXiv:1803.11175. ↗ our card
  2. Ernest Davis. 2016. How to write science questions that are easy for people and hard for computers. AI magazine, 37(1):13–22. find
  3. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805. ↗ our card
  4. Catherine Havasi, Robert Speer, Kenneth Arnold, Henry Lieberman, Jason Alonso, and Jesse Moeller. 2010. Open mind common sense: Crowd-sourcing for common sense. In Workshops at the Twenty-Fourth AAAI Conference on Artificial Intelligence. find
  5. Jeremy Howard and Sebastian Ruder. 2018. Universal language model fine-tuning for text classification. arXiv preprint arXiv:1801.06146. ↗
  6. Lifu Huang, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2019. Cosmos qa: Machine reading comprehension with contextual commonsense reasoning. arXiv preprint arXiv:1909.00277. ↗ our card
  7. Hector Levesque, Ernest Davis, and Leora Morgenstern. 2012. The winograd schema challenge. In Thirteenth International Conference on the Principles of Knowledge Representation and Reasoning. find
  8. Simon Ostermann, Ashutosh Modi, Michael Roth, Stefan Thater, and Manfred Pinkal. 2018a. Mcscript: A novel dataset for assessing machine comprehension using script knowledge. arXiv preprint arXiv:1803.05223. ↗
  9. Simon Ostermann, Michael Roth, Ashutosh Modi, Stefan Thater, and Manfred Pinkal. 2018b. Semeval-2018 task 11: Machine comprehension using commonsense knowledge. In Proceedings of the 12th International Workshop on semantic evaluation, pages 747–757. find
  10. Melissa Roemmele, Cosmin Adrian Bejan, and Andrew S Gordon. 2011. Choice of plausible alternatives: An evaluation of commonsense causal reasoning. In 2011 AAAI Spring Symposium Series. find
  11. Sirwe Saeedi, Aliakbar Panahi, Seyran Saeedi, and Alvis C Fong. 2020. Cs-nlp team at semeval-2020 task 4: Evaluation of state-of-the-artnlp deep learning architectures on commonsense reasoning task. arXiv preprint arXiv:2006.01205. ↗
  12. Chenglei Si, Shuohang Wang, Min-Yen Kan, and Jing Jiang. 2019. What does bert learn from multiple-choice reading comprehension datasets? arXiv preprint arXiv:1910.12391. ↗
  13. Saja Tawalbeh and Mohammad AL-Smadi. 2020. Is this sentence valid? an arabic dataset for commonsense validation. arXiv preprint arXiv:2008.10873. ↗ our card
  14. Cunxiang Wang, Shuailong Liang, Yue Zhang, Xiaonan Li, and Tian Gao. 2019. Does it make sense? and why? a pilot study for sense making and explanation. arXiv preprint arXiv:1906.00363. ↗
  15. Cunxiang Wang, Shuailong Liang, Yili Jin, Yilong Wang, Xiaodan Zhu, and Yue Zhang. 2020. Semeval-2020 task 4: Commonsense validation and explanation. arXiv preprint arXiv:2007.00236. ↗
  16. K. Xu, Jing Tin, and J. Youn. 2019. A bert based model for multiple-choice reading comprehension. find
  17. Rowan Zellers, Yonatan Bisk, Roy Schwartz, and Yejin Choi. 2018. Swag: A large-scale adversarial dataset for grounded commonsense inference. arXiv preprint arXiv:1808.05326. ↗
  18. Sheng Zhang, Rachel Rudinger, Kevin Duh, and Benjamin Van Durme. 2017. Ordinal common-sense inference. Transactions of the Association for Computational Linguistics, 5:379–395. our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/al-tawalbeh-2020-benchmark-arabic-dataset-commonsense-explanation/