Elamur Research⌕
☰
home / papers / 2021 / Empathetic BERT2BERT Conversational Model: Learning Arabic Language Generation with Little Data

Empathetic BERT2BERT Conversational Model: Learning Arabic Language Generation with Little Data

Paper · 2021 · Tarek Naous, Wissam Antoun, Reem Mahmoud et al.

Summary

Arabic NLP has strong understanding models such as AraBERT, but generating text, especially open-domain dialogue, is still difficult. Conversational corpora are scarce, and no pretrained Arabic dialogue model exists. The authors build a chatbot that responds with empathy. Their model is a transformer encoder-decoder in the BERT2BERT style, and both halves start from AraBERT weights. Only the cross-attention between encoder and decoder is learned from scratch. They fine-tune it on ArabicEmpatheticDialogues, a modest corpus machine-translated from the English EmpatheticDialogues, segment the text with Farasa, and decode with Top-K sampling. The model is compared with a Bi-LSTM Seq2Seq baseline, a variant that receives the emotion label, and an identical transformer trained without pretrained weights. It reaches a perplexity of 17.0 and gains about 5 BLEU over the previous best system. Human judges also rate it far above the alternatives. The work shows that reusing an understanding-oriented checkpoint can substitute for costly generative pretraining in a low-resource language. Its main weakness is that it answers emotionally even to neutral small talk.

Key points

  • ArabicEmpatheticDialogues has 36,628 utterance–response pairs. Each pair carries one of 32 emotion labels, and the paper groups these into six coarse classes following Parrott's emotion tree.
  • Input and output are each capped at 150 tokens. Training maximizes log-likelihood, and generation uses Top-K sampling instead of beam search.
  • BERT2BERT reaches PPL 17.0 and BLEU 5.58. The baselines are a Bi-LSTM Seq2Seq, EmoPrepend (the emotion label prepended to the input), and BERT2BERT-UN (same architecture, random initialization).
  • 85 native Arabic speakers scored responses from 0 to 5 on Empathy, Relevance and Fluency, and BERT2BERT averaged about 4.3/5. EmoPrepend beat the baseline but stayed under 3. BERT2BERT-UN was dropped from the survey because its automatic scores were too weak.
  • On neutral greetings such as "how are you?" the model still produces emotional replies. The authors attribute this to fine-tuning data that contains only emotional exchanges and to AraBERT having been pretrained on news text.

Interesting ideas

  • I1. The decoder is warm-started from an encoder-only checkpoint, leaving only cross-attention random, which gives a working generator without any generative pretraining or large dialogue corpus.
  • I2. An architecture-identical model with random initialization is included as a control, which isolates exactly how much the pretrained weights contribute.
  • I3. Top-K sampling is chosen over beam search because beam search tends to produce bland, repeated high-probability outputs.
  • I4. The failure analysis on neutral inputs shows that a model trained only on one kind of exchange applies that style everywhere, and the authors propose deciding first whether an empathetic reply fits at all.
  • I5. Human evaluation splits quality into three separate questions (emotion inference, topical relevance, linguistic correctness) instead of one overall score.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (back): A graph-to-text realizer could start its decoder from an existing English encoder checkpoint, so it trains on our small graph→text pairs instead of from scratch.
  • I2 (learn): Every new model in gparse or the teacher tiers could ship with a same-architecture "cold" twin as a negative control, so gains from pretraining are not confused with gains from architecture.
  • I3 (back): The text→graph→text compressor and paraphrase generation could use sampled candidates, then keep only those whose re-parsed graph matches the source graph.
  • I4 (frames): Speech-act and emotion detection in prag could act as an explicit gate that picks the response register before generation, keeping the decision inspectable.
  • I5 (answer): Answers from the readers could be judged separately on correctness, grounding in the text world, and fluency, so a failure can be traced to the stage that caused it.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 3Research subjects and languages the work is about.Semitic languages · 2,908 worksDialogue systems · 624 worksText generation · 202 works

Cite

Cite as Tarek Naous, Wissam Antoun, Reem Mahmoud and Hazem Hajj. 2021. Empathetic BERT2BERT Conversational Model: Learning Arabic Language Generation with Little Data. Proceedings of the Sixth Arabic Natural Language Processing Workshop.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{naous2021empathetic,
  title = {{Empathetic BERT2BERT Conversational Model: Learning Arabic Language Generation with Little Data}},
  author = {Tarek Naous, Wissam Antoun, Reem Mahmoud and Hazem Hajj},
  booktitle = {Proceedings of the Sixth Arabic Natural Language Processing Workshop},
  year = {2021},
  url = {https://aclanthology.org/2021.wanlp-1.17/}
}

References

31 references, 18 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ahmed Abdelali, Kareem Darwish, Nadir Durrani, and Hamdy Mubarak. 2016. Farasa: A fast and furious segmenter for arabic. In Proceedings of the 2016 conference of the North American chapter of the association for computational linguistics: Demonstrations, pages 11–16.
  2. Dana Abu Ali and Nizar Habash. 2016. Botta: An arabic dialect chatbot. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: System Demonstrations, pages 208–212.
  3. Wissam Antoun, Fady Baly, and Hazem Hajj. 2020. AraBERT: Transformer-based model for arabic language understanding. In Proceedings of the 4th Workshop on Open-Source Arabic Corpora and Processing Tools, with a Shared Task on Offensive Language Detection, pages 9–15. ↗
  4. Obeida ElJundi, Wissam Antoun, Nour El Droubi, Hazem Hajj, Wassim El-Hajj, and Khaled Shaban. 2019. hulmona: The universal language model in arabic. In Proceedings of the Fourth Arabic Natural Language Processing Workshop, pages 68–77.
  5. Ahmed Fadhil and Ahmed AbuRa’ed. 2019. OlloBot - towards a text-based Arabic health conversational agent: Evaluation and results. In Proceedings of the International Conference on Recent Advances in Natural Language Processing (RANLP 2019), pages 295–303. ↗
  6. Angela Fan, Mike Lewis, and Yann Dauphin. 2018. Hierarchical neural story generation. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 889–898.
  7. Mohammad Hijjawi, Zuhair Bandar, Keeley Crockett, and David Mclean. 2014. ArabChat: an arabic conversational agent. In 2014 6th International Conference on Computer Science and Information Technology (CSIT), pages 227–237. IEEE. find
  8. Minlie Huang, Xiaoyan Zhu, and Jianfeng Gao. 2020. Challenges in building intelligent open-domain dialog systems. ACM Transactions on Information Systems (TOIS), 38(3):1–32. ↗
  9. Daphne Ippolito, Reno Kriz, Joao Sedoc, Maria Kustikova, and Chris Callison-Burch. 2019. Comparison of diverse decoding methods from conditional language models. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3752–3762. find
  10. Guillaume Klein, Yoon Kim, Yuntian Deng, Jean Senellart, and Alexander M Rush. 2017. Opennmt: Open-source toolkit for neural machine translation. In Proceedings of ACL 2017, System Demonstrations, pages 67–72.
  11. Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. 2019. Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. arXiv preprint arXiv:1910.13461. ↗
  12. Yanran Li, Hui Su, Xiaoyu Shen, Wenjie Li, Ziqiang Cao, and Shuzi Niu. 2017. Dailydialog: A manually labelled multi-turn dialogue dataset. In Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 986–995. ↗
  13. Zhaojiang Lin, Peng Xu, Genta Indra Winata, Farhad Bin Siddique, Zihan Liu, Jamin Shin, and Pascale Fung. 2020. CAiRE: an end-to-end empathetic chatbot. In AAAI, pages 13622–13623. ↗
  14. Zhiqiang Ma, Rui Yang, Baoxiang Du, and Yan Chen. 2020. A control unit for emotional conversation generation. IEEE Access, 8:43168–43176. ↗
  15. Navonil Majumder, Pengfei Hong, Shanshan Peng, Jiankun Lu, Deepanway Ghosal, Alexander Gelbukh, Rada Mihalcea, and Soujanya Poria. 2020. Mime: Mimicking emotions for empathetic response generation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 8968–8979. ↗
  16. Tarek Naous, Christian Hokayem, and Hazem Hajj. 2020. Empathy-driven arabic conversational chatbot. In Proceedings of the Fifth Arabic Natural Language Processing Workshop, pages 58–68.
  17. W Gerrod Parrott. 2001. Emotions in social psychology: Essential readings. Psychology Press. find
  18. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1–67. ↗
  19. Hannah Rashkin, Eric Michael Smith, Margaret Li, and Y-Lan Boureau. 2019. Towards empathetic open-domain conversation models: A new benchmark and dataset. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5370–5381. ↗
  20. Sascha Rothe, Shashi Narayan, and Aliaksei Severyn. 2020. Leveraging pre-trained checkpoints for sequence generation tasks. Transactions of the Association for Computational Linguistics, 8:264–280. ↗
  21. Ashish Sharma, Adam Miner, David Atkins, and Tim Althoff. 2020. A computational approach to understanding empathy expressed in text-based mental health support. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 5263–5276.
  22. Jamin Shin, Peng Xu, Andrea Madotto, and Pascale Fung. 2020. Generating empathetic responses by looking ahead the user’s sentiment. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 7989–7993. IEEE. ↗
  23. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in neural information processing systems, pages 5998–6008.
  24. Özge Nilay Yalçın. 2019. Evaluating empathy in artificial agents. In 2019 8th International Conference on Affective Computing and Intelligent Interaction (ACII), pages 1–7. IEEE. ↗
  25. Özge Nilay Yalçın. 2020. Empathy framework for embodied conversational agents. Cognitive Systems Research, 59:123–132. ↗
  26. Özge Nilay Yalçın and Steve DiPaola. 2018. A computational model of empathy for interactive agents. Biologically Inspired Cognitive Architectures, 26:20– 25. ↗
  27. Özge Nilay Yalçın and Steve DiPaola. 2019. M-path: a conversational system for the empathic virtual agent. In Biologically Inspired Cognitive Architectures Meeting, pages 597–607. Springer. find
  28. Jingqing Zhang, Yao Zhao, Mohammad Saleh, and Peter Liu. 2020a. Pegasus: Pre-training with extracted gap-sentences for abstractive summarization. In International Conference on Machine Learning, pages 11328–11339. PMLR. ↗
  29. Saizheng Zhang, Emily Dinan, Jack Urbanek, Arthur Szlam, Douwe Kiela, and Jason Weston. 2018. Personalizing dialogue agents: I have a dog, do you have pets too? In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2204– 2213. ↗
  30. Yizhe Zhang, Siqi Sun, Michel Galley, Yen-Chun Chen, Chris Brockett, Xiang Gao, Jianfeng Gao, Jingjing Liu, and William B Dolan. 2020b. DI-ALOGPT: Large-scale generative pre-training for conversational response generation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 270–278. find
  31. Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In Proceedings of the IEEE International Conference on Computer Vision, pages 19–27. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/naous-2021-empathetic-bert2bert-conversational-model-learning/