Elamur Research⌕
☰
home / papers / 2015 / Turn Segmentation into Utterances for Arabic Spontaneous Dialogues and Instance Messages

Turn Segmentation into Utterances for Arabic Spontaneous Dialogues and Instance Messages

Paper · 2015 · AbdelRahim A. Elmadany, Sherif M. Abdou and Mervat Gheith

Summary

The paper addresses how to split a speaker's turn in Egyptian Arabic dialogues and instant messages (IM) into utterances, the smaller units that each carry one dialogue act. Earlier Arabic dialogue-act work treated a whole turn as one utterance, and existing Arabic segmenters target formal MSA text and depend on punctuation, conjunctions, diacritics or lexical cues. Informal dialect transcripts and chats often lack these signals.

The authors propose USeg, a machine-learning segmenter driven by context. Its features come from annotated data, including morphological features produced by MADAMIRA, and it deliberately avoids punctuation, diacritization and lexical cue lists. No Egyptian dialogue corpus existed, so they built JANA from call-center calls (banks, flights) and mobile-operator online-support chats. The calls were transcribed with Transcriber, then segmented and tagged with dialogue acts by hand.

On 3,001 turns, USeg reaches an F1 of 90.74% and an accuracy of 95.98%. The authors present these as the first reported results for this task in Egyptian dialect, and the work is a prerequisite for dialogue-act classification of dialect dialogues.

Key points

  • JANA corpus, first release: 3,001 turns.
    • 1,091 turns needed segmentation, and these yielded 2,815 utterances.
    • There are 4,725 utterances in total.
    • Averages are about 6.7 words per turn and 4.3 words per utterance.
    • The word count is about 20K (the table says 20,113, the text says 20,311).
  • Sources: 200 dialogues were collected starting in August 2013. The release uses 52 spoken dialogues (26 bank, 26 flight) and 30 IM chats from mobile network operators. JANA is promised free for academic and non-profit use.
  • Annotation: utterances carry dialogue-act labels from the authors' own Arabic inquiry-answer DA schema, with tags such as Greeting, SelfIntroduce, Taking_Request, Service_Question, Inform and Disagree.
  • Features: morphological analysis and disambiguation come from MADAMIRA. No punctuation, diacritics or cue-word lists are used.
  • Results: F1 90.74% and accuracy 95.98%, measured against expert manual segmentation.

Interesting ideas

  • I1. The paper separates the turn (a speaker's whole contribution) from the utterance (one act), so one turn can contain several acts that each need their own label.
  • I2. Segmentation is learned from morphology and context rather than punctuation, which keeps it working on unpunctuated chat and transcripts.
  • I3. The paper lists spontaneous-speech phenomena that break naive splitting: self-repairs with apologies, repeated negations, hesitations and code-switching.
  • I4. The corpus reports how many turns actually needed splitting (about a third), which makes clear what a trivial "never split" baseline already achieves.
  • I5. One corpus serves two layers, segmentation and dialogue-act labels, which lets errors in the first step be traced into the second.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (frames): speech-act tagging in prag could first cut multi-act sentences or messages into act-sized units, so each unit gets one explainable speech-act frame.
  • I2 (tokens): the en tokenizer could gain a fallback boundary detector that uses tagger and morphology features when input lacks reliable punctuation, such as chat logs and ASR text.
  • I3 (tree): these could be turned into a targeted test suite of disfluent inputs to check that the parser does not attach repaired material ("on 12, sorry, 11") as ordinary arguments.
  • I4 (learn): evaluations of segmenters or other boundary decisions should always report a do-nothing negative control alongside accuracy, so that inflated scores are easy to spot.
  • I5 (learn): silver data could keep segment boundaries and act labels as separate, linked layers, so a wrong act can be blamed on the boundary or on the classifier.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Cairo UniversityGiza · Egypt42 works · 1,350 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 2Research subjects and languages the work is about.Semitic languages · 2,908 worksDialogue systems · 624 works

Cite

Cite as AbdelRahim A. Elmadany, Sherif M. Abdou and Mervat Gheith. 2015. Turn Segmentation into Utterances for Arabic Spontaneous Dialogues and Instance Messages. arXiv:1505.03081 · International Journal on Natural Language Computing (IJNLC) Vol. 4, No.2,April 2015.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.5121/ijnlc.2015.4208 · arXiv 1505.03081

@misc{elmadany2015turn,
  title = {{Turn Segmentation into Utterances for Arabic Spontaneous Dialogues and Instance Messages}},
  author = {AbdelRahim A. Elmadany, Sherif M. Abdou and Mervat Gheith},
  year = {2015},
  doi = {10.5121/ijnlc.2015.4208},
  eprint = {1505.03081},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.5121/ijnlc.2015.4208}
}

References

25 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. N. Webb, "Cue-Based Dialogue Act Classification," Ph.D. dissertation, Department of Computer Science, University of Sheffield, England, 2010. find
  2. D. Traum and P. A. Heeman, "Utterance units in spoken dialogue," in Dialogue processing in spoken language systems, ed. Berlin Heidelberg: Springer, 1997, pp. 125-140. find
  3. M. Graja, M. Jaoua, and L. H. Belguith, "Discriminative Framework for Spoken Tunisian Dialect Understanding," presented at the 2nd International Conference on Statistical Language and Speech Processing, SLSP 2014, 2013. find
  4. A. Pasha, M. Al-Badrashiny, M. Diab, A. E. Kholy, R. Eskander, N. Habash, M. Pooleery, O. Rambow, and R. M. Roth, "MADAMIRA: A Fast, Comprehensive Tool for Morphological Analysis and Disambiguation of Arabic," presented at the Language Resources and Evaluation Conference (LREC 2014), 2014.
  5. M. Elmahdy, G. Rainer, M. Wolfgang, and A. Slim, "Survey on common Arabic language forms from a speech recognition point of view," In Proceeding of International conference on Acoustics (NAG-DAGA), 2009, pp. 63-66. find
  6. O. F. Zaidan and C. Callison-Burch, "Arabic dialect identification," Computational Linguistics, vol. 52 (1), 2012. ↗
  7. J. Ang, Y. Liu, and E. Shriberg, "Automatic Dialog Act Segmentation and Classification in Multiparty Meetings," In Proceeding of IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP '05), 2005, pp. 1061 - 1064. find
  8. E. Ivanovic, "Automatic utterance segmentation in instant messaging dialogue," In Proceeding of The Australasian Language Technology Workshop, 2005, pp. 241-249. find
  9. M. Zimmermann, Y. Liu, E. Shriberg, and A. Stolcke, "Toward Joint Segmentation and Classification of Dialog Acts in Multiparty Meetings," In Proceeding of Proc. Multimodal Interaction and Related Machine Learning Algorithms Workshop (MLMI–05), 2005. find
  10. L. Ding and C. Zong, "Utterance segmentation using combined approach based on bi-directional Ngram and maximum entropy," Proceedings of the second SIGHAN workshop on Chinese language processing. Association for Computational Linguistics, vol. 17 2003. find
  11. L. Shala, V. Rus, and A. Graesser, "Automatic Speech Act Classification In Arabic," In Proceeding of Subjetividad y Procesos Cognitivos Conference 2010, pp. 284-292. find
  12. Y. Bahou, L. H. Belguith, and A. B. Hamadou, "Towards a Human-Machine Spoken Dialogue in Arabic," In Proceeding of Workshop on HLT & NLP within the Arabic world: Arabic Language and local languages processing: Status Updates and Prospects, at the 6th Language Resources and Evaluation Conference (LREC‟08), Marrakech, Maroc, 2008. find
  13. C. Lhioui, A. Zouaghi, and M. Zrigui, "A Combined Method Based on Stochastic and Linguistic Paradigm for the Understanding of Arabic Spontaneous Utterances," In Proceeding of CICLing 2013, Computational Linguistics and Intelligent Text Processing Lecture Notes in Computer Science, Samos, Greece, 2013, pp. 549-558. find
  14. M. Hijjawi, Z. Bandar, and K. Crockett, "User‟s Utterance Classification Using Machine Learning for Arabic Conversational Agents," In Proceeding of 5th International Conference on Computer Science and Information Technology (CSIT), 2013, pp. 223-232. find
  15. M. Hijjawi, Z. Bandar, K. Crockett, and D. Mclean, "ArabChat: an Arabic Conversational Agent," In Proceeding of 6th International Conference on Computer Science and Information Technology (CSIT), 2014, pp. 227-237. find
  16. L. Belguith, L. Baccour, and G. Mourad, "Segmentation de textes arabes basée sur l'analyse contextuelle des signes de ponctuations et de certaines particules," In Proceeding of 12th Conference on Natural Language Processing (TALN‟2005), 2005, pp. 451-456. find
  17. A. A. Touir, H. Mathkour, and W. Al-Sanea, "Semantic-Based Segmentation of Arabic Texts. ," Information Technology Journal, vol. 7 (7), 2008. ↗
  18. I. Keskes, F. Benamara, and L. H. Belguith, "Clause-based Discourse Segmentation of Arabic Texts," In Proceeding of The eighth international conference on Language Resources and Evaluation (LREC), Istanbul, 2012.
  19. I. Khalifa, Z. A. Feki, and A. Farawila, "Arabic Discourse Segmentation Based on Rhetorical Methods," International Journal of Electric & Computer Sciences IJECS-IJENS, vol. 11 (01), 2011. find
  20. A. A. Elmadany, S. M. Abdou, and M. Gheith, "Recent Approaches to Arabic Dialogue Acts Classifications," 4th International Conferences on Natural Language Processing (NLP-2015) - Computer Science & Information Technology (CS & IT) Series, vol. 5 (4), pp. 117–129, 2015. find
  21. A. A. Elmadany, S. M. Abdou, and M. Gheith, "Arabic Inquiry-Answer Dialogue Acts Annotation Schema," IOSR Journal of Engineering (IOSRJEN), vol. 04 (12-V2), pp. 32-36, 2014. ↗
  22. T. Kudo and Y. Matsumoto, "Chunking with support vector machines," In Proceeding of NAACL-01, 2001.
  23. Y. Kudo and Y. Matsumoto, "Use of Support Vector Learning for Chunk Identification " In Proceeding of CoNLL, Lisbon, Portugal, 2000, pp. 142-144. find
  24. M. A. Meselhi, H. M. A. Bakr, I. Ziedan, and K. Shaalan, "A Novel Hybrid Approach to Arabic Named Entity Recognition," presented at the The 10th China Workshop on Machine Translation (CWMT 2014), 2014. find
  25. S. Pradhan, K. Hacioglu, V. Krugler, W. Ward, J. H. Martin, and D. Jurafsky, "Support vector learning for semantic argument classification," Machine Learning 60, vol. Machine Learning (1-3), pp. 11-39, 2005. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/elmadany-2015-turn-segmentation-utterances-arabic-spontaneous/