Elamur Research⌕
☰
home / papers / 2020 / Mapping the Dialog Act Annotations of the LEGO Corpus into ISO 24617-2 Communicative Functions

Mapping the Dialog Act Annotations of the LEGO Corpus into ISO 24617-2 Communicative Functions

Paper · 2020 · Eugénio Ribeiro, Ricardo Ribeiro and David Martins de Matos

Summary

ISO 24617-2 gives dialog act research a shared annotation scheme, but very little data follows it. The only public source is the DialogBank, with 17 English and 9 Dutch dialogs, which is too small for data-hungry neural recognizers. The authors take the LEGO corpus, 347 Let's Go bus-information calls whose original dialog act tags had been largely ignored because they are highly domain-specific. They convert those tags into ISO communicative functions. For system turns, which come from slot-filled templates, they inspect the templates behind each label and assign functions to them. User labels are mapped separately. The result does not cover every dimension or the full ISO annotation structure. Still, it is a sizable, publicly released resource (LEGO-ISO). In preliminary experiments with the English DialogBank as the gold standard, adding the mapped dialogs to training improved recognition of Task-dimension communicative functions. This suggests that converting legacy annotations can help a standard gain wider adoption.

Key points

  • LEGO is a subset of CMU's Let's Go Bus Information System recorded in 2006. It contains 347 calls and 14,186 utterances: 9,083 system and 5,103 user.
  • The original tag sets are separate for each side: 28 system labels (top label Confirm Understood, about 10%) and 22 user labels (top labels Place Information, about 17%, and Unqualified / Unrecognized, about 15%).
  • Example system mappings:
    • Announce Querying becomes Pausing (Time Management), plus Promise in the Task dimension when the system commits to looking something up.
    • Announce Restart becomes Interaction Structuring (Discourse Structuring).
  • Evaluation used the 17 English DialogBank dialogs as gold standard, with exact match and hierarchical F-measure. Adding the mapped data narrowed the precision–recall gap from about eight points to two.
  • The mapped annotations are distributed as LEGO-ISO.

Interesting ideas

  • I1. Very narrow, domain-bound labels can be easier to lift into a general scheme than vague ones, because each narrow tag has an unambiguous parent.
  • I2. When the text is generated from templates, label the templates rather than every instance, then propagate the labels automatically.
  • I3. A single segment can legitimately carry functions in several dimensions at once, such as stalling for time while also promising an action.
  • I4. Hierarchical F-measure gives partial credit for predicting an ancestor of the correct label, and the precision–recall gap shows whether the model guesses too specifically.
  • I5. Incomplete but larger converted annotations can be added to training while scoring only on a small, trusted gold set.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (frames): Speech-act labels from small domain datasets could be rolled up into prag's general act inventory through an explicit, auditable mapping table.
  • I2 (learn): Templated or synthetic sentences from self-play could carry gold frames assigned once per template, giving cheap and consistent training data.
  • I3 (frames): Prag's speech-act layer could allow parallel acts per clause across separate dimensions instead of forcing one label.
  • I4 (learn): Evaluation of hierarchical outputs, such as WordNet senses or VerbNet classes, could score ancestor hits and track that gap as a diagnostic.
  • I5 (learn): Partial silver frames from converted legacy corpora can supplement training, as long as gains are always measured on held-out gold data.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🔬Instituto de Engenharia de Sistemas e Computadores Investigação e DesenvolvimentoLisbon · Portugal21 works · 3,723 citations

📚 Shelves 4 Computer Science · 9,086 works Artificial Intelligence · 8,889 works Arts and Humanities · 9,296 works Language and Linguistics · 9,206 works

🧭 Topics 3 English · 1,373 works Dialogue systems · 401 works Speech act theory · 160 works

Cite

Cite as Eugénio Ribeiro, Ricardo Ribeiro and David Martins de Matos. 2020. Mapping the Dialog Act Annotations of the LEGO Corpus into ISO 24617-2 Communicative Functions. Proceedings of the Twelfth Language Resources and Evaluation Conference.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{ribeiro2020mapping,
  title = {{Mapping the Dialog Act Annotations of the LEGO Corpus into ISO 24617-2 Communicative Functions}},
  author = {Eugénio Ribeiro, Ricardo Ribeiro and David Martins {de Matos}},
  booktitle = {Proceedings of the Twelfth Language Resources and Evaluation Conference},
  year = {2020},
  url = {https://aclanthology.org/2020.lrec-1.67/}
}

References

27 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Allen, J. F. and Schubert, L. K. (1991). The TRAINS Project. Technical Report Technical Report 382 and TRAINS Technical Note 91-1, Computer Science Dept., University of Rochester. ↗
  2. Anderson, A. H., Bader, M., Bard, E. G., Boyle, E., Doherty, G., Garrod, S., Isard, S., Kowtko, J., McAllister, J., Miller, J., et al. (1991). The HCRC Map Task Corpus. Language and Speech, 34(4):351–366. ↗
  3. Brester, C., Semenkin, E., Sidorov, M., and Semenk-ina, O. (2015). Multicriteria Neural Network Design in the Speech-Based Emotion Recognition Problem. In ICINCO, volume 1, pages 621–628. find
  4. Bunt, H., Alexandersson, J., Choe, J.-W., Fang, A. C., Hasida, K., Petukhova, V., Popescu-Belis, A., and Traum, D. (2012). Iso 24617-2: A semantically-based standard for dialogue annotation. In LREC, pages 430–437. our card
  5. Bunt, H., Petukhova, V., Malchanau, A., Wijnhoven, K., and Fang, A. (2016). The dialogbank. In LREC, pages 3151–3158. our card
  6. Bunt, H., Petukhova, V., Traum, D., and Alexandersson, J. (2017). Dialogue Act Annotation with the ISO 24617-2 Standard. In Multimodal Interaction with W3C Standards. Springer, pp. 109–135. ↗
  7. Bunt, H., Petukhova, V., Malchanau, A., Fang, A., and Wijnhoven, K. (2019). The DialogBank: Dialogues with Interoperable Annotations. Language Resources and Evaluation, 53(2):213–249. ↗
  8. Carroll, J. M. and Tanenhaus, M. K. (1978). Functional Clauses and Sentence Segmentation. Journal of Speech, Language, and Hearing Research, 21(4):793–808. ↗
  9. Geertzen, J., Girard, Y., Morante, R., Van der Sluis, I., Van Dam, H., Suijkerbuijk, B., Van der Werf, R., and Bunt, H. (2004). The DIAMOND Project. In CATA-LOG. find
  10. Griol, D., Iglesias, J. A., Ledezma, A., and Sanchis, A. (2016). A Two-Stage Combining Classifier Model for the Development of Adaptive Dialog Systems. International Journal of Neural Systems, 26(1):1650002. ↗
  11. Jurafsky, D., Shriberg, E., and Biasca, D. (1997). Switchboard SWBD-DAMSL Shallow-Discourse-Function Annotation Coder Manual. Technical Report Draft 13, Institute of Cognitive Science, University of Colorado. find
  12. Kiritchenko, S., Matwin, S., and Famili, A. F. (2005). Functional Annotation of Genes using Hierarchical Text Categorization. In BioLINK SIG. find
  13. Mikolov, T., Sutskever, I., Chen, K., Corrado, G. S., and Dean, J. (2013). Distributed Representations of Words and Phrases and their Compositionality. In NIPS, pages 3111–3119. ↗ our card
  14. Petukhova, V., Gropp, M., Klakow, D., Schmidt, A., Eigner, G., Topf, M., Srb, S., Motlicek, P., Potard, B., Dines, J., Deroo, O., Egeler, R., Meinz, U., and Liersch, S. (2014). The DBOX Corpus Collection of Spoken Human-Human and Human-Machine Dialogues. In LREC, pages 252– 258. find
  15. Prust, H., Minnen, G., and Beun, R.-J. (1985). Transcriptie Dialoogexperiment Juni/Juli 1984. Technical Report IPO Rapport 481, Eindhoven University of Technology. find
  16. Raux, A., Bohus, D., Langner, B., Black, A. W., and Eskenazi, M. (2006). Doing Research on a Deployed Spoken Dialogue System: One Year of Let’s Go! Experience. In INTERSPEECH, pages 65–68. find
  17. Ribeiro, E., Batista, F., Trancoso, I., Lopes, J., Ribeiro, R., and Martins de Matos, D. (2016). Assessing User Expertise in Spoken Dialog System Interactions. In Iber-SPEECH, pages 245–254. find
  18. Ribeiro, E., Ribeiro, R., and Martins de Matos, D. (2019a). A Multilingual and Multidomain Study on Dialog Act Recognition using Character-Level Tokenization. Information, 10(3):94. ↗
  19. Ribeiro, E., Ribeiro, R., and Martins de Matos, D. (2019b). Deep Dialog Act Recognition using Multiple Token, Segment, and Context Information Representations. Journal of Artificial Intelligence Research, 66:861–899. ↗ our card
  20. Schmitt, A., Ultes, S., and Minker, W. (2012). A Parameterized and Annotated Spoken Dialog Corpus of the CMU Let’s Go Bus Information System. In LREC, pages 3369– 3373. find
  21. Searle, J. R. (1969). Speech Acts: An Essay in the Philosophy of Language. Cambridge University Press. ↗ our card
  22. Sidorov, M., Brester, C., Semenkin, E., and Minker, W. (2014). Speaker State Recognition with Neural Network-Based Classification and Self-Adaptive Heuristic Feature Selection. In ICINCO, volume 1, pages 699–703. find
  23. Strik, H., Russel, A., van den Heuvel, H., Cucchiarini, C., and Boves, L. (1997). A Spoken Dialog System for the Dutch Public Transport Information Service. International Journal of Speech Technology, 2(2):119–129. ↗
  24. Ultes, S., Schmitt, A., and Minker, W. (2013). On Quality Ratings for Spoken Dialogue Systems – Experts vs. Users. In NAACL-HLT, pages 569–578. 10. find
  25. Language Resource References Harry Bunt and Kars Wijnhoven and Volha Petukhova and Andrei Malchanau and Alex Chengyu Fang. (2016). The DialogBank. Tilburg University, https://dialogbank.uvt.nl/. ↗ our card
  26. Eugénio Ribeiro and Ricardo Ribeiro and David Martins de Matos. (2019). LEGO-ISO. L 2 F - Spoken Language Systems Laboratory, INESC-ID Lisboa, https://gitlab.hlt.inesc-id.pt/eugenio/lego-iso/. ↗
  27. Alexander Schmitt and Stefan Ultes and Wolfgang Minker. (2012). LEGO Spoken Dialogue Corpus. University of Ulm, https://www.ultes.eu/ressources/lego-spoken-dialogue-corpus/. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/ribeiro-2020-mapping-dialog-act-annotations-lego/