Elamur Research⌕
☰
home / papers / 2010 / Domain Adaptation with Unlabeled Data for Dialog Act Tagging

Domain Adaptation with Unlabeled Data for Dialog Act Tagging

Paper · 2010 · Anna Margolis, Karen Livescu and Mari Ostendorf

Summary

The paper tackles dialog act (DA) tagging when no labeled data exists for the target domain, or even for the target language. Utterances are sorted into four coarse classes: Statement, Question, Backchannel, and Incomplete. The authors run every train/test pairing among three conversational corpora: MRDA (meetings), Swbd (Switchboard DAMSL, telephone talk between strangers), and SpCH (Spanish Callhome, calls among family and friends). When the languages differ, the test side is converted with Google Translate.

The classifier is a linear SVM (Liblinear) over n-gram counts plus an utterance-length feature. Two ways of exploiting unlabeled target data are compared:

  • Shared-feature restriction: keep only features seen in both domains.
  • Structural Correspondence Learning (SCL): a multi-view variant that splits each utterance into its first word and the remainder.

Translation costs little when train and test domains match. Cross-language transfer, however, hurts badly, mostly because backchannels go undetected. Both adaptation methods help the cross-language pairs, and they do so largely for the same reason: both learn that backchannel words go with short utterances. This is a caution that "adaptation gains" can reduce to a single simple cue.

Key points

  • Four classes are mapped consistently from three corpora: MRDA, Swbd (Swbd-DAMSL), and SpCH. Train sets are cut to the SpCH size of 29k utterances so comparisons are fair.
  • Features are unigrams, bigrams, and trigrams with utterance-boundary tags, plus z-normalized length. Punctuation and case are removed to mimic perfect ASR output. The classifier is a one-vs-rest L2-regularized linear SVM in Liblinear.
  • Testing on Swbd with SpCH training:
    • Baseline reaches 74.5% accuracy and 57.2% average per-class recall.
    • Shared-only reaches 77.4 / 64.2.
    • SCL reaches 76.8 / 64.8.
    • A length-only classifier gets 77.7 accuracy but only 48.2 recall.
    • The majority-class baseline is 67.7 / 25.0.
  • Across languages, English-trained models found only about 20% of translated Spanish backchannels. In-domain, backchannel detection exceeds 80%.
  • SCL setup:
    • Pivots are chosen by SCL-MI: a feature qualifies if it appears no fewer than 10 times in both domains and is among the top 500 MI features for a class.
    • SVD keeps at most 100 dimensions per view.
    • SCL hurt the Swbd/MRDA pair noticeably; for example, Swbd→MRDA recall fell from 71.6 to 68.1.

Interesting ideas

  • I1. Adding a classifier that uses only one trivial cue (utterance length) exposed that both sophisticated adaptation methods were mostly exploiting that cue, which is a cheap way to check what an "improvement" really comes from.
  • I2. Splitting an utterance down the middle made the two halves predict each other mainly through length, so the learned representation collapsed onto length; choosing the split as first word versus the rest avoided this.
  • I3. The first word carries much of the dialog-act signal (question words, backchannel tokens), so position-aware features beat a plain bag of n-grams for short conversational turns.
  • I4. Translating both train and test in-domain served as a control: the small loss showed that machine translation preserves most dialog-act information, which separated translation damage from domain damage.
  • I5. Restricting a model to features attested in both domains is a simple, label-free adaptation baseline that was often as good as SCL and more robust on close domains.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Every new silver-data or repair-rule gain could be compared against a single-shallow-feature control (length, first token) as a standard negative control.
  • I2 (learn): When building agreement or co-training views, check whether the views share a trivial confound before trusting their mutual predictions.
  • I3 (frames): The speech-act classifier in prag could treat the head and first token of the UD tree as explicit, explainable cues for Question vs Backchannel vs Statement.
  • I4 (back): Round-tripping text through translation via the meta-world and re-tagging speech acts could measure how much pragmatic force survives generation.
  • I5 (tree): When moving the perceptron parser to a new genre, pruning to features seen in both the source treebank and unlabeled target text is a cheap first baseline before heavier adaptation.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of WashingtonSeattle · United States263 works · 180,581 citations🎓Toyota Technological Institute at ChicagoChicago · United States29 works · 10,273 citations

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 9,086 worksArtificial Intelligence · 8,889 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 works

🧭 Topics 2Research subjects and languages the work is about.Dialogue systems · 401 worksSpeech act theory · 160 works

Cite

Cite as Anna Margolis, Karen Livescu and Mari Ostendorf. 2010. Domain Adaptation with Unlabeled Data for Dialog Act Tagging. Proceedings of the 2010 Workshop on Domain Adaptation for Natural Language Processing.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{margolis2010domain,
  title = {{Domain Adaptation with Unlabeled Data for Dialog Act Tagging}},
  author = {Anna Margolis, Karen Livescu and Mari Ostendorf},
  booktitle = {Proceedings of the 2010 Workshop on Domain Adaptation for Natural Language Processing},
  year = {2010},
  url = {https://aclanthology.org/W10-2607/}
}

References

26 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Anthony Aue and Michael Gamon. 2005. Customizing sentiment classifiers to new domains: a case study. In Proc. International Conference on Recent Advances in NLP. find
  2. Nuria Bel, Cornelis H. A. Koster, and Marta Villegas. 2003. Cross-lingual text categorization. In Research and Advanced Technology for Digital Libraries, pages 126–139. Springer Berlin / Heidelberg. find
  3. John Blitzer, Ryan McDonald, and Fernando Pereira. 2006. Domain adaptation with structural correspondence learning. In Proc. of the 2006 Conference on Empirical Methods in Natural Language Processing, pages 120–128.
  4. John Blitzer, Mark Dredze, and Fernando Pereira. 2007. Biographies, Bollywood, boom-boxes and blenders: Domain adaptation for sentiment classification. In Proceedings of the 45th Annual Meeting of the Association of Computational Linguistics, pages 440–447.
  5. John Blitzer, Dean P. Foster, and Sham M. Kakade. 2009. Zero-shot domain adaptation: A multi-view approach. Technical report, Toyota Technological Institute TTI-TR-2009-1. find
  6. Mark G. Core and James F. Allen. 1997. Coding dialogs with the DAMSL annotation scheme. In Proc. of the Working Notes of the AAAI Fall Symposium on Communicative Action in Humans and Machines. find
  7. Mark Dredze, John Blitzer, Partha Pratim Talukdar, Kuzman Ganchev, João Graca, and Fernando Pereira. 2007. Frustratingly hard domain adaptation for dependency parsing. In Proc. of the CoNLL Shared Task Session of EMNLP-CoNLL 2007, pages 1051–1055.
  8. Rong-En Fan, Kai-Wei Chang, Cho-Jui Hsieh, Xiang-Rui Wang, and Chih-Jen Lin. 2008. Liblinear: A library for large linear classification. Journal of Machine Learning Research, 9:1871–1874. ↗
  9. Umit Guz, Gokhan Tur, Dilek Hakkani-Tür, and Sébastien Cuendet. 2010. Cascaded model adaptation for dialog act segmentation and tagging. Computer Speech & Language, 24(2):289–306. ↗
  10. Jiayuan Huang, Alexander J. Smola, Arthur Gretton, Karsten M. Borgwardt, and Bernhard Schölkopf. 2007. Correcting sample selection bias by unlabeled data. In Advances in Neural Information Processing Systems 19, pages 601–608. ↗
  11. Minwoo Jeong, Chin Y. Lin, and Gary G. Lee. 2009. Semi-supervised speech act recognition in emails and forums. In Proc. of the 2009 Conference on Empirical Methods in Natural Language Processing, pages 1250–1259. find
  12. Dan Jurafsky, Liz Shriberg, and Debra Biasca. 1997. Switchboard SWBD-DAMSL shallow-discourse-function annotation coders manual, draft 13. Technical report, University of Colorado at Boulder Technical Report 97-02. find
  13. Lori Levin, Ann Thymé-Gobbel, Alon Lavie, Klaus Ries, and Klaus Zechner. 1998. A discourse coding scheme for conversational Spanish. In Proc. The 5th International Conference on Spoken Language Processing, pages 2335–2338. ↗
  14. Yang Liu. 2006. Using SVM and error-correcting codes for multiclass dialog act classification in meeting corpus. In Proc. Interspeech, pages 1938–1941. find
  15. Sinno J. Pan, James T. Kwok, and Qiang Yang. 2008. Transfer learning via dimensionality reduction. In Proc. of the Twenty-Third AAAI Conference on Artificial Intelligence. find
  16. Brian Roark and Michiel Bacchiani. 2003. Supervised and unsupervised PCFG adaptation to novel domains. In Proc. of the 2003 Conference of the North American Chapter of the Association for Computational Linguistics on Human Language Technology, pages 126–133. find
  17. Sophie Rosset, Delphine Tribout, and Lori Lamel. 2008. Multi-level information and automatic dialog act detection in human–human spoken dialogs. Speech Communication, 50(1):1–13. ↗
  18. Elizabeth Shriberg, Raj Dhillon, Sonali Bhagat, Jeremy Ang, and Hannah Carvey. 2004. The ICSI meeting recorder dialog act (MRDA) corpus. In Proc. of the 5th SIGdial Workshop on Discourse and Dialogue, pages 97–100. ↗
  19. Andreas Stolcke, Klaus Ries, Noah Coccaro, Elizabeth Shriberg, Rebecca Bates, Daniel Jurafsky, Paul Taylor, Rachel Martin, Carol Van Ess-Dykema, and Marie Meteer. 2000. Dialogue act modeling for automatic tagging and recognition of conversational speech. Computational Linguistics, 26:339–373. ↗
  20. Dinoj Surendran and Gina-Anne Levow. 2006. Dialog act tagging with support vector machines and hidden Markov models. In Proc. Interspeech, pages 1950– 1953. find
  21. Gokhan Tur, Dilek Hakkani-Tür, and Robert E. Schapire. 2005. Combining active and semi-supervised learning for spoken language understanding. Speech Communication, 45(2):171–186. ↗
  22. Gokhan Tur, Umit Guz, and Dilek Hakkani-Tür. 2007. Model adaptation for dialog act tagging. In Proc. IEEE Spoken Language Technology Workshop, pages 94–97. find
  23. Gokhan Tur. 2005. Model adaptation for spoken language understanding. In Proc. IEEE International Conference on Acoustics, Speech, and Signal Processing, pages 41–44. find
  24. Anand Venkataraman, Luciana Ferrer, Andreas Stolcke, and Elizabeth Shriberg. 2003. Training a prosody-based dialog act tagger from unlabeled data. In Proc. IEEE International Conference on Acoustics, Speech, and Signal Processing, volume I, pages 272–275. find
  25. Nick Webb and Ting Liu. 2008. Investigating the portability of corpus-derived cue phrases for dialogue act classification. In Proc. of the 22nd International Conference on Computational Linguistics (Coling 2008), pages 977–984. find
  26. Dikan Xing, Wenyuan Dai, Gui-Rong Xue, and Yong Yu. 2007. Bridged refinement for transfer learning. In Knowledge Discovery in Databases: PKDD 2007, pages 324–335. Springer Berlin / Heidelberg. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/margolis-2010-domain-adaptation-unlabeled-data-dialog/