Elamur Research⌕
☰
home / papers / 2013 / Improving the Quality of Minority Class Identification in Dialog Act Tagging

Improving the Quality of Minority Class Identification in Dialog Act Tagging

Paper · 2013 · Adinoyi Omuya, Vinodkumar Prabhakaran and Owen Rambow

Summary

The paper addresses a common weakness of dialog act (DA) taggers: they score well on overall accuracy but miss rare classes. In email, the rarest is REQUEST-ACTION. The authors start from the SVM tagger of Hu et al. (2009) and use the Enron-derived email corpus from that work, segmented into Dialog Functional Units (DFUs). Each DFU is labelled with one of four DAs.

They change the standard one-against-all multi-class SVM in two ways:

  • Per-class feature optimization. Each binary classifier gets its own feature set, tuned for its own F-measure.
  • Cascaded decision rule. The binary classifiers are consulted from the rarest class to the most frequent, and the first positive answer wins. Only when every classifier says no does the system fall back to picking the most confident one.

The final system reduces F-measure error on REQUEST-ACTION by 22.8% and overall error by about 10%. It also improves a downstream task that detects Overt Displays of Power. The broader lesson is that treating a multi-class SVM as an opaque package hides easy gains for minority classes.

Key points

  • Data. The corpus has 122 Enron email threads, 360 messages, 20,740 tokens and 1406 DFUs. The class distribution is very uneven:

    • REQUEST-ACTION: 35 (2.5%)
    • REQUEST-INFORMATION: 151
    • CONVENTIONAL: 357
    • INFORM: 853

    Labels occurring 5 or fewer times were dropped.

  • Setup. The classifiers are linear-kernel SVMs built with ClearTK's SVMLight wrapper and evaluated with 5-fold cross validation. POS tags come from the OpenNLP tagger.

  • Features fall into three groups:

    • lexical: word, POS and mixed n-grams, first/last POS and lemma, modal count, question mark;
    • verbal: position and identity of the first verb;
    • message-level: position and size of the DFU and the message.

    Feature selection is exhaustive within each group, followed by a search over combinations of groups.

  • Main results.

    SystemREQUEST-ACTION FAccuracy
    Baseline (BAS)40.791.3
    Per-class optimization (DAC)50.091.7
    DAC + rare-class preference (DAC MP)54.291.8
    DAC + full cascade (DAC CMP)54.292.2 (10.6% error reduction)

    With the full cascade, every other class improves too.

  • Extrinsic task (Overt Displays of Power).

    DA tags usedODP F-measure
    Gold DA tags65.8
    BAS tags52.6
    DAC CMP tags54.2
    No DA features50.0

    DAC CMP tags close 12.1% of the gap between BAS and gold, and give a 26.6% error reduction over using no DA features.

Interesting ideas

  • I1. Tuning a separate feature set for each one-vs-rest classifier lets each class use its own best evidence: word n-grams for INFORM and CONVENTIONAL, POS n-grams for the two request types.
  • I2. Confidence-based argmax drowns out rare classes, whose classifiers have weaker weights simply because they see less data; asking the rare-class classifiers first fixes this without retraining anything.
  • I3. Inspecting the top weights of each trained classifier shows interpretable cues, such as "please" for action requests and a question mark that counts for information requests but against INFORM.
  • I4. The extrinsic evaluation feeds predicted rather than gold tags into a downstream task, keeps the cross-validation folds strictly separate, and reports the result against both a gold-tag ceiling and a no-feature floor.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (frames): The speech-act detectors in prag could each keep their own small, inspectable feature set instead of one shared vector, which would make each decision easier to explain.
  • I2 (frames): Speech-act or irony decisions could be ordered as a cascade with rare acts like REQUEST-ACTION checked first, and the explanation trace would record which rule fired.
  • I3 (learn): After training, top-weight dumps per class could serve as a sanity check and as seeds for self-induced repair rules.
  • I4 (learn): When an upstream layer such as tagging or parsing is improved, its value could be measured by how much of the downstream gap between the floor and gold input it closes.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Columbia UniversityNew York · United States149 works · 48,852 citations

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 9,086 worksArtificial Intelligence · 8,889 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 works

🧭 Topics 2Research subjects and languages the work is about.Dialogue systems · 401 worksSpeech act theory · 160 works

Cite

Cite as Adinoyi Omuya, Vinodkumar Prabhakaran and Owen Rambow. 2013. Improving the Quality of Minority Class Identification in Dialog Act Tagging. Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{omuya2013improving,
  title = {{Improving the Quality of Minority Class Identification in Dialog Act Tagging}},
  author = {Adinoyi Omuya, Vinodkumar Prabhakaran and Owen Rambow},
  booktitle = {Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies},
  year = {2013},
  doi = {10.13140/2.1.1516.4163},
  url = {https://doi.org/10.13140/2.1.1516.4163}
}

References

16 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Apoorv Agarwal, Adinoyi Omuya, Aaron Harnly, and Owen Rambow. 2012. A Comprehensive Gold Standard for the Enron Organizational Hierarchy. In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 161–165, Jeju Island, Korea, July. Association for Computational Linguistics. find
  2. J. L. Austin. 1975. How to Do Things with Words. Harvard University Press, Cambridge, Mass. ↗
  3. William W. Cohen, Vitor R. Carvalho, and Tom M. Mitchell. 2004. Learning to Classify Email into “Speech Acts” . In Dekang Lin and Dekai Wu, editors, Proceedings of EMNLP 2004, pages 309–316, Barcelona, Spain, July. Association for Computational Linguistics.
  4. Isabelle Guyon, Jason Weston, Stephen Barnhill, and Vladimir Vapnik. 2002. Gene Selection for Cancer Classification using Support Vector Machines. Mach. Learn., 46:389–422, March. ↗
  5. Jun Hu, Rebecca Passonneau, and Owen Rambow. 2009. Contrasting the Interaction Structure of an Email and a Telephone Corpus: A Machine Learning Approach to Annotation of Dialogue Function Units. In Proceedings of the SIGDIAL 2009 Conference, London, UK, September. Association for Computational Linguistics. find
  6. Thorsten Joachims. 1999. Making Large-Scale SVM Learning Practical. In Bernhard Schölkopf, Christopher J.C. Burges, and A. Smola, editors, Advances in Kernel Methods - Support Vector Learning, Cambridge, MA, USA. MIT Press. ↗
  7. J. Kim, G. Chern, D. Feng, E. Shaw, and E. Hovy. 2006. Mining and Assessing Discussions on the Web Through Speech Act Analysis. In Proceedings of the Workshop on Web Content Mining with Human Language Technologies at the 5th International Semantic Web Conference. find
  8. S.N. Kim, L. Cavedon, and T. Baldwin. 2010a. Classifying Dialogue Acts in One-on-one Live Chats. In Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing, pages 862–871. Association for Computational Linguistics.
  9. S.N. Kim, L. Wang, and T. Baldwin. 2010b. Tagging and Linking Web Forum Posts. In Proceedings of the Fourteenth Conference on Computational Natural Language Learning, pages 192–202. Association for Computational Linguistics. find
  10. Hsuan-Tien Lin, Chih-Jen Lin, and Ruby C. Weng. 2007. A Note on Platt’s Probabilistic Outputs for Support Vector Machines. Mach. Learn., 68:267–276, October. ↗
  11. Philip V. Ogren, Philipp G. Wetzler, and Steven Bethard. 2008. ClearTK: A UIMA toolkit for statistical natural language processing. In Towards Enhanced Interoperability for Large HLT Systems: UIMA for NLP workshop at Language Resources and Evaluation Conference (LREC). find
  12. Vinodkumar Prabhakaran, Owen Rambow, and Mona Diab. 2012. Predicting Overt Display of Power in Written Dialogs. In Human Language Technologies: The 2012 Annual Conference of the North American Chapter of the Association for Computational Linguistics, Montreal, Canada, June. Association for Computational Linguistics. find
  13. J.R. Searle. 1976. A Classification of Illocutionary Acts. Language in society, 5(01):1–23. ↗
  14. A. Stolcke, K. Ries, N. Coccaro, E. Shriberg, R. Bates, D. Jurafsky, P. Taylor, R. Martin, C.V. Ess-Dykema, and M. Meteer. 2000. Dialogue Act Modeling for Automatic Tagging and Recognition of Conversational Speech. Computational linguistics, 26(3):339–373. ↗
  15. J.Y. Yeh and A. Harnly. 2006. Email Thread Reassembly Using Similarity Matching. In Third Conference on Email and Anti-Spam (CEAS), pages 27–28. ↗
  16. R. Zhang, D. Gao, and W. Li. 2012. Towards Scalable Speech Act Recognition in Twitter: Tackling Insufficient Training Data. EACL 2012, page 18.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/omuya-2013-improving-quality-minority-class-identification/