Elamur Research⌕
☰
home / papers / 2012 / Behind the Article: Recognizing Dialog Acts in Wikipedia Talk Pages

Behind the Article: Recognizing Dialog Acts in Wikipedia Talk Pages

Summary

Wikipedia Talk pages are where editors decide how to improve an article. Before this work, no one had applied machine learning to classify these contributions, and no corpus captured this kind of coordination. The authors built a dialog act schema focused on article improvement. They based it on a manual study of thirty Simple English Wikipedia Talk pages and on an existing taxonomy of information quality problems. The schema has 17 labels in four groups: article criticism, explicit performatives, information content and interpersonal attitude. A single turn can serve several purposes, so annotation is multi-label at the turn level. Applying the schema to 100 Talk pages produced the SEWD corpus, which is freely available. The authors then trained separate classifiers for each label using Naive Bayes, J48 and SMO. They balanced the data by downsampling negative instances and selected features with χ² or Information Gain. The final pipeline reaches an average F1 of .82, close to human agreement (.79). This makes it feasible to link discussions with concrete edits and to show readers what is being debated about an article.

Key points

  • The SEWD (Simple English Wikipedia Discussion) corpus contains 100 segmented, annotated Talk pages, which is roughly 15% of all relevant Talk pages in Simple English Wikipedia.
  • The 17-label tagset includes criticism types (e.g., CM, CW, CU, CL, COBJ), performatives (PSR, PREF, PFC, PPC), information labels (IP, IS, IC) and attitude labels (ATT+, ATTP, ATT-).
  • With all features, micro-averaged F1 was 0.57 for Naive Bayes, 0.66 for J48 and 0.72 for SMO. Selecting the top 100, 200 or 300 features per label raised the best pipeline to .73 macro and .82 micro.
  • Indentation, time since the previous turn, and turn position within a topic were the most useful non-lexical features. Explicit reference features did not help.
  • The labels CU and CO had low inter-annotator agreement, even though classifiers handled them reasonably well. The authors conclude that the definitions of these two labels need refinement.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Speech act theory

Cited by

23 papers of our library cite it (of the 140673 whose bibliographies we read); 2 of them have a page here, 21 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{ferschke2012behind,
  title = {{Behind the Article: Recognizing Dialog Acts in Wikipedia Talk Pages}},
  author = {Ferschke, Oliver and Gurevych, Iryna and Chebotar, Yevgen},
  booktitle = {Proceedings of the 13th Conference of the European Chapter of the Association for Computational Linguistics},
  year = {2012},
  url = {https://aclanthology.org/E12-1079/}
}

References

33 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ron Artstein and Massimo Poesio. 2008. Inter-Coder Agreement for Computational Linguistics. Computational Linguistics, 34(4):555–596, December. ↗
  2. John L. Austin. 1962. How to Do Things with Words. Clarendon Press, Cambridge, UK. find
  3. Emily M. Bender, Jonathan T. Morgan, Meghan Oxley, Mark Zachry, Brian Hutchinson, Alex Marin, Bin Zhang, and Mari Ostendorf. 2011. Annotating Social Acts: Authority Claims and Alignment Moves in Wikipedia Talk Pages. In Proceedings of the Workshop on Language in Social Media, pages 48–57, Portland, Oregon, USA. find
  4. Jean Carletta. 1996. Assessing Agreement on Classification Tasks: The Kappa Statistic. Computational Linguistics, 22(2):249–254. ↗ our card
  5. Tamitha Carpenter and Emi Fujioka. 2011. The Role and Identification of Dialog Acts in Online Chat. In Proceesings of the Workshop on Analyzing Microtext at the 25th AAAI Conference on Artificial Intelligence, San Francisco, CA, USA. find
  6. William W. Cohen, Vitor R. Carvalho, and Tom M. Mitchell. 2004. Learning to Classify Email into ”Speech Acts”. In Proceedings of the 2004 Conference on Empirical Methods in Natural Language Processing, pages 309–316, Barcelona, ES. find our card
  7. Mark G. Core and James F. Allen. 1997. Coding dialogs with the DAMSL annotation scheme. In Proceedings of the Working Notes of the AAAI Fall Symposium on Communicative Action in Humans and Machines, pages 28–35, Cambridge, MA, USA. find
  8. Han De Vries, Marc N. Elliott, David E. Kanouse, and Stephanie S. Teleki. 2008. Using Pooled Kappa to Summarize Interrater Agreement across Many Items. Field Methods, 20(3):272–282. ↗
  9. David Ferrucci and Adam Lally. 2004. UIMA: An Architectural Approach to Unstructured Information Processing in the Corporate Research Environment. Natural Language Engineering, 10:327–348. ↗
  10. Oliver Ferschke, Torsten Zesch, and Iryna Gurevych. 2011. Wikipedia Revision Toolkit: Efficiently Accessing Wikipedia’s Edit History. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies. System Demonstrations, pages 97–102, Portland, OR, USA. ↗
  11. Paul Grice. 1975. Logic and Conversation. In Peter Cole and Jerry L. Morgan, editors, Syntax and Semantics, volume 3. New York: Academic Press. ↗
  12. Mark Hall, Eibe Frank, Geoffrey Holmes, Bernhard Pfahringer, Peter Reutemann, and Ian H. Witten. 2009. The WEKA Data Mining Software: An Update. SIGKDD Explorations, 11:10–18. ↗
  13. George Hripcsak and Adam S. Rothschild. 2005. Agreement, the f-measure, and reliability in information retrieval. Journal of the American Medical Informatics Association, 12(3):296–298. ↗
  14. Dan Jurafsky, Liz Shriberg, and Debbra Biasca. 1997. Switchboard SWBD-DAMSL Shallow-Discourse-Function Annotation Coders Manual. Technical Report Draft 13, University of Colorado, Institute of Cognitive Science. find
  15. Jihie Kim, Jia Li, and Taehwan Kim. 2010a. Towards Identifying Unresolved Discussions in Student Online Forums. In Proceedings of the NAACL HLT 2010 Fifth Workshop on Innovative Use of NLP for Building Educational Applications, pages 84– 91, Los Angeles, CA, USA. find
  16. Su Nam Kim, Li Wang, and Timothy Baldwin. 2010b. Tagging and linking web forum posts. In Proceedings of the Fourteenth Conference on Computational Natural Language Learning, CoNLL ’10, pages 192–202, Stroudsburg, PA, USA. find
  17. Klaus Krippendorff. 1980. Content Analysis: An Introduction to Its Methodology. Thousand Oaks, CA: Sage Publications. find
  18. J. Richard Landis and Gary G. Koch. 1977. An Application of Hierarchical Kappa-type Statistics in the Assessment of Majority Agreement among Multiple Observers. Biometrics, 33(2):363–374, June. ↗
  19. David Laniado, Riccardo Tasso, Yana Volkovich, and Andreas Kaltenbrunner. 2011. When the Wikipedians Talk: Network and Tree Structure of Wikipedia Discussion Pages. In Proceedings of the 5th International AAAI Conference on Weblogs and Social Media, Dublin, IE. find
  20. Tom Mitchell. 1997. Machine Learning. McGraw-Hill Education (ISE Editions), 1st edition. find
  21. Rebecca Passonneau. 2006. Measuring Agreement on Set-valued Items (MASI) for Semantic and Pragmatic Annotation. In Proceedings of the Fifth International Conference on Language Resources and Evaluation, Genoa, IT. find our card
  22. John C. Platt. 1998. Fast training of support vector machines using sequential minimal optimization. In Advances in Kernel Methods: Support Vector Learning, pages 185–208, Cambridge, MA, USA. ↗
  23. Ilona R. Posner and Ronald M. Baecker. 1992. How People Write Together. In Proceedings of the 25th Hawaii International Conference on System Sciences, pages 127–138, Wailea, Maui, HI, USA. find
  24. Ross Quinlan. 1992. C4.5: Programs for Machine Learning. Morgan Kaufmann, 1st edition. find our card
  25. Jodi Schneider, Alexandre Passant, and John G. Breslin. 2011. Understanding and Improving Wikipedia Article Discussion Spaces. In Proceedings of the 26th Symposium on Applied Computing, Taichung, TW. find
  26. John R. Searle. 1969. Speech Acts. Cambridge University Press, Cambridge, UK. find our card
  27. John R. Searle. 1976. A classification of illocutionary acts. Language in Society, 5:1–23. ↗
  28. Besiki Stvilia, Michael B. Twidale, Linda C. Smith, and Les Gasser. 2008. Information Quality Work Organization in Wikipedia. Journal of the American Society for Information Science, 59:983–1001. ↗
  29. Grigorios Tsoumakas, Ioannis Katakis, and Ioannis P. Vlahavas. 2010. Mining multi-label data. In Data Mining and Knowledge Discovery Handbook, pages 667–685. Springer. ↗
  30. Fernanda Viégas, Martin Wattenberg, Jesse Kriss, and Frank Ham. 2007. Talk Before You Type: Coordination in Wikipedia. In Proceedings of the 40th Annual Hawaii International Conference on System Sciences, Waikoloa, Big Island, HI, USA. find
  31. Eti Yaari, Shifra Baruchson-Arbib, and Judit Bar-Ilan. 2011. Information quality assessment of community generated content: A user study of Wikipedia. Journal of Information Science, 37:487–498. ↗
  32. Yiming Yang and Jan O. Pedersen. 1997. A Comparative Study on Feature Selection in Text Categorization. In Proceedings of the Fourteenth International Conference on Machine Learning, pages 412–420, San Francisco, CA, USA. find
  33. Torsten Zesch, Christof Müller, and Iryna Gurevych. 2008. Extracting Lexical Semantic Knowledge from Wikipedia and Wiktionary. In Proceedings of the 6th International Conference on Language Resources and Evaluation, Marrakech, MA. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/papers/ferschke-2012-behind-article-recognizing-dialog-acts/