Elamur Research⌕
☰
home / papers / 2019 / Target Based Speech Act Classification in Political Campaign Text

Target Based Speech Act Classification in Political Campaign Text

Paper · 2019 · Shivashankar Subramanian, Trevor Cohn and Timothy Baldwin

Summary

The paper looks at pragmatics in election campaign material. For each utterance it asks two questions: what the politician is doing (the speech act), and which party the utterance concerns (the target). The authors built a new corpus of press releases and speech transcripts from Labor and Liberal, the two major parties in the 2016 Australian federal election. They annotated it with an eight-class speech act scheme adapted to politics. The scheme separates pledges by type and specificity, and it adds past-action and verdictive categories. Each utterance also gets a target label: Labor, Liberal or none.

The models are biGRU utterance classifiers. The authors try ELMo contextual embeddings, cross-view training (CVT) on unlabeled campaign text, and the speaker's party as metadata. They test on gold utterances and also with automatic segmentation. The best speech act model is a semi-supervised biGRU with ELMo. For targets, adding speaker metadata gives a further gain.

This matters because pledge detection at different levels of specificity could support checking, after an election, whether parties kept their promises.

Key points

  • Speech act classes: assertive, commissive-action-specific, commissive-action-vague, commissive-outcome, directive, expressive, past-action, verdictive.
    • The distribution is skewed: assertive is 40.8%, verdictive 25.4%, directive only 1.7%.
  • Annotation agreement:
    • Utterance segmentation reaches Krippendorff's α = 0.84 on 15 double-annotated documents.
    • Per-class speech act κ ranges from 0.72 to 0.92.
    • Target party κ ranges from 0.86 to 0.92.
  • CVT setup: auxiliary student models fit the main model's predictions on unlabeled election speeches through a KL-divergence consensus loss. Three restricted-view variants were tested: forward GRU only, separate forward and backward GRUs, and word dropout.
  • Best configuration: biGRU ELMo + CVT worddrop + Meta.
    • Commissive-outcome is the hardest class.
    • Common confusions are assertive vs verdictive, vague vs specific action pledges, and Labor vs Liberal targets.
  • Downstream analysis:
    • Predicted acts were split by policy area, using a Logistic Regression classifier trained on ABC Fact Check data.
    • The resulting party emphasis tracks the Comparative Manifesto Project shares.
    • Specific pledges outnumber vague ones.
    • The dataset is released at github.com/shivashankarrs/Speech-Acts.

Interesting ideas

  • I1. Sentences are cut into utterances whenever the speech act or the target changes, so each unit has exactly one act and one addressee; this makes labels clean even for mixed sentences like "we pledge X and call on them to do Y".
  • I2. The utterance is tagged with what is being done and also with whom it is about, which separates the speaker from the party being promised to, demanded of or criticized.
  • I3. Who is speaking is used as a cheap feature to resolve targets that are never named, such as "we" or "our party".
  • I4. In cross-view training, weaker students that see only part of the input (one direction, or text with words dropped) must match the full model's predictions on unlabeled text, which regularizes it without any new labels.
  • I5. The classifier's output is checked against an independent external resource (manifesto sentence shares) instead of only held-out F1, which validates the system on real downstream use.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (frames): prag could split clauses on act/target change before assigning speech acts, so one UD sentence can produce several act frames, each with its own target.
  • I2 (frames): speech act frames could carry an explicit target slot filled through coreference, so a directive points to the agent expected to act.
  • I3 (frames): document-level speaker metadata could serve as a prior for resolving first-person pronouns inside quoted or authored text, before trying harder coreference cues.
  • I4 (learn): when silver data is scarce, partial-view variants of a parser or tagger could be trained to agree with the full model on raw text, and their disagreements could flag unstable sentences.
  • I5 (learn): evaluation could add aggregate checks against independent knowledge, for example whether extracted event or role distributions match known corpus statistics, as a sanity check alongside negative controls.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓The University of MelbourneMelbourne · Australia60 works · 21,199 citations

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 9,086 worksArtificial Intelligence · 8,889 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 worksSocial Sciences · 744 worksGeneral Social Sciences · 87 works

🧭 Topics 1Research subjects and languages the work is about.Speech act theory · 160 works

Cite

Cite as Shivashankar Subramanian, Trevor Cohn and Timothy Baldwin. 2019. Target Based Speech Act Classification in Political Campaign Text. Proceedings of the Eighth Joint Conference on Lexical and Computational Semantics (*SEM 2019).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/s19-1030 · arXiv 1905.07856

@inproceedings{subramanian2019target,
  title = {{Target Based Speech Act Classification in Political Campaign Text}},
  author = {Shivashankar Subramanian, Trevor Cohn and Timothy Baldwin},
  booktitle = {Proceedings of the Eighth Joint Conference on Lexical and Computational Semantics (*SEM 2019)},
  year = {2019},
  doi = {10.18653/v1/s19-1030},
  eprint = {1905.07856},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/s19-1030}
}

References

31 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Hugo Hernault, Danushka Bollegala, and Mitsuru Ishizuka. 2010. A sequential model for discourse segmentation. In Proceedings of the International Conference on Intelligent Text Processing and Computational Linguistics, pages 315–326. find
  2. J. Artés. 2011. Do Spanish politicians keep their election promises? Party Politics, 19(1):143–158. find
  3. J. L. Austin. 1962. How to do things with words. Clarendon Press, Oxford. find
  4. Mohit Iyyer, Varun Manjunatha, Jordan Boyd-Graber, and Hal Daumé III. 2015. Deep unordered composition rivals syntactic methods for text classification. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing, pages 1681–1691.
  5. Steven Bird, Ewan Klein, and Edward Loper. 2009. Natural Language Processing with Python: Analyzing Text with the Natural Language Toolkit. O’Reilly Media, Inc. find
  6. Jean Carletta. 1996. Assessing agreement on classification tasks: the kappa statistic. Computational Linguistics, 22(2):249–254. ↗
  7. Minwoo Jeong, Chin-Yew Lin, and Gary Geunbae Lee. 2009. Semi-supervised speech act recognition in emails and forums. In Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing, pages 1250–1259. find
  8. Kyunghyun Cho, Bart van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. Learning phrase representations using RNN encoder–decoder for statistical machine translation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing, pages 1724–1734. ↗
  9. Shafiq Joty and Enamul Hoque. 2016. Speech act modeling of written asynchronous conversations with task-specific embeddings and conditional structured Ashequl Qadir and Ellen Riloff. 2011. Classifying sentences as speech acts in message board posts. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 748– 758. models. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, pages 1746–1756.
  10. Shafiq Joty and Tasnim Mohiuddin. 2018. Modeling speech acts in asynchronous conversations: A neural-CRF approach. Computational Linguistics, pages 1–36. ↗
  11. Mehdi Sajjadi, Mehran Javanmardi, and Tolga Tas-dizen. 2016. Regularization with stochastic transformations and perturbations for deep semi-supervised learning. In Proceedings of the Advances in Neural Information Processing Systems, pages 1163–1171. find
  12. Klaus Krippendorff. 2011. Computing krippendorff’s alpha-reliability. Technical Report, University of Pennsylvania. find
  13. David Lazer, Alex Sandy Pentland, Lada Adamic, Sinan Aral, Albert Laszlo Barabasi, Devon Brewer, Nicholas Christakis, Noshir Contractor, James Fowler, Myron Gutmann, et al. 2009. Life in the network: The coming age of computational social science. Science, 323(5915):721. find
  14. J. R. Searle. 1976. A Taxonomy of Illocutionary Acts. Linguistic Agency University of Trier. find
  15. Matthew A. Shapiro, Libby Hemphill, and Jahna Otter-bacher. 2018. Updates to congressional speech acts on Twitter. In Proceedings of the American Political Science Association Annual Meeting. find
  16. Annie Louis and Ani Nenkova. 2011. Automatic identification of general and specific sentences by leveraging discourse annotations. In Proceedings of 5th International Joint Conference on Natural Language Processing, pages 605–613. find
  17. Robert Thomson, Terry Royed, and Elin Naurin. 2010. The Program-to-Policy Linkage: A Comparative Study of Election Pledges and Government Policies in the United States, the United Kingdom, the Netherlands and Ireland. In Proceedings of the American Political Science Association Annual Meeting. find
  18. Christopher Lucas, Richard A Nielsen, Margaret E Roberts, Brandon M Stewart, Alex Storer, and Dustin Tingley. 2015. Computer-assisted text analysis for comparative politics. Political Analysis, 23(2):254–277. ↗
  19. Andrea Volkens, Pola Lehmann, Theres Matthieß, Nicolas Merz, Sven Regel, and Bernhard Weßels. 2017. The Manifesto Data Collection. Manifesto Project (MRG/CMP/MARPOR). Version 2017b. Wissenschaftszentrum Berlin für Sozialforschung. find
  20. Luca Lugini and Diane Litman. 2017. Predicting specificity in classroom discussion. In Proceedings of the 12th Workshop on Innovative Use of NLP for Building Educational Applications, pages 52–61. find
  21. Soroush Vosoughi and Deb Roy. 2016. Tweet acts: A speech act classifier for Twitter. In Proceedings of the Tenth International AAAI Conference On Web And Social Media, pages 711–715. ↗
  22. Stefan Müller. 2018. Prospective and retrospective rhetoric: A new dimension of party competition and campaign strategies. In Manifesto Corpus Conference. find
  23. Xiang Wei, Boqing Gong, Zixia Liu, Wei Lu, and Liqiang Wang. 2018. Improving the improved training of Wasserstein GANs: A consistency term and its dual effect. In Proceedings of the International Conference on Learning Representations. ↗
  24. Nona Naderi and Graeme Hirst. 2018. Using context to identify the language of face-saving. In Proceedings of the 5th Workshop on Argument Mining, pages 111–120. find
  25. Renxian Zhang, Dehong Gao, and Wenjie Li. 2012. Towards scalable speech act recognition in Twitter: tackling insufficient training data. In Proceedings of the Workshop on Semantic Analysis in Social Media, pages 18–27.
  26. E. Naurin. 2011. Election Promises, Party Behaviour and Voter Perception. Palgrave Macmillan. find
  27. E. Naurin. 2014. Is a promise a promise? Election pledge fulfillment in comparative perspective using Sweden as an example. West European Politics. find
  28. Tianyu Zhao and Tatsuya Kawahara. 2017. Joint learning of dialog act segmentation and recognition in spoken dialog using neural networks. In Proceedings of the Eighth International Joint Conference on Natural Language Processing, pages 704–712.
  29. Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. GloVe: Global vectors for word representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing, pages 1532–1543.
  30. M Zimmermann, A Stolcke, and E Shriberg. 2006. Joint segmentation and classification of dialog acts in multiparty meetings. In Proceedings of the 2006 IEEE International Conference on Acoustics, Speech and Signal Processing, pages 581–584. find
  31. Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2227–2237.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/subramanian-2019-target-based-speech-act-classification/