Elamur Research⌕
☰
home / papers / 2018 / Don't get Lost in Negation: An Effective Negation Handled Dialogue Acts Prediction Algorithm for Twitter Customer Service Conversations

Don't get Lost in Negation: An Effective Negation Handled Dialogue Acts Prediction Algorithm for Twitter Customer Service Conversations

Paper · 2018 · Mansurul Bhuiyan, Amita Misra, Saurabh Tripathy et al.

Summary

Companies increasingly handle customer support on Twitter, and virtual agents need to know what each turn is doing: asking, complaining, thanking, and so on. Earlier dialogue-act classifiers for this setting ignored negation, even though "working" and "not working" call for very different replies. The authors gathered 2235 customer-care tweets that each contain at least one explicit negation cue. Crowd workers labelled them with 12 acts taken from the 25-act Twitter customer-service taxonomy of Oraby et al. The team then built a per-utterance SVM classifier. Its features change depending on whether words fall inside or outside a detected negation scope. To find scope, they tried several lightweight heuristics (POS-based, first emotional word, punctuation, fixed windows) and compared them with a CRF scope detector trained on formal text. Some form of negation handling raised F-score for almost every act. Simple windows came out ahead most often, and the formal-text CRF did poorly on tweets. The work suggests that cheap, robust rules can beat a stronger model that was trained on a different register.

Key points

  • Corpus: 2235 negated customer-care utterances. Each was labelled by 5 CrowdFlower annotators and resolved by majority vote, with average agreement of 73.3%. The label set is the 12 most frequent acts from the taxonomy in [20].
  • Cue lexicon: a fixed list of explicit negators (no, not, never, nor, without, barely, seldom, contracted forms written without apostrophes, etc.). Affixal negation (un-, dis-, -less) is deliberately left out.
  • Scope heuristics:
    • POS-based: take the first noun, adjective or non-auxiliary verb after the cue, skipping stopwords and stopping at punctuation or another cue.
    • First emotional word: take the first emotional word, found with sentiment lexicons.
    • Window: take X words before and Y words after the cue, with X and Y each from 0 to 3. The -1/+2 window was best.
  • Features: sentiment scores from five lexicons (including Sentiment140, NRC Hashtag and NRC Emotion), aggregated as min/max/mean/sum, computed separately inside and outside the scope. Evaluation used 5-fold CV (3 train / 1 tune / 1 test).
  • Results:
    • Negation handling helped every act except socialact_thanks (0.93 without vs 0.89 with windows).
    • Example gains: answer_other 0.21→0.34, question_wh 0.64→0.72, question_yesno 0.39→0.45.
    • The CRF, trained on SEM 2012 Conan Doyle data, lagged the heuristics.
    • The non-sequential model matched or beat the sequential model of [20] on most acts, because many of the conversations are only one or two turns long.

Interesting ideas

  • I1. Instead of one sentiment profile per utterance, compute it twice, for words inside and outside the negation scope, so the classifier sees that "happy" under "not" behaves differently.
  • I2. Pick the scope heuristic (window size, POS rule) by how much it helps the downstream classifier, rather than by agreement with scope annotations that do not exist for the target domain.
  • I3. A tool trained on formal fiction lost to crude rules on informal tweets, showing that register mismatch can erase the advantage of a learned model.
  • I4. The POS-based rule takes the first content word after the cue, skips stopwords, and stops at punctuation or a second cue, giving a minimal and cheap scope guess.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (frames): speech-act and implicature rules in prag could treat polarity-bearing words under a negation scope as a separate signal from unnegated ones, instead of averaging them.
  • I2 (learn): when tuning a component without gold data in a new register, use the gain on a downstream reader as the selection criterion, with a no-component run as the negative control.
  • I3 (learn): evaluations should include an explicit out-of-register test slice, so that silver-data-trained parsers or taggers are not trusted blindly on chatty text.
  • I4 (tree): this could serve as an explainable fallback for negation attachment when the UD parse is low-confidence or broken on noisy input.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 9,086 worksArtificial Intelligence · 8,889 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 works

🧭 Topics 2Research subjects and languages the work is about.Dialogue systems · 401 worksSpeech act theory · 160 works

Cite

Cite as Mansurul Bhuiyan, Amita Misra, Saurabh Tripathy, Jalal Mahmud and Rama Akkiraju. 2018. Don't get Lost in Negation: An Effective Negation Handled Dialogue Acts Prediction Algorithm for Twitter Customer Service Conversations. arXiv:1807.06107.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1807.06107 · arXiv 1807.06107

@misc{bhuiyan2018don,
  title = {{Don't get Lost in Negation: An Effective Negation Handled Dialogue Acts Prediction Algorithm for Twitter Customer Service Conversations}},
  author = {Mansurul Bhuiyan, Amita Misra, Saurabh Tripathy, Jalal Mahmud and Rama Akkiraju},
  year = {2018},
  doi = {10.48550/arxiv.1807.06107},
  eprint = {1807.06107},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1807.06107}
}

References

31 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. H. Bunt, J. Alexandersson, J. Carletta, J.-W. Choe, A. C. Fang, K. Hasida, K. Lee, V. Petukhova, A. Popescu-Belis, L. Romary, C. Soria, and D. Traum. Towards an ISO standard for dialogue act annotation. Seventh conference on International Language Resources and Evaluation (LREC’10), pages 2548–2555, 2010.
  2. M. G. Core and J. Allen. Coding dialogs with the damsl annotation scheme. In AAAI fall symposium on communicative action in humans and machines, volume 56. Boston, MA, 1997. find
  3. I. G. Councill, R. McDonald, and L. Velikovich. What’s great and what’s not: Learning to classify the scope of negation for improved sentiment analysis. In Proceedings of the Workshop on Negation and Speculation in NIPS, pages 51–59, 2010. find
  4. X. Ding, B. Liu, and P. S. Yu. A holistic lexicon-based approach to opinion mining. In Proceedings of the 2008 WSDM conference, pages 231–240. ACM, 2008. find
  5. F. Fancellu, A. Lopez, and B. Webber. Neural Networks For Negation Scope Detection, pages 495–504. Association for Computational Linguistics, 2016. find
  6. A. Hogenboom, P. Van Iterson, B. Heerschop, F. Frasincar, and U. Kaymak. Determining negation scope and strength in sentiment analysis. In IEEE SME Conference, pages 2589–2594, 2011. find
  7. E. Ivanovic. Dialogue act tagging for instant messaging chat sessions. Proceedings of the ACL Student Research Workshop on - ACL ’05, (June):79, 2005.
  8. E. Ivanovic. Automatic instant messaging dialogue using statistical models and dialogue acts. Masters Research thesis, 2008. find
  9. D. Jurafsky, E. Shriberg, and D. Biasca. Switchboard-damsl labeling project coder’s manual, draft 13. technical report 97-02, university of colorado, institute of cognitive science. boulder, co., 1997. find
  10. S. N. Kim, L. Cavedon, and T. Baldwin. Classifying Dialogue Acts in Multi-party Live Chats. 26th Pacific Asia Conference on Language, Information and Computation, pages 463–472, 2012.
  11. S. N. Kim, L. Cavedon, and T. Baldwin. Classifying Dialogue Acts in One-on-one Live Chats. 26th Pacific Asia Conference on Language, Information and Computation, (October):463–472, 2012.
  12. Y. Kim. Convolutional neural networks for sentence classification. arXiv preprint arXiv:1408.5882, 2014. ↗
  13. S. Kiritchenko, X. Zhu, and S. M. Mohammad. Sentiment analysis of short informal texts. Journal of Artificial Intelligence Research, 50:723–762, 2014. ↗
  14. T. Klüwer, H. Uszkoreit, and F. Xu. Using syntactic and semantic based relations for dialogue act recognition. Proceedings of COLING 2010, (August):570–578, 2010.
  15. N. Konstantinova, S. C. De Sousa, N. P. C. Díaz, M. J. M. López, M. Taboada, and R. Mitkov. A review corpus annotated for negation, speculation and their scope. In LREC, pages 3190–3195, 2012. find
  16. E. Lapponi, E. Velldal, L. Ovrelid, and J. Read. Uio2: Sequence-labeling negation using dependency features. In Proceedings of the First Joint Conference on Lexical and Computational Semantics, SemEval ’12, 2012. find
  17. S. M. Mohammad and P. D. Turney. Emotions evoked by common words and phrases: Using mechanical turk to create an emotion lexicon. In Proceedings of the NAACL HLT 2010 workshop, pages 26–34, 2010.
  18. R. Morante and W. Daelemans. A metalearning approach to processing the scope of negation. In Proceedings of the Thirteenth Conference on Computational Natural Language Learning, pages 21–29, 2009. find
  19. R. Morante, S. Schrauwen, and W. Daelemans. Annotation of negation cues and their scope: Guidelines v1. Computational linguistics and psycholinguistics technical report series, CTRS-003, 2011. find
  20. S. Oraby, P. Gundecha, J. Mahmud, M. Bhuiyan, and R. Akkiraju. "how may i help you?": Modeling twitter customer serviceconversations using fine-grained dialogue acts. In Proceedings of the 22Nd IUI Conference, pages 343–355, 2017. find
  21. W. Packard, E. M. Bender, J. Read, S. Oepen, and R. Dridan. Simple negation scope resolution through deep parsing: A semantic solution to a semantic problem. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics, pages 69–78, Baltimore, USA, 2014. find
  22. J. Read, E. Velldal, L. Øvrelid, and S. Oepen. Uio1: Constituent-based discriminative ranking for negation resolution. In E. Agirre, J. Bos, and M. T. Diab, editors, Pro. of the 1st Conf. on Lexical and Computational Semantics, pages 310–318, 2012. find
  23. J. Reitan, J. Faret, B. Gamback, and L. Bungum. Negation scope detection for twitter sentiment analysis. In WASSA@EMNLP, 2015. find
  24. J. Reitan, J. Faret, B. Gambäck, and L. Bungum. Negation scope detection for twitter sentiment analysis. Association for Computational Linguistics, 2015. find
  25. A. Schiffrin. Modelling Speech Acts in Conversational Discourse. (May), 2005. find
  26. G. Szarvas, V. Vincze, R. Farkas, and J. Csirik. The bioscope corpus: Annotation for negation, uncertainty and their scope in biomedical texts. In Proceedings of the Workshop on Current Trends in Biomedical Natural Language Processing, BioNLP ’08, pages 38–45, 2008.
  27. G. Tur, U. Guz, and D. Hakkani-Tür. Model adaptation for dialog act tagging. 2006 IEEE ACL Spoken Language Technology Workshop, SLT 2006, Proceedings, pages 94–97, 2006. find
  28. S. Vosoughi and D. Roy. Tweet acts: A speech act classifier for twitter. arXiv preprint arXiv:1605.05156, 2016. ↗
  29. J. P. White. Uwashington: Negation resolution using machine learning methods. In *SEM 2012: The First Joint Conference on Lexical and Computational Semantics – Volume 1: Proceedings of the main conference and the shared task, and Volume 2: Proceedings of the Sixth International Workshop on Semantic Evaluation (SemEval 2012), pages 335–339. Association for Computational Linguistics, 2012. find
  30. T. Wilson, J. Wiebe, and P. Hoffmann. Recognizing contextual polarity in phrase-level sentiment analysis. In Proceedings of the Conference on Human Language Technology and Empirical Methods in Natural Language Processing, HLT ’05, pages 347–354, Stroudsburg, PA, USA, 2005. Association for Computational Linguistics.
  31. R. Zhang, D. Gao, and W. Li. What Are Tweeters Doing: Recognizing Speech Acts in Twitter. Analyzing Microtext, pages 86–91, 2011. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/bhuiyan-2018-don-t-get-lost-negation/