Elamur Research⌕
☰
home / papers / 2020 / Augmenting Small Data to Classify Contextualized Dialogue Acts for Exploratory Visualization

Augmenting Small Data to Classify Contextualized Dialogue Acts for Exploratory Visualization

Paper · 2020 · Abhinav Kumar, Barbara Di Eugenio, Jillian Aurisano et al.

Summary

The authors are building an assistant that helps people explore data through visualizations. They want it to recognize what users ask for, even when a request sits inside a stretch of thinking aloud. They use CHICAGO-CRIME-VIS, a corpus of their own: 16 subjects talked to a hidden human Visualization Expert while studying Chicago crime data to decide where police should be deployed. Dialogues are split into contextualized actionable requests (CARs). Each CAR has three parts: the setup talk before a request, the actionable request (AR) itself, and the concluding talk after the expert responds. The corpus is small, so the authors enlarge the training data with paraphrase-based augmentation. Slots in the paraphrases are filled by heuristic rules instead of manual annotation. Classification has two layers, with the lower layer conditioned on the upper layer's decision. They compare a Balanced Bagging Classifier, a CRF and several LSTMs. Every model improves with augmentation, and the CRF performs best. LSTMs gain only modestly, even with embeddings trained on the domain. The takeaway is that on small data, cheaper traditional models can still be the better choice.

Key points

  • Corpus size: CHICAGO-CRIME-VIS has 3,179 transcribed utterances, with 38,105 tokens and 1,879 word types. Users interacted through speech and gesture in front of a large display.
  • Annotation: 449 utterances were coded as ARs across 8 dialogue-act types, with intercoder agreement κ = 0.74.
  • Rare classes dropped: APPEARANCE (7 cases) and HIGH-LEVEL-QUERY (2 cases) were excluded. The remaining CARs contain 440 ARs and 1,096 think-aloud utterances, which is 1,536 of the 3,179 utterances.
  • CAR shape: Setup averages about 1.8 utterances and conclusion about 2. A window starts and ends at utterances that mention a data attribute.
  • Results: Augmentation helped BAGC, CRF and the LSTMs alike. The CRF beat all the neural variants while being much cheaper to train.

Interesting ideas

  • I1. Think-aloud talk before and after a request is treated as useful local context for interpreting the request, not as self-directed noise to discard.
  • I2. Context windows are bounded cheaply: they open and close at utterances that mention a domain attribute, with no separate segmentation model.
  • I3. Labels are predicted in two layers, where a coarse decision first narrows the fine-grained label choices beneath it.
  • I4. Heuristic slot assignment allows class-preserving paraphrase augmentation without paying for slot annotation on the corpus and on every generated paraphrase.
  • I5. The evaluation weighs accuracy against training cost and shows that a CRF can beat LSTMs on small data.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (frames): Speech-act detection in prag could read the neighbouring sentences of a turn as a context window, so that an indirect request is interpreted together with the reasoning around it.
  • I2 (world): Episode boundaries in a text's world could be placed where a tracked entity or attribute is mentioned again, giving explainable segments for ordering events.
  • I3 (frames): Speech-act classification could first decide request versus commentary and then choose the specific act, which keeps each decision small and traceable.
  • I4 (learn): Small gold sets could be expanded with paraphrases while rule-identified slots stay fixed, so the original labels remain valid for the new silver examples.
  • I5 (learn): When training on small treebanks, Mova could always run the perceptron parser as a baseline against gparse and report compute cost next to LAS.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 9,086 worksArtificial Intelligence · 8,889 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 works

🧭 Topics 2Research subjects and languages the work is about.Dialogue systems · 401 worksSpeech act theory · 160 works

Cite

Cite as Abhinav Kumar, Barbara Di Eugenio, Jillian Aurisano and Andrew Johnson. 2020. Augmenting Small Data to Classify Contextualized Dialogue Acts for Exploratory Visualization. Proceedings of the Twelfth Language Resources and Evaluation Conference.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{kumar2020augmenting,
  title = {{Augmenting Small Data to Classify Contextualized Dialogue Acts for Exploratory Visualization}},
  author = {Abhinav Kumar, Barbara {Di Eugenio}, Jillian Aurisano and Andrew Johnson},
  booktitle = {Proceedings of the Twelfth Language Resources and Evaluation Conference},
  year = {2020},
  url = {https://aclanthology.org/2020.lrec-1.74/}
}

References

50 references, 15 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ahmadvand, A., Choi, J. I., and Agichtein, E. (2019). Contextual dialogue act classification for open-domain conversational agents. In The 42nd International ACM SI-GIR, pages 1273–1276. Ang, J., Liu, Y., and Shriberg, E. ). Automatic dialog act segmentation and classification in multiparty meetings. In IEEE International Conference on Acoustics, Speech, and Signal Processing, volume 1.
  2. Aurisano, J., Kumar, A., Gonzales, A., Reda, K., Leigh, J., Di Eugenio, B., and Johnson, A. (2015). Show me data”: Observational study of a conversational interface in visual data exploration. In IEEE VIS, volume 15, page 1. find
  3. Aurisano, J., Kumar, A., Gonzalez, A., Leigh, J., Di Eugenio, B., and Johnson, A. (2016). Articulate2: Toward a conversational interface for visual data exploration. In IEEE VIS, volume 16, page 1. find
  4. Benotti, L. (2009). Clarification potential of instructions. In The SIGDIAL 2009 Conference, pages 196–205, London, UK, September. Association for Computational Linguistics. find
  5. Berant, J. and Liang, P. (2014). Semantic parsing via paraphrasing. In The 52nd Annual Meeting of the Association for Computational Linguistics, volume 1, pages 1415– 1425.
  6. Brehmer, M. and Munzner, T. (2013). A multi-level typology of abstract visualization tasks. IEEE Transactions on Visualization and Computer Graphics, 19(12):2376– 2385. ↗
  7. Campagna, G., Xu, S., Moradshahi, M., Socher, R., and Lam, M. S. (2019). Genie: a generator of natural language semantic parsers for virtual assistant commands. In Proceedings of the 40th ACM SIGPLAN Conference on Programming Language Design and Implementation, pages 394–410. ACM. find
  8. Chen, L., Javaid, M., Di Eugenio, B., and Žefran, M. (2015). The roles and recognition of haptic-ostensive actions in collaborative multimodal human-human dialogues. Computer Speech & Language, 32:201–231, Nov. ↗
  9. Cox, K., Grinter, R. E., Hibino, S. L., Jagadeesan, L. J., and Mantilla, D. (2001). A multi-modal natural language interface to an information visualization environment. International Journal of Speech Technology, 4(3-4):297– 314. ↗
  10. Dhamdhere, K., McCurley, K. S., Nahmias, R., Sundararajan, M., and Yan, Q. (2017). Analyza: Exploring data with conversation. In The 22nd International Conference on Intelligent User Interfaces, IUI ’17, pages 493– 504. find
  11. Dong, L., Mallinson, J., Reddy, S., and Lapata, M. (2017). Learning to paraphrase for question answering. arXiv preprint arXiv:1708.06022. ↗
  12. Fader, A., Zettlemoyer, L., and Etzioni, O. (2013). Paraphrase-driven learning for open question answering. In The 51st Annual Meeting of the Association for Computational Linguistics, volume 1, pages 1608–1618.
  13. Gao, T., Dontcheva, M., Adar, E., Liu, Z., and Karahalios, K. G. (2015). Datatone: Managing ambiguity in natural language interfaces for data visualization. In The 28th Annual ACM Symposium on User Interface Software & Technology, pages 489–500. ACM. find
  14. Grammel, L., Tory, M., and Storey, M.-A. (2010). How information visualization novices construct visualizations. IEEE Transactions on Visualization and Computer Graphics, 16(6):943–952. ↗
  15. Hakkani-Tür, D., Tür, G., Celikyilmaz, A., Chen, Y.-N., Gao, J., Deng, L., and Wang, Y.-Y. (2016). Multi-domain joint semantic frame parsing using bi-directional rnn-lstm. In Interspeech, pages 715–719. find
  16. Hoque, E., Setlur, V., Tory, M., and Dykeman, I. (2017). Applying pragmatics principles for interaction with visual analytics. IEEE Transactions on Visualization and Computer Graphics, 24(1):309–318. ↗
  17. Hou, Y., Liu, Y., Che, W., and Liu, T. (2018). Sequence-to-sequence data augmentation for dialogue language understanding. In The 27th International Conference on Computational Linguistics, pages 1234–1245, August. ↗
  18. Jia, R. and Liang, P. (2016). Data recombination for neural semantic parsing. arXiv preprint arXiv:1606.03622. ↗
  19. Jovanovic, N., op den Akker, R., and Nijholt, A. (2006). Addressee identification in face-to-face meetings. In 11th Conference of the European Chapter of the Association for Computational Linguistics, pages 169–176. find
  20. Jurafsky, D., Shriberg, E., and Biasca, D. (1997). Switchboard swbd-damsl labeling project coder’s manual. Draft 13. Technical Report 97-02. find
  21. Katsakioris, M. M., Hastie, H., Konstas, I., and Laskov, A. (2019). Corpus of multimodal interaction for collaborative planning. In Proceedings of the Combined Workshop on Spatial Language Understanding (SpLU) and Grounded Communication for Robotics (RoboNLP), pages 1–6. find
  22. Khanpour, H., Guntakandla, N., and Nielsen, R. (2016). Dialogue act classification in domain-independent conversations using a deep recurrent neural network. In The 26th International Conference on Computational Linguistics (COLING), pages 2012–2021.
  23. Kim, S. N., Cavedon, L., and Baldwin, T. (2010). Classifying dialogue acts in one-on-one live chats. In The 2010 Conference on Empirical Methods in Natural Language Processing, pages 862–871. Association for Computational Linguistics.
  24. Kingma, D. P. and Ba, J. (2014). Adam: A method for stochastic optimization. CoRR, abs/1412.6980. ↗
  25. Kumar, A., Aurisano, J., Di Eugenio, B., Johnson, A., Gonzalez, A., and Leigh, J. (2016). Towards a dialogue system that supports rich visualizations of data. In Proceedings of the 17th Annual Meeting of the Special Interest Group on Discourse and Dialogue, pages 304–309. find
  26. Kumar, A., Di Eugenio, B., Aurisano, J., Johnson, A., Alsaiari, A., Flowers, N., Gonzalez, A., and Leigh, J. (2017). Towards multimodal coreference resolution for exploratory data visualization dialogue: Context-based annotation and gesture identification. In The 21st Workshop on the Semantics and Pragmatics of Dialogue (SemDial 2017–SaarDial)(August 2017), volume 48. find
  27. Kumar, H., Agarwal, A., Dasgupta, R., and Joshi, S. (2018). Dialogue act sequence labeling using hierarchical encoder with crf. In Thirty-Second AAAI Conference on Artificial Intelligence. ↗
  28. Kumar, A., Aurisano, J., Di Eugenio, B., and Johnson, A. (2020 (forthcoming)). Intelligent assistant for exploring data visualizations. In Thirty-Third International Flairs Conference 2020. find
  29. Li, X. and Roth, D. (2002). Learning question classifiers. In The 19th International Conference on Computational Linguistics, pages 1–7.
  30. Manuvinakurike, R., Bui, T., Chang, W., and Georgila, K. (2018). Conversational image editing: Incremental intent identification in a new dialogue task. In The 19th Annual SIGdial Meeting on Discourse and Dialogue, pages 284–295. find
  31. Mesnil, G., Dauphin, Y., Yao, K., Bengio, Y., Deng, L., Hakkani-Tur, D., He, X., Heck, L., Tur, G., Yu, D., et al. (2015). Using recurrent neural networks for slot filling in spoken language understanding. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 23(3):530–539. ↗
  32. Mikolov, T., Chen, K., Corrado, G., and Dean, J. (2013). Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781. ↗
  33. Pennington, J., Socher, R., and Manning, C. (2014). Glove: Global vectors for word representation. In The Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1532–1543.
  34. Popescu-Belis, A. and Estrella, P. (2007). Generating usable formats for metadata and annotations in a large meeting corpus. In Proceedings of the 45th Annual Meeting of the ACL on Interactive Poster and Demonstration Sessions, pages 93–96. Association for Computational Linguistics. find
  35. Popov, I. O., Schraefel, M., Hall, W., and Shadbolt, N. (2011). Connecting the dots: a multi-pivot approach to data exploration. In International Semantic Web Conference, pages 553–568. find
  36. Reda, K., Johnson, A. E., Leigh, J., and Papka, M. E. (2014). Evaluating user behavior and strategy during visual exploration. In The Fifth Workshop on Beyond Time and Errors: Novel Evaluation Methods for Visualization, pages 41–45. ACM. find
  37. Reithinger, N., Fedeler, D., Kumar, A., Lauer, C., Pecourt, E., and Romary, L. (2005). Miamm—a multimodal dialogue system using haptics. In Advances in Natural Multimodal Dialogue Systems, pages 307–332. Springer. ↗
  38. Schegloff, E. A. and Sacks, H. (1973). Opening up closings. Semiotica, 8(4):289–327. ↗
  39. Searle, J. R. (1975). Indirect Speech Acts. In P. Cole et al., editors, Syntax and Semantics 3. Speech Acts. Academic Press. ↗
  40. Setlur, V., Battersby, S. E., Tory, M., Gossweiler, R., and Chang, A. X. (2016). Eviza: A natural language interface for visual analysis. In The 29th Annual Symposium on User Interface Software and Technology, pages 365– 377. ACM. find
  41. Shriberg, E., Dhillon, R., Bhagat, S., Ang, J., and Carvey, H. (2004). The icsi meeting recorder dialog act (mrda) corpus. In The 5th SIGdial Workshop on Discourse and Dialogue. ↗
  42. Stolcke, A., Ries, K., Coccaro, N., Shriberg, E., Bates, R., Jurafsky, D., Taylor, P., Martin, R., van Ess-Dykema, C., and Meteer, M. (2000). Dialogue act modeling for automatic tagging and recognition of conversational speech. Computational Linguistics, 26(3):339–373. ↗
  43. Strubell, E., Ganesh, A., and McCallum, A. (2019). Energy and policy considerations for deep learning in NLP. In The 57th Annual Meeting of the AssStolckeociation for Computational Linguistics, pages 3645–3650, July.
  44. Sun, Y., Leigh, J., Johnson, A., and Lee, S. (2010). Articulate: A semi-automated model for translating natural language queries into meaningful visualizations. In International Symposium on Smart Graphics, pages 184– 195. Springer. ↗
  45. Tavafi, M., Mehdad, Y., Joty, S., Carenini, G., and Ng, R. (2013). Dialogue act recognition in synchronous and asynchronous conversations. In The SIGDIAL 2013 Conference, pages 117–121. ↗
  46. Van Someren, M., Barnard, Y., and Sandberg, J. (1994). The think aloud method: a practical approach to modelling cognitive processes. London: Academic Press. find
  47. Vu, N. T. (2016). Sequential convolutional neural networks for slot filling in spoken language understanding. arXiv preprint arXiv:1606.07783. ↗
  48. Wang, Y., Deng, L., and Acero, A. (2011). Semantic frame-based spoken language understanding. Spoken Language Understanding: Systems for Extracting Semantic Information from Speech, pages 41–91. ↗
  49. Wieting, J., Mallinson, J., and Gimpel, K. (2017). Learning paraphrastic sentence embeddings from back-translated bitext. In Empirical Methods in Natural Language Processing (EMNLP). find
  50. Yoo, K. M., Shin, Y., and Lee, S.-g. (2018). Data augmentation for spoken language understanding via joint variational generation. arXiv preprint arXiv:1809.02305, 09. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/kumar-2020-augmenting-small-data-classify-contextualized/