Elamur Research⌕
☰
home / papers / 2014 / Dialogue Act Modeling for Non-Visual Web Access

Dialogue Act Modeling for Non-Visual Web Access

Paper · 2014 · Vikas Ashok, Yevgen Borodin, Svetlana Stoyanchev et al.

Summary

Blind people usually browse the Web with screen readers, which read pages aloud and depend on many keyboard shortcuts. This makes multi-step jobs like online shopping or booking slow and tiring. The authors want a spoken dialogue browser where users state their goals and the system handles clicks and form filling. As a first step, they model what users say. They ran a wizard-of-oz study with 24 blind participants, half experienced and half beginners, on six everyday web tasks. The resulting corpus was annotated with a DAMSL-inspired dialogue act scheme, with separate tagsets for user and system turns. They then tested combinations of lexical/syntactic, contextual and task-related features with several classifiers, including J48 decision trees and random forests. A compact set of simple features reached over 90% precision and recall. The paper argues that light tree-based models are therefore enough, which keeps latency low. Fast responses are a key usability requirement for blind users.

Key points

  • Corpus size: 96 dialogs, one per participant per task, totalling 792 user and 774 system utterances and roughly 22 hours of speech.
  • Tasks: shopping, email, flight booking, hotel booking, job search and university admission.
  • User acts: Command-Intention, Command-Task, Command-Multiple, Command-Navigation, Information-Task, Question-Task, Help-Task and Self-Talk.
  • System acts: Prompt, Short-Response, Long-Response, Question-Answer, Help-Response and others.
  • Annotation reliability: Fleiss' κ was above 0.80 for every task, which counts as excellent agreement.
  • Feature ablations:
    • Dropping word-presence features (P) cost up to 15% precision and 16% recall.
    • Dropping word-position features (B) cost about 11% in both.
    • Adding unigrams or the task name to the best group (G6) did not change overall scores, which suggests the features are task-independent.
  • Error analysis: the weakest acts were Command-Multiple, Question-Task and Help-Task. Typical failures were terse noun-phrase questions such as "Price?" and misleading cue words, for example a leading "I" or "continue".

Interesting ideas

  • I1. The classifier relies on a few transparent boolean cues, such as whether a cue word occurs, whether it opens the utterance, and what the previous dialogue act was, rather than a full bag of words, and it still reaches high accuracy; compact cues like these are easy to audit.
  • I2. High-confidence, high-support rules are pulled out of the trained J48 tree to show which feature combinations drive each label, turning a learned model into readable logic.
  • I3. Task independence is tested by adding the task name as a feature and confirming it brings no gain, which works as a cheap negative control.
  • I4. The error analysis shows elliptical noun-phrase fragments like "Layover?" or "Price?" are confused with commands or information because they lack wh-words, so surface cues alone miss them.
  • I5. Per-label ablations show that some features help only specific acts, for example unigrams for Help-Task and syntax for Question-Task, even when overall scores barely move.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (frames): speech-act tagging in prag could use similar explicit cues, read from the UD tree root and first token plus the previous act, so every label comes with a stated reason.
  • I2 (learn): learned speech-act or repair decisions could be distilled into ranked rules filtered by confidence and support, then kept as explainable self-induced rules.
  • I3 (learn): evaluations could add a "leaky" feature such as source genre or test-set id and check it adds nothing, proving the model generalises rather than memorising.
  • I4 (frames): a fragment ending in "?" with no verb could be routed to a question act whose missing predicate is restored from the current topic in the world model.
  • I5 (learn): evaluation reports could break gains down per speech act or relation, so small but targeted improvements are not discarded on aggregate scores.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Columbia UniversityNew York · United States149 works · 48,852 citations🏢AT&TBedminster · United States29 works · 15,393 citations🎓Stony Brook UniversityStony Brook · United States27 works · 2,057 citations

Also on the byline: Charmtech Labs.

📚 Shelves 8The field of science this work belongs to (OpenAlex fields).Computer Science · 9,086 worksArtificial Intelligence · 8,889 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 worksSocial Sciences · 744 worksHealth Professions · 14 worksOccupational Therapy · 4 worksHuman Factors and Ergonomics · 2 works

🧭 Topics 2Research subjects and languages the work is about.Dialogue systems · 401 worksSpeech act theory · 160 works

Cite

Cite as Vikas Ashok, Yevgen Borodin, Svetlana Stoyanchev and IV Ramakrishnan. 2014. Dialogue Act Modeling for Non-Visual Web Access. Proceedings of the 15th Annual Meeting of the Special Interest Group on Discourse and Dialogue (SIGDIAL).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.3115/v1/w14-4317

@inproceedings{ashok2014dialogue,
  title = {{Dialogue Act Modeling for Non-Visual Web Access}},
  author = {Vikas Ashok, Yevgen Borodin, Svetlana Stoyanchev and IV Ramakrishnan},
  booktitle = {Proceedings of the 15th Annual Meeting of the Special Interest Group on Discourse and Dialogue (SIGDIAL)},
  year = {2014},
  doi = {10.3115/v1/w14-4317},
  url = {https://doi.org/10.3115/v1/w14-4317}
}

References

24 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. AFB. 2013. Facts and figures on american adults with vision loss. http://www.afb.org/info/blindness-statistics/adults/facts-and-figures/235, January. ↗
  2. Apple-Inc. 2013. Voiceover for os x. http://www.apple.com/accessibility/osx/voiceover/. ↗
  3. Srinivas Bangalore and Amanda J Stent. 2009. Incremental parsing models for dialog task structure. In Proceedings of the 12th Conference of the European Chapter of the Association for Computational Linguistics, pages 94–102. Association for Computational Linguistics. find
  4. Yevgen Borodin, Jeffrey P Bigham, Glenn Dausch, and IV Ramakrishnan. 2010. More than meets the eye: a survey of screen-reader browsing strategies. In Proceedings of the 2010 International Cross Disciplinary Conference on Web Accessibility (W4A), page 13. ACM. find
  5. Kristy Elizabeth Boyer, Eun Young Ha, Robert Phillips, Michael D Wallis, Mladen A Vouk, and James C Lester. 2010. Dialogue act modeling in a complex task-oriented domain. In Proceedings of the 11th Annual Meeting of the Special Interest Group on Discourse and Dialogue, pages 297–305. Association for Computational Linguistics.
  6. Harry Bunt. 2011. Multifunctionality in dialogue. Computer Speech & Language, 25(2):222–245. ↗
  7. Jean Carletta, Stephen Isard, Gwyneth Doherty-Sneddon, Amy Isard, Jacqueline C Kowtko, and Anne H Anderson. 1997. The reliability of a dialogue structure coding scheme. Computational linguistics, 23(1):13–31. ↗
  8. Ananlada Chotimongkol. 2008. Learning the structure of task-oriented conversations from the corpus of in-domain dialogs. Ph.D. thesis, SRI International. find
  9. Mark G Core and James Allen. 1997. Coding dialogs with the damsl annotation scheme. In AAAI fall symposium on communicative action in humans and machines, pages 28–35. Boston, MA. find
  10. Maxine Eskenazi, Alexander I Rudnicky, Karin Gregory, Paul C Constantinides, Robert Brennan, Christina L Bennett, and Jwan Allen. 1999. Data collection and processing in the carnegie mellon communicator. In EUROSPEECH. find
  11. César González Ferreras and Valentı́n Cardeñoso-Payo. 2005. Development and evaluation of a spoken dialog system to access a newspaper web site. In IN-TERSPEECH, pages 857–860. find
  12. J.L. Fleiss. 1973. Statistical methods for rates and proportions Rates and proportions. Wiley. find
  13. Freedom-Scientific. 2014. Screen reading software from freedom scientific. http://www.freedomscientific.com/products/fs/jaws-product-page.asp. ↗
  14. Mark Hall, Eibe Frank, Geoffrey Holmes, Bernhard Pfahringer, Peter Reutemann, and Ian H Witten. 2009. The weka data mining software: an update. ACM SIGKDD explorations newsletter, 11(1):10– 18. ↗
  15. Dan Klein and Christopher D Manning. 2003. Accurate unlexicalized parsing. In Proceedings of the 41st Annual Meeting on Association for Computational Linguistics-Volume 1, pages 423–430. Association for Computational Linguistics.
  16. Cheongjae Lee, Sangkeun Jung, Kyungduk Kim, Donghyeon Lee, and Gary Geunbae Lee. 2010. Recent approaches to dialog management for spoken dialog systems. JCSE, 4(1):1–22. ↗
  17. George A Miller. 1995. Wordnet: a lexical database for english. Communications of the ACM, 38(11):39–41. ↗
  18. Kiyonori Ohtake, Teruhisa Misu, Chiori Hori, Hideki Kashioka, and Satoshi Nakamura. 2009. Annotating dialogue acts to construct dialogue systems for consulting. In Proceedings of the 7th Workshop on Asian Language Resources, pages 32–39. Association for Computational Linguistics. find
  19. Kiyonori Ohtake, Teruhisa Misu, Chiori Hori, Hideki Kashioka, and Satoshi Nakamura. 2010. Dialogue acts annotation for nict kyoto tour dialogue corpus to construct statistical dialogue systems. In LREC. find
  20. Yury Puzis, Yevgen Borodin, Rami Puzis, and IV Ramakrishnan. 2013. Predictive web automation assistant for people with vision impairments. In Proceedings of the 22nd international conference on World Wide Web, pages 1031–1040. International World Wide Web Conferences Steering Committee. find
  21. Vivek Kumar Rangarajan Sridhar, Srinivas Bangalore, and Shrikanth Narayanan. 2009. Combining lexical, syntactic and prosodic cues for improved online dialog act tagging. Computer Speech & Language, 23(4):407–422. ↗
  22. John R Searle. 1975. Indirect speech acts. Syntax and semantics, 3:59–82. ↗
  23. Andreas Stolcke, Klaus Ries, Noah Coccaro, Elizabeth Shriberg, Rebecca Bates, Daniel Jurafsky, Paul Taylor, Rachel Martin, Carol Van Ess-Dykema, and Marie Meteer. 2000. Dialogue act modeling for automatic tagging and recognition of conversational speech. Computational linguistics, 26(3):339–373. ↗
  24. Lu Wang, Larry Heck, and Dilek Hakkani-Tur. 2014. Leveraging semantic web search and browse sessions for multi-turn spoken dialog systems. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/ashok-2014-dialogue-act-modeling-non-visual/