Elamur Research⌕
☰
home / papers / 2021 / Predicting pragmatic discourse features in the language of adults with autism spectrum disorder

Predicting pragmatic discourse features in the language of adults with autism spectrum disorder

Paper · 2021 · Christine Yang, Duanchen Liu, Qingyun Yang et al.

Summary

People with autism spectrum disorder (ASD) often struggle with the social side of talk. Measuring this usually requires expert hand-annotation of transcripts, which is too slow for clinics. Earlier computational work also covered only children and examiner-led speech. The authors are building a corpus of spontaneous peer conversations between adults (with ASD or typically developing) and neurotypical partners, recorded during a map-directions task and a deserted-island negotiation task. Each utterance is rated on a 1–3 scale for politeness, uncertainty and informativeness.

They train several neural classifiers on these ratings: an LSTM, CNNs over GloVe and word2vec, and a feedforward network over BERT embeddings. These are compared with simple baselines and with existing pretrained classifiers built on text corpora. The BERT feedforward model performs best, with F1 above 0.82 for all three features, and clearly beats prior tools.

However, when its predictions are used to compare the ASD and TD groups, it fails to reproduce the significant differences found with manual labels. This shows that accurate utterance-level tagging does not guarantee valid group-level conclusions.

Key points

  • The corpus so far has 22 dyads: 14 ASD and 8 TD participants aged 18–30, plus 11 neurotypical conversational partners. It contains 9,267 participant utterances (5,742 ASD, 3,525 TD), segmented into C-units and transcribed in Praat.
  • Each utterance received two ratings from a pool of three annotators, and the two ratings were averaged. Krippendorff's α was 0.57 for politeness, 0.75 for uncertainty and 0.90 for informativeness.
  • The existing-tool baselines were:
    • an SVM trained on the Stanford Politeness Corpus;
    • logistic regression trained on the Szeged Uncertainty Corpus;
    • logistic regression trained on SQUINKY! for informativeness.
  • The majority-class baseline was already strong for politeness (F1 0.77), because most utterances are rated neutral. Relative to this baseline, the BERT feedforward model gained 0.10 F1 on politeness, a margin that grew to 0.33 on uncertainty and reached 0.42 on informativeness.
  • Manual labels showed task-dependent group differences, for example more uncertainty from ASD speakers in the open-ended island task. Model predictions reversed several of these trends, and none of their group differences reached statistical significance.

Interesting ideas

  • I1. After scoring a tagger with F1, the authors rerun the downstream statistical comparison (a t-test between groups) on predicted labels and check whether the conclusions match those from gold labels. This tests whether the tool is fit for its real purpose, beyond per-item accuracy.
  • I2. Uncertainty is graded partly by speech-act form: statements score lowest, while polar and either-or questions, hedges and heavy filler score medium, and open wh-questions and explicit confusion score highest. This gives a reproducible rubric without relying on a keyword list.
  • I3. Politeness is tied to how directives are phrased: bare imperatives rank low, while conditional or question-framed requests ("if you could make a left") rank high. In effect, the scale is a measure of indirect speech acts.
  • I4. When pretrained tools output continuous scores on a different scale, the authors threshold those outputs onto the target 1–3 categories so that baselines are compared fairly. They also report majority, stratified and random baselines to expose label skew.
  • I5. The transcripts explicitly mark abandoned and interrupted utterances, fillers, partial words and discourse markers, and segment speech into C-units (an independent clause with its modifiers). This keeps disfluency information available as signal rather than discarding it as noise.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Evaluation could include a "conclusion agreement" check, confirming that answers or world statistics derived from silver parses match those derived from gold ones, not just that LAS is high.
  • I2 (frames): The speech-act layer in prag could assign an epistemic-certainty level to each utterance from its question type and hedge constructions, feeding how confidently a fact enters the world of the text.
  • I3 (frames): Indirect-directive patterns could be recognized on the UD tree (conditional clause or interrogative with a second-person subject plus an action verb) and mapped to a request frame with a politeness attribute.
  • I4 (learn): Witness parsers or LLM teachers with mismatched output schemes could be mapped onto one shared label set before voting. Trivial-baseline controls would flag datasets where high scores come from class imbalance.
  • I5 (tokens): The tokenizer could tag fillers, cut-off words and abandoned-clause boundaries as distinct token types, so later stages can use them as uncertainty cues instead of mis-parsing them.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Boston CollegeBoston · United States2 works · 14 citations

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 9,086 worksArtificial Intelligence · 8,889 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 worksNeuroscience · 95 worksCognitive Neuroscience · 95 works

🧭 Topics 1Research subjects and languages the work is about.Discourse analysis · 394 works

Cite

Cite as Christine Yang, Duanchen Liu, Qingyun Yang, Zoey Liu and Emily Prud'hommeaux. 2021. Predicting pragmatic discourse features in the language of adults with autism spectrum disorder. Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing: Student Research Workshop.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2021.acl-srw.29

@inproceedings{yang2021predicting,
  title = {{Predicting pragmatic discourse features in the language of adults with autism spectrum disorder}},
  author = {Christine Yang, Duanchen Liu, Qingyun Yang, Zoey Liu and Emily Prud'hommeaux},
  booktitle = {Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing: Student Research Workshop},
  year = {2021},
  doi = {10.18653/v1/2021.acl-srw.29},
  url = {https://doi.org/10.18653/v1/2021.acl-srw.29}
}

References

32 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Catherine Adams. 2002. Practitioner review: The assessment of language pragmatics. Journal of child psychology and psychiatry, 43(8):973–987. ↗
  2. A Anderson, M Bader, E Bard, E Boyle, G. M Doherty, S Garrod, S Isard, J Kowtko, J McAllister, J Miller, C Sotillo, H. S Thompson, and R Weinert. 1991. The HCRC map task corpus. Language and Speech, 34(4):351–366. ↗
  3. Ron Artstein and Massimo Poesio. 2008. Inter-coder agreement for computational linguistics. Computational Linguistics, 34(4):555–596. ↗
  4. Dorothy VM Bishop, Janet Chan, Catherine Adams, Joanne Hartley, and Fiona Weir. 2000. Conversational responsiveness in specific language impairment: Evidence of disproportionate pragmatic difficulties in a subset of children. Development and psychopathology, 12(2):177–199. ↗
  5. Paul Boersma and David Weenink. 2001. Praat, a system for doing phonetics by computer. Glot international, 5:341–345. find
  6. Allison R Canfield, Inge-Marie Eigsti, Ashley de Marchena, and Deborah Fein. 2016. Story goodness in adolescents with autism spectrum disorder (ASD) and in optimal outcomes from ASD. Journal of Speech, Language, and Hearing Research, 59(3):533–545. ↗
  7. Cristian Danescu-Niculescu-Mizil, Moritz Sudhof, Dan Jurafsky, Jure Leskovec, and Christopher Potts. 2013. A computational approach to politeness with application to social factors. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 250–259, Sofia, Bulgaria. Association for Computational Linguistics. ↗
  8. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics. ↗
  9. Martin J Eales. 1993. Pragmatic impairments in adults with childhood diagnoses of autism or developmental receptive language disorder. Journal of autism and developmental disorders, 23(4):593–617. ↗
  10. Uta Frith. 1994. Autism and theory of mind in everyday life. Social development, 3(2):108–124. find
  11. Adam Goodkind, Michelle Lee, Gary E Martin, Molly Losh, and Klinton Bicknell. 2018. Detecting language impairments in autism: A computational analysis of semi-structured conversations with vector semantics. Proceedings of the Society for Computation in Linguistics, 1(1):12–22. ↗
  12. Kyle Gorman, Lindsay Olson, Alison Presmanes Hill, Rebecca Lunsford, Peter A Heeman, and Jan PH van Santen. 2016. Uh and um in children with autism spectrum disorders or language impairment. Autism Research, 9(8):854–865. find
  13. Dawn Hendricks. 2010. Employment and adults with autism spectrum disorders: Challenges and strategies for success. Journal of Vocational Rehabilitation, 32(2):125–134. ↗
  14. Leo Kanner. 1943. Autistic disturbances of affective content. Nervous Child, 2:217–250. find
  15. Mieke P Ketelaars, Juliane Cuperus, Kino Jansonius, and Ludo Verhoeven. 2010. Pragmatic language impairment and associated behavioural problems. International Journal of Language & Communication Disorders, 45(2):204–214. ↗
  16. Shibamouli Lahiri. 2015. Squinky! a corpus of sentence-level formality, informativeness, and implicature. ↗
  17. Walter Loban. 1976. Language Development: Kindergarten through Grade Twelve. NCTE Committee on Research Report No. 18. ERIC. find
  18. Sebastian Schuster, Yuxing Chen, and Judith Degen. 2020. Harnessing the linguistic signal to predict scalar inferences. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5387–5403, Online. Association for Computational Linguistics. ↗
  19. Karen Gainer Sirota. 2004. Positive politeness as discourse process: Politeness practices of high-functioning children with autism and asperger syndrome. Discourse Studies, 6(2):229–251. ↗
  20. Catherine Lord, Michael Rutter, Pamela DiLavore, and Susan Risi. 2002. Autism Diagnostic Observation Schedule (ADOS). Western Psychological Services. find
  21. Molly Losh and Peter C Gordon. 2014. Quantifying narrative ability in autism spectrum disorder: A computational linguistic analysis of narrative coherence. Journal of autism and developmental disorders, 44(12):3016–3025. ↗
  22. Benjamin S. Meyers, Nuthan Munaiah, Andrew Meneely, and Emily Prud’hommeaux. 2019. Pragmatic characteristics of security conversations: An exploratory linguistic analysis. In 2019 IEEE/ACM 12th International Workshop on Cooperative and Human Aspects of Software Engineering (CHASE), pages 79–82. find
  23. Benjamin S Meyers, Nuthan Munaiah, Emily Prud’hommeaux, Andrew Meneely, Josephine Wolff, Cecilia Ovesdotter Alm, and Pradeep Murukannaiah. 2018. A dataset for identifying actionable feedback in collaborative software development. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 126–131. find
  24. Tomas Mikolov, Ilya Sutskever, Kai Chen, Gregory S. Corrado, and Jeffrey Dean. 2013. Distributed representations of words and phrases and their compositionality. In Conference on Neural Information Processing Systems, pages 3111–3119. ↗
  25. Julia Parish-Morris, Mark Liberman, Neville Ryant, Christopher Cieri, Leila Bateman, Emily Ferguson, and Robert T Schultz. 2016. Exploring autism spectrum disorders using hlt. In Proceedings of the Third Workshop on Computational Linguistics and Clinical Psychology, pages 74–84. find
  26. Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. Glove: Global vectors for word representation. In Empirical Methods in Natural Language Processing (EMNLP), pages 1532–1543. ↗
  27. Emily Prud’hommeaux, Eric Morley, Masoud Rouhizadeh, Laura Silverman, Jan van Santeny, Brian Roarkz, Richard Sproatz, Sarah Kauper, and Rachel DeLaHunta. 2014. Computational analysis of trajectories of linguistic development in autism. In 2014 IEEE Spoken Language Technology Workshop (SLT), pages 266–271. IEEE. find
  28. Michael Rutter, Anthony Bailey, and Catherine Lord. 2003. Social Communication Questionnaire (SCQ). Western Psychological Services, Los Angeles. find
  29. Veronika Vincze. 2014. Uncertainty detection in hungarian texts. In Proceedings of COLING 2014, the 25th International Conference on Computational Linguistics: Technical Papers, pages 1844–1853. find
  30. Joanne Volden and Catherine Lord. 1991. Neologisms and idiosyncratic language in autistic speakers. Journal of Autism and Developmental Disorders, 21:109–130. ↗
  31. Andrew JO Whitehouse, Helen J Watt, EA Line, and Dorothy VM Bishop. 2009. Adult psychosocial outcomes of children with specific language impairment, pragmatic language impairment and autism. International Journal of Language & Communication Disorders, 44(4):511–528. ↗
  32. EC Young, JJ Diehl, D Morris, SL Hyman, and L Ben-netto. 2005. Pragmatic language disorders in children with autism: The use of two formal tests to distinguish affected children from controls. Language, Speech, and Hearing Services in Schools, 36:62–72. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/yang-2021-predicting-pragmatic-discourse-features-language/