Elamur Research⌕
☰
home / papers / 2015 / SQUINKY! A Corpus of Sentence-level Formality, Informativeness, and Implicature

SQUINKY! A Corpus of Sentence-level Formality, Informativeness, and Implicature

Paper · 2015 · Shibamouli Lahiri

Summary

Studies of formality had mostly worked at the level of whole documents or single words. Sentence-level resources were tiny or limited to one domain. This paper builds SQUINKY!, a general-purpose corpus of 7,032 English sentences from news, blogs and forums. Amazon Mechanical Turk workers rated each sentence on a 1–7 Likert scale for three linked pragmatic properties: formality, informativeness and implicature. The design draws on Heylighen and Dewaele's view that formality is a continuum tied to how much context a reader must supply.

The author checked reliability in two ways. Two separate MTurk rounds were run and their mean ratings correlated. The new ratings were also compared with earlier, more controlled formality annotations. Formality and informativeness turned out to be annotated consistently, while implicature was far less stable. The analyses cover:

  • correlations among the three variables,
  • differences between genres,
  • agreement with automatic style measures such as the deixis-based F-score,
  • how ratings change with a sentence's position in its document.

At the time, it was the largest resource of its kind.

Key points

  • 7,032 sentences were rated on a 1–7 scale for formality, informativeness and implicature via MTurk. The genres are news, blogs and forums.
  • Reliability was measured by correlating mean ratings across two independent MTurk rounds. Formality was also checked against the pilot of Lahiri and Lu (2011), which had 600 sentences rated 1–5 by two linguistics students.
  • Formality and informativeness agreed well across rounds; implicature proved much harder to rate consistently.
  • Human formality and informativeness ratings correlated only weakly with Heylighen and Dewaele's F-score and with the I-score. Implicature showed a small negative correlation with the F-score.
  • In the positional analysis, news sentences were the most formal and informative, then blogs, then forums. News declined from its opening sentences, forums stayed flat and low, and implicature showed no clear trend.

Interesting ideas

  • I1. Properties that vary by degree, such as formality, are rated on an ordinal scale instead of a yes/no label, because the paper argues any text can always be made more explicit.
  • I2. Annotation reliability is judged by repeating the whole rating process independently and correlating the averaged results. A smaller, carefully controlled set serves as an outside anchor.
  • I3. The author treats formality as the amount of missing context, which comes from two sources: deixis, which can be counted, and implicature, which needs judgment.
  • I4. Cheap automatic proxies (F-score, lexical density, length) are compared against human ratings to see whether they actually capture the target property. Here they mostly did not.
  • I5. Each document is split into position deciles and the average rating per decile is plotted to show how style and information content shift through a text.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (frames): prag could give register and implicature a graded strength rather than a boolean flag, so later stages can weigh how much unstated context a sentence leaves open.
  • I2 (learn): run LLM teacher tiers or witness parsers twice on the same silver batch and check a small gold anchor, keeping only label types that reproduce across runs.
  • I3 (frames): a sentence's "missing context" could be estimated as unresolved deictic and coreferent mentions plus open implicature slots, both available from the frame layer.
  • I4 (learn): every new learned score should be compared against trivial baselines like sentence length, as a negative control that exposes shortcut signals.
  • I5 (back): the text → graph → text compressor could use a positional informativeness profile, keeping the dense lead sentences of news-like texts while treating forum-like texts evenly.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

📚 Shelves 5 Computer Science · 9,086 works Artificial Intelligence · 8,889 works Arts and Humanities · 9,296 works Language and Linguistics · 9,206 works Literature and Literary Theory · 340 works

🧭 Topics 2 English · 1,373 works Implicature · 72 works

Cite

Cite as Shibamouli Lahiri. 2015. SQUINKY! A Corpus of Sentence-level Formality, Informativeness, and Implicature. arXiv:1506.02306.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1506.02306 · arXiv 1506.02306

@misc{lahiri2015squinky,
  title = {{SQUINKY! A Corpus of Sentence-level Formality, Informativeness, and Implicature}},
  author = {Shibamouli Lahiri},
  year = {2015},
  doi = {10.48550/arxiv.1506.02306},
  eprint = {1506.02306},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1506.02306}
}

References

41 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Fadi Abu Sheikha and Diana Inkpen. 2011. Generation of Formal and Informal Sentences. In Proceedings of the 13th European Workshop on Natural Language Generation, pages 187–193, Nancy, France, September. Association for Computational Linguistics. find
  2. Fadi Abu Sheikha and Diana Inkpen. 2012. Learning to Classify Documents According to Formal and Informal Style. Submitted to Linguistic Issues in Language Technology (LiLT). ↗
  3. Luciana Benotti and Patrick Blackburn. 2011. Classical planning and causal implicatures. In Michael Beigl, Henning Christiansen, Thomas R. Roth-Berghofer, Anders Kofod-Petersen, Kenny R. Coventry, and Hedda R. Schmidtke, editors, Modeling and Using Context, volume 6967 of Lecture Notes in Computer Science, pages 26–39. Springer Berlin Heidelberg. find
  4. Luciana Benotti. 2010. Implicature as an Interactive Process. Ph.D. thesis, Université Henri Poincaré - Nancy I, January. ↗
  5. Douglas Biber. 1988. Variation Across Speech and Writing. Cambridge University Press. ↗
  6. Julian Brooke and Graeme Hirst. 2013. Hybrid Models for Lexical Acquisition of Correlated Styles. In Proceedings of the Sixth International Joint Conference on Natural Language Processing, pages 82–90, Nagoya, Japan, October. Asian Federation of Natural Language Processing. find
  7. Julian Brooke and Graeme Hirst. 2014. Supervised Ranking of Co-occurrence Profiles for Acquisition of Continuous Lexical Attributes. In Proceedings of COLING 2014, the 25th International Conference on Computational Linguistics: Technical Papers, pages 2172–2183, Dublin, Ireland, August. Dublin City University and Association for Computational Linguistics. find
  8. Julian Brooke, Tong Wang, and Graeme Hirst. 2010. Automatic Acquisition of Lexical Formality. In Proceedings of the 23rd International Conference on Computational Linguistics: Posters, COLING ’10, pages 90– 98, Stroudsburg, PA, USA. Association for Computational Linguistics. find
  9. Robyn Carston. 1998. Informativeness, Relevance, and Scalar Implicature. In Robyn Carston and Seiji Uchida, editors, Relevance Theory: Applications and Implications, pages 179–236. John Benjamins Publishing Co., Amsterdam. ↗
  10. Cristian Danescu-Niculescu-Mizil, Moritz Sudhof, Dan Jurafsky, Jure Leskovec, and Christopher Potts. 2013. A computational approach to politeness with application to social factors. In ACL (1), pages 250–259. The Association for Computer Linguistics. ↗ our card
  11. Cristian Danescu-Niculescu-Mizil. 2012. A Computational Approach to Linguistic Style Coordination. Ph.D. thesis, Cornell University. find
  12. Judith Degen. 2015. Investigating the distribution of some (but not all) implicatures using corpora and web-based methods. ↗ our card
  13. Nina Dethlefs, Heriberto Cuayáhuitl, Helen Hastie, Verena Rieser, and Oliver Lemon. 2014. Cluster-based Prediction of User Ratings for Stylistic Surface Realisation. In Proceedings of the 14th Conference of the European Chapter of the Association for Computational Linguistics, pages 702–711, Gothenburg, Sweden, April. Association for Computational Linguistics. find
  14. Arthur C. Graesser, Danielle S. McNamara, Max M. Louwerse, and Zhiqiang Cai. 2004. Coh-Metrix: Analysis of text on cohesion and language. Behavior Research Methods, Instruments, & Computers, 36(2):193–202. ↗
  15. Herbert Paul Grice. 1975. Logic and Conversation. In Peter Cole and Jerry L. Morgan, editors, Syntax and Semantics: Vol. 3: Speech Acts, pages 41–58. Academic Press, New York. ↗
  16. Robert M. Harnish. 1976. Logical Form and Implicature. In Thomas G. Bever, Jerrold J. Katz, and D. Terence Langendoen, editors, An Integrated Theory of Linguistic Ability, pages 313–392. Thomas Y. Crowell, New York. find
  17. Francis Heylighen and Jean-Marc Dewaele. 1999. Formality of Language: definition, measurement and behavioral determinants. Technical report, Center “Leo Apostel”, Free University of Brussels. find
  18. Eduard H. Hovy. 1990. Pragmatics and Natural Language Generation. Artificial Intelligence, 43(2):153– 197, May. ↗
  19. Richard Hudson. 1994. About 37% of Word-Tokens are Nouns. Language, 70(2):pp. 331–339. ↗
  20. Kirill Kireyev. 2009. Semantic-based Estimation of Term Informativeness. In Proceedings of Human Language Technologies: The 2009 Annual Conference of the North American Chapter of the Association for Computational Linguistics, pages 530–538, Boulder, Colorado, June. Association for Computational Linguistics. find
  21. Shibamouli Lahiri and Xiaofei Lu. 2011. Inter-rater Agreement on Sentence Formality. CoRR, abs/1109.0069. ↗
  22. Shibamouli Lahiri, Prasenjit Mitra, and Xiaofei Lu. 2011. Informality Judgment at Sentence Level and Experiments with Formality Score. In Proceedings of the 12th International Conference on Computational Linguistics and Intelligent Text Processing - Volume Part II, CICLing’11, pages 446–457, Berlin, Heidelberg. Springer-Verlag. find
  23. Helen Leckie-Tarry and David Birch. 1995. Language and Context: A Functional Linguistic Theory of Register. Pinter Publishers. find
  24. William J. M. Levelt. 1989. Speaking: From Intention to Articulation. MIT Press, Cambridge, MA. find
  25. Nancy S. Levin and Ellen F. Prince. 1986. Gapping and Causal Implicature. Paper in Linguistics, 19(3):351– 364. ↗
  26. Haiying Li, Zhiqiang Cai, and Arthur C. Graesser. 2013. Comparing Two Measures for Formality. In Proceedings of the Florida Artificial Intelligence Research Society Conference. find
  27. Rensis Likert. 1932. A Technique for the Measurement of Attitudes. Archives of Psychology, 22(140):1–55. find
  28. Ifigeneia Machili. 2014. Writing in the workplace: Variation in the Writing Practices and Formality of Eight Multinational Companies in Greece. Ph.D. thesis, University of the West of England. find
  29. George Armitage Miller. 1995. WordNet: A Lexical Database for English. Commun. ACM, 38(11):39–41, November. our card
  30. Scott Nowson, Jon Oberlander, and Alastair J. Gill. 2005. Weblogs, Genres, and Individual Differences. In Proceedings of the 27th Annual Conference of the Cognitive Science Society, pages 1666–1671. find
  31. Anna Papafragou and Julien Musolino. 2003. Scalar implicatures: experiments at the semantics-pragmatics interface. Cognition, 86(3):253 – 282. find
  32. Kelly Peterson, Matt Hohensee, and Fei Xia. 2011. Email Formality in the Workplace: A Case Study on the Enron Corpus. In Proceedings of the Workshop on Languages in Social Media, LSM ’11, pages 86–95, Stroudsburg, PA, USA. Association for Computational Linguistics. find
  33. Xuan-Hieu Phan. 2006. CRFTagger: CRF English POS Tagger. find
  34. Christopher Potts. 2005. The Logic of Conventional Implicatures. Oxford Studies in Theoretical Linguistics. Oxford University Press, Oxford. ↗
  35. Christopher Potts. 2012. Conventional implicature and expressive content. In Claudia Maienborn, Klaus von Heusinger, and Paul Portner, editors, Semantics: An International Handbook of Natural Language Meaning, volume 3, pages 2516–2536. Mouton de Gruyter, Berlin. This article was written in 2008. find
  36. Martin Rajman and Tony Hartley. 2001. Automatically predicting MT systems rankings compatible with Fluency, Adequacy or Informativeness scores. In Procs. 4th ISLE Workshop on MT Evaluation, MT Summit VIII, pages 29–34. find
  37. Jason D. M. Rennie and Tommi Jaakkola. 2005. Using Term Informativeness for Named Entity Detection. In Proceedings of the 28th Annual International ACM SI-GIR Conference on Research and Development in Information Retrieval, SIGIR ’05, pages 353–360, New York, NY, USA. ACM. find
  38. Laura Teddiman. 2009. Contextuality and Beyond: Investigating an Online Diary Corpus. In Eytan Adar, Matthew Hurst, Tim Finin, Natalie S. Glance, Nicolas Nicolov, and Belle L. Tseng, editors, ICWSM. The AAAI Press. find
  39. Mika Timonen, Timo Toivanen, Yue Teng, Chao Cheng, and Liang He. 2012. Informativeness-based Keyword Extraction from Short Documents. In Ana L. N. Fred, Joaquim Filipe, Ana L. N. Fred, and Joaquim Filipe, editors, KDIR, pages 411–421. SciTePress. find
  40. Jean Ure. 1971. Lexical density and register differentiation. Applications of Linguistics, pages 443–452. find
  41. Zhaohui Wu and Clyde Lee Giles. 2013. Measuring Term Informativeness in Context. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 259–269, Atlanta, Georgia, June. Association for Computational Linguistics. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/lahiri-2015-squinky-corpus-sentence-level-formality/