Elamur Research⌕
☰
home / papers / 2020 / An Evaluation Dataset for Identifying Communicative Functions of Sentences in English Scholarly Papers

An Evaluation Dataset for Identifying Communicative Functions of Sentences in English Scholarly Papers

Paper · 2020 · Kenichi Iwatsuki, Florian Boudin and Akiko Aizawa

Summary

Writing-assistance tools for scholarly English suggest stock phrases. They typically match keywords, so typing "propose" only returns phrases containing that word. Searching by communicative function, such as announcing the paper's aim, would serve writers better, but no public annotated data exists for testing systems that detect these functions. Iwatsuki, Boudin and Aizawa build such a resource. They take labelled example expressions from Academic Phrasebank as seeds and use them to retrieve real, unedited sentences, with surrounding context, from the ACL Anthology Sentence Corpus (AASC). Several human annotators then assign function labels and mark minimal formulaic spans. Following earlier work, the authors treat the whole sentence as the unit carrying a function. They assume the function should be recoverable from a sentence embedding, so they frame evaluation as ranking sentence vectors against a target function. They compare models such as word2vec and BERT. Contextualised models trained in-domain (Beltagy et al., 2019) score best, yet every model falls well short of human performance. This suggests that functional, not just semantic, sentence representation is still an open problem.

Key points

  • The dataset contains AASC sentences from the ACL Anthology (a single discipline: computational linguistics). Each sentence is labelled with communicative functions and minimal formulaic expressions, using a label set derived from Academic Phrasebank (Morley, 2014).
  • Academic Phrasebank alone is unsuitable as ground truth for three reasons:
    • its entries are mostly sentence fragments;
    • they lack context;
    • they mix disciplines from humanities to medicine, whose rhetorical habits differ.
  • In the retrieved corpus sentences, citations are replaced by placeholder tokens (e.g. "CITE-p-20-1-18").
  • The proposed task ranks sentence representations by a given function. In-domain contextualised models win, but all models trail humans by a wide margin.
  • Paper: LREC 2020, pp. 1712–1720. Data released at github.com/Alab-NII/FECFevalDataset.

Interesting ideas

  • I1. The resource is bootstrapped from a curated list of fragmentary, templated examples (with slots like "X"), used as queries to pull authentic in-context sentences, which humans then label; this yields realistic evaluation data cheaply while keeping a principled label set.
  • I2. Function is separated from wording: different surface phrasings can perform the same rhetorical act, so queries and labels should be functions rather than keywords.
  • I3. The authors deliberately sidestep disputes about minimal span by attaching the function to the whole sentence, while still annotating the formulaic core separately.
  • I4. A representation is probed by asking whether it can rank items by a target property, and the result is compared against human ceiling performance; a large gap is itself reported as a finding.
  • I5. Genre and discipline are controlled because rhetorical conventions vary across fields, so evaluating on a mixed-domain resource would blur conclusions.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): seed short speech-act or construction templates, harvest matching corpus sentences, and have parsers or the LLM teacher propose labels before human spot-checks, producing silver data tied to an explicit inventory.
  • I2 (frames): prag could tag sentences with a communicative-function or speech-act layer independent of lexical choice, letting "we propose" and "this paper presents" map to the same act.
  • I3 (frames): prag could assign speech acts at sentence level and record the triggering subtree from the UD tree as an explanation, keeping both robustness and traceability.
  • I4 (learn): Mova's evaluations could include human or oracle ceilings and ranking-style probes, so gains are judged against what is achievable, not only against baselines.
  • I5 (answer): keep test sets for school courses, word problems and fables separate, and report per-genre scores rather than one pooled number.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓The University of TokyoTokyo · Japan45 works · 8,692 citations🔬National Institute of InformaticsTokyo · Japan21 works · 3,045 citations🔬Laboratoire des Sciences du Numérique de NantesNantes · France3 works · 416 citations

📚 Shelves 5 Computer Science · 9,086 works Artificial Intelligence · 8,889 works Arts and Humanities · 9,296 works Language and Linguistics · 9,206 works Literature and Literary Theory · 340 works

🧭 Topics 2 English · 1,373 works Evaluation & benchmarks · 685 works

Cite

Cite as Kenichi Iwatsuki, Florian Boudin and Akiko Aizawa. 2020. An Evaluation Dataset for Identifying Communicative Functions of Sentences in English Scholarly Papers. Proceedings of the Twelfth Language Resources and Evaluation Conference.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{iwatsuki2020evaluation,
  title = {{An Evaluation Dataset for Identifying Communicative Functions of Sentences in English Scholarly Papers}},
  author = {Kenichi Iwatsuki, Florian Boudin and Akiko Aizawa},
  booktitle = {Proceedings of the Twelfth Language Resources and Evaluation Conference},
  year = {2020},
  url = {https://aclanthology.org/2020.lrec-1.212/}
}

References

46 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ackermann, K. and Chen, Y.-H. (2013). Developing the academic collocation list (ACL) — a corpus-driven and expert-judged approach. Journal of English for Academic Purposes, 12(4):235 – 247. ↗
  2. Akbik, A., Blythe, D., and Vollgraf, R. (2018). Contextual string embeddings for sequence labeling. In Proceedings of the 27th International Conference on Computational Linguistics, pages 1638–1649. our card
  3. AlHassan, L. and Wood, D. (2015). The effectiveness of focused instruction of formulaic sequences in augmenting l2 learners’ academic writing skills: A quantitative research study. Journal of English for Academic Purposes, 17:51–62. ↗
  4. Beltagy, I., Lo, K., and Cohan, A. (2019). SciBERT: A pretrained language model for scientific text. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the Ninth International Joint Conference on Natural Language Processing, pages 3606–3611. ↗ our card
  5. Biber, D., Connor, U., and Upton, T. A. (2007). Discourse on the Move: Using Corpus Analysis to Describe Discourse Structures. John Benjamins Publishing. find
  6. Brooke, J., Hammond, A., Jacob, D., Tsang, V., Hirst, G., and Shein, F. (2015). Building a lexicon of formulaic language for language learners. In Proceedings of the 11th Workshop on Multiword Expressions, pages 96– 104. find
  7. Chang, J. and Chang, J. (2015). WriteAhead2: Mining lexical grammar patterns for assisted writing. In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Demonstrations, pages 106–110. find
  8. Chen, Y.-H. and Baker, P. (2010). Lexical bundles in L1 and L2 academic writing. Language Learning & Technology, 14(2):30–49. ↗
  9. Conneau, A. and Kiela, D. (2018). SentEval: An evaluation toolkit for universal sentence representations. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018). our card
  10. Conneau, A., Kiela, D., Schwenk, H., Barrault, L., and Bordes, A. (2017). Supervised learning of universal sentence representations from natural language inference data. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 670–680. our card
  11. Cortes, V. (2013). The purpose of this study is to: Connecting lexical bundles and moves in research article introductions. Journal of English for Academic Purposes, 12(1):33–43. ↗
  12. Cotos, E., Huffman, S., and Link, S. (2015). Furthering and applying move/step constructs: Technology-driven marshalling of Swalesian genre theory for EAP pedagogy. Journal of English for Academic Purposes, 19:52 – 72. ↗
  13. Cunningham, K. J. (2017). A phraseological exploration of recent mathematics research articles through key phrase frames. Journal of English for Academic Purposes, 25:71 – 83. ↗
  14. Dayrell, C., Candido, A. J., Lima, G., Machado, D. J., Copestake, A., Feltrim, V., Tagnin, S., and Aluisio, S. (2012). Rhetorical move detection in English abstracts: Multi-label sentence classifiers and their annotated corpora. In Proceedings of the Eight International Conference on Language Resources and Evaluation (LREC’12), pages 1604–1609. our card
  15. Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. (2019). BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186. our card
  16. Durrant, P. and Mathews-Aydınlı, J. (2011). A function-first approach to identifying formulaic language in academic writing. English for Specific Purposes, 30:58–72. ↗
  17. Fiacco, J., Cotos, E., and Rosé, C. (2019). Towards enabling feedback on rhetorical structure with neural sequence models. In Proceedings of the 9th International Conference on Learning Analytics & Knowledge, pages 310–319. find
  18. Halliday, M. A. K. and Matthiessen, C. M. (2014). Halliday’s Introduction to Functional Grammar. Routledge. ↗
  19. Hirohata, K., Okazaki, N., Ananiadou, S., and Ishizuka, M. (2008). Identifying sections in scientific abstracts using conditional random fields. In Proceedings of the Third International Joint Conference on Natural Language Processing: Volume-I, pages 381–388. find
  20. Hyland, K. (2008). As can be seen: Lexical bundles and disciplinary variation. English for Specific Purposes, 27(1):4–21. ↗
  21. Iwatsuki, K. and Aizawa, A. (2018). Using formulaic expressions in writing assistance systems. In Proceedings of the 27th International Conference on Computational Linguistics, pages 2678–2689. find
  22. Jalali, Z. S. and Moini, M. R. (2014). Structure of lexical bundles in introduction section of medical research articles. Procedia - Social and Behavioral Sciences, 98:719 – 726. Proceedings of the International Conference on Current Trends in ELT. find
  23. Jeong, S., Nam, S., and Park, H.-Y. (2014). An ontology-based biomedical research paper authoring support tool. Science Editing, 1(1):37–42. ↗
  24. Kiros, R., Zhu, Y., Salakhutdinov, R. R., Zemel, R., Urtasun, R., Torralba, A., and Fidler, S. (2015). Skip-thought vectors. In C. Cortes, et al., editors, Advances in Neural Information Processing Systems 28, pages 3294–3302. Curran Associates, Inc. ↗
  25. Liu, Y., Wang, X., Liu, M., and Wang, X. (2016). Write-righter: An academic writing assistant system. In Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence, pages 4373–4374. find
  26. Lorés, R. (2004). On RA abstracts: from rhetorical structure to thematic organisation. English for Specific Purposes, 23(3):280 – 302. find
  27. Lu, X., Yoon, J., and Kisselev, O. (2018). A phrase-frame list for social science research article introductions. Journal of English for Academic Purposes, 36:76 – 85. ↗
  28. Manning, C. D. and Schütze, H., (1999). Foundations of Statistical Natural Language Processing, page 535. MIT Press. find
  29. Maswana, S., Kanamaru, T., and Tajino, A. (2015). Move analysis of research articles across five engineering fields: What they share and what they do not. Ampersand, 2:1–11. ↗
  30. Mikolov, T., Sutskever, I., Chen, K., Corrado, G. S., and Dean, J. (2013). Distributed representations of words and phrases and their compositionality. In Advances in Neural Information Processing Systems, pages 3111– 3119. ↗ our card
  31. Mizumoto, A., Hamatani, S., and Imao, Y. (2017). Applying the bundle–move connection approach to the development of an online writing support tool for research articles. Language Learning, 67(4):885–921. ↗
  32. Morley, J. (2014). Academic Phrasebank. http://www.phrasebank.manchester.ac.uk/. ↗
  33. Omidian, T., Shahriari, H., and Siyanova-Chanturia, A. (2018). A cross-disciplinary investigation of multi-word expressions in the moves of research article abstracts. Journal of English for Academic Purposes, 36:1 – 14. ↗
  34. Ozturk, I. (2007). The textual organisation of research article introductions in applied linguistics: Variability within a single discipline. English for Specific Purposes, 26(1):25 – 38. ↗
  35. Pan, F., Reppen, R., and Biber, D. (2016). Comparing patterns of L1 versus L2 English academic professionals: Lexical bundles in telecommunications research journals. Journal of English for Academic Purposes, 21:60 – 71. ↗
  36. Pendar, N. and Cotos, E. (2008). Automatic identification of discourse moves in scientific article introductions. In Proceedings of the Third Workshop on Innovative Use of NLP for Building Educational Applications, pages 62– 70. find
  37. Pennington, J., Socher, R., and Manning, C. (2014). Glove: Global vectors for word representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing, pages 1532–1543. our card
  38. Qin, J. (2014). Use of formulaic bundles by non-native English graduate writers and published authors in applied linguistics. System, 42:220 – 231. ↗
  39. Simpson-Vlach, R. and Ellis, N. C. (2010). An Academic Formulas List: New Methods in Phraseology Research. Applied Linguistics, 31(4):487–512. ↗
  40. Swales, J. M. (1981). Aspects of Article Introductions. The University of Michigan Press. find
  41. Swales, J. M. (1990). Genre Analysis: English in academic and research settings. Cambridge University Press. find
  42. Swales, J. M. (2004). Research Genres: Explorations and Applications. Cambridge University Press. find
  43. Thelwall, M. (2019). The rhetorical structure of science? a multidisciplinary analysis of article headings. Journal of Informetrics, 13(2):555 – 563. ↗
  44. Wang, A., Singh, A., Michael, J., Hill, F., Levy, O., and Bowman, S. (2018). GLUE: A multi-task benchmark and analysis platform for natural language understanding. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 353–355. Wu, J.-C., Chang, Y.-C., Liou, H.-C., and Chang, J. S. ↗ our card
  45. (2006). Computational analysis of move structures in academic abstracts. In Proceedings of the COLING/ACL 2006 Interactive Presentation Sessions, pages 41–44. find
  46. Yen, T.-H., Wu, J.-C., Chang, J., Boisson, J., and Chang, J. (2015). WriteAhead: Mining grammar patterns in corpora for assisted writing. In Proceedings of ACL-IJCNLP 2015 System Demonstrations, pages 139–144. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/iwatsuki-2020-evaluation-dataset-identifying-communicative-functions/