Elamur Research⌕
☰
home / papers / 2021 / Development of an Extractive Title Generation System Using Titles of Papers of Top Conferences for Intermediate English Students

Development of an Extractive Title Generation System Using Titles of Papers of Top Conferences for Intermediate English Students

Paper · 2021 · Kento Kaku, Masato Kikuchi, Tadachika Ozono et al.

Summary

The paper targets students with intermediate or beginner English who find it hard to write good academic paper titles because they do not know what titles in their field usually look like. The authors build an extractive title generator that starts from a paper's abstract and runs in three stages:

  • A BERT-based tagger marks which abstract words are likely to appear in the title.
  • These words are grouped into short "title parts", which the user can edit. The parts are then permuted into candidate titles.
  • A grammar checker removes candidates whose phrase structure does not match patterns collected from real titles. A BERT evaluation model, trained on titles from top AI conferences, scores the remaining candidates.

The output is much shorter and more title-like than TextRank or LexRank summaries, and its ROUGE scores are comparable to theirs. The evaluation model separated top-conference titles from other titles better than Japanese students did. The grammar filter discarded more than half of the ungrammatical candidates while keeping most of the well-formed ones. This matters because writing tools for non-native authors can be built without any learner corpus of titles.

Key points

  • Keyword extraction: BERT does per-token binary classification. A token is labelled 1 if it occurs in the paper's real title, and a threshold on a linear score decides which tokens are kept.
  • Grammar checker: it uses phrase/clause-level syntax-tree skeletons, with POS leaves removed, harvested from 9,232 AAAI titles. On candidate permutations it reached TP 0.14, FP 0.36, FN 0.03 and TN 0.47, so roughly 57% of the bad candidates were removed and about 82% of the good ones were kept.
  • Title evaluation model: it scored 0.93 on AAAI titles, 0.88 on IJCAI and 0.70 on JSAI, a non-top venue. Master's students (M2) scored 0.73 and fourth-year undergraduates (B4) scored 0.63 on AAAI titles.
  • Comparison with summarisers: average output length was 7.9 words, against 23.7 for LexRank and 33.1 for TextRank. ROUGE-1 was 0.0558, against 0.0613 for LexRank and 0.0456 for TextRank.
  • Worked example (Stachniss & Burgard, AAAI 2005): the true title appeared only as candidate 7, with score 0.66. Ranking is therefore still weak.

Interesting ideas

  • I1. When no corpus of human errors exists, real titles with scrambled word order serve as synthetic ungrammatical negatives for training a well-formedness judge.
  • I2. Grammaticality is checked by matching the phrase-level skeleton of a candidate's tree, with leaves removed, against an inventory of skeletons seen in real text, which is an explainable and non-neural filter.
  • I3. Supervision comes for free from aligned pairs: an abstract token is labelled positive simply if it appears in the paired title, with no manual annotation needed.
  • I4. Words are grouped into multiword "title parts" before reordering, which keeps technical terms intact and sharply shrinks the permutation space.
  • I5. The evaluation scorer is benchmarked against human groups of different expertise (B4, M2) rather than only against other systems.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Scrambled versions of real sentences could act as cheap negative controls, checking that the parser and frame scorers really drop in confidence on broken input.
  • I2 (back): Generated sentences could be accepted only if their UD tree shape (deprel/POS skeleton) appears among shapes harvested from treebanks, giving a transparent sanity check on linearisation.
  • I3 (back): The same trick on text/summary pairs could teach the compressor which graph nodes must survive into the short version of a text.
  • I4 (back): Generation could order whole phrase units (entity names, compounds) rather than single tokens, cutting the search and protecting multiword terms.
  • I5 (answer): Reporting student-reader scores next to human baselines at different levels would show where Mova sits on a realistic skill scale.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Nagoya Institute of TechnologyNagoya · Japan2 works · 1 citation

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 9,086 worksArtificial Intelligence · 8,889 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 worksHistory and Philosophy of Science · 18 works

🧭 Topics 1Research subjects and languages the work is about.English · 1,373 works

Cite

Cite as Kento Kaku, Masato Kikuchi, Tadachika Ozono and Toramatsu Shintani. 2021. Development of an Extractive Title Generation System Using Titles of Papers of Top Conferences for Intermediate English Students. arXiv:2110.04204 · 2021 10th International Congress on Advanced Applied Informatics (IIAI-AAI), pp.59-64, 2021.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.1109/iiai-aai53430.2021.00010 · arXiv 2110.04204

@misc{kaku2021development,
  title = {{Development of an Extractive Title Generation System Using Titles of Papers of Top Conferences for Intermediate English Students}},
  author = {Kento Kaku, Masato Kikuchi, Tadachika Ozono and Toramatsu Shintani},
  year = {2021},
  doi = {10.1109/iiai-aai53430.2021.00010},
  eprint = {2110.04204},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.1109/iiai-aai53430.2021.00010}
}

References

12 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. J. Devlin, M. Chang, K. Lee, K. Toutanova, “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding,” Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, vol.1, pp.4171-4186, 2019. TABLE IV E XPERIMENTAL RESULTS OF GRAMMAR CHECKER . N OTE THAT ALMOST ALL THE TEST SENTENCES , WHICH ARE PERMUTATIONS OF EXTRACTED KEYWORDS , ARE GRAMMATICALLY INCORRECT . Manual judgment Positive Negative TP FP 0.14 0.36 FN TN 0.03 0.47 Assessment by grammar checker Positive Negative
  2. M. Banko, V. O. Mittal, and M. J. Witbrock, “Headline Generation Based on Statistical Translation,” 38th Annual Meeting of the Association for Computational Linguistics, pp.318-325, 2000. find
  3. R. Jin, A. G. Hauptmann, “A New Probabilistic Model for Title Generation” 19th International Conference on Computational Linguistics, pp.404-410, 2002. find
  4. D. Bahdanau, K. Cho, Y. Bengio, “Neural machine translation by jointly learning to align and translate,” 3rd International Conference on Learning Representations, pp.1-15, 2015. ↗
  5. A. Rush, S. Chopra, and J. Weston, “A neural attention model for abstractive sentence summarization,” Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pp.379-389, 2015. find
  6. J. Cheng, M. Lapata, “Neural summarization by extracting sentences and words,” Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, vol.1, pp.484-494, 2016.
  7. M. Rei, H. Yannakoudakis, “Compositional Sequence Labeling Models for Error Detection in Learner Writing,” Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, vol.1, pp.1181-1191, 2016.
  8. Z. Yuan, T. Briscoe, “Grammatical error correction using neural machine translation,” The 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp.380–386, 2016. find
  9. T. Ohbe, T. Ozono, T. Shintani, “A Title Generator for a Paper Writing Support Agent,” Joint Agent Workshop & Symposium 2017, pp.232-237, 2017. find
  10. P. Mishra, C. Diwan, S. Srinivasa, G. Srinivasaraghavan, “Automatic Title Generation for Text with Pre-trained Transformer Language Model,” 15th IEEE International Conference on Semantic Computing, pp.17-24, 2021. find
  11. G. Erkan, D. R. Radev, “LexRank: Graph-based Lexical Centrality as Salience in Text Summarization,” Journal of Artificial Intelligence Research, vol.22, pp.457-479, 2004. ↗
  12. R. Mihalcea, P. Tarau, “TextRank: Bringing Order into Text,” Proceedings of the 2004 Conference on Empirical Methods in Natural Language Processing, pp.404-411, 2004. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/kaku-2021-development-extractive-title-generation-system/