Elamur Research⌕
☰
home / papers / 2020 / Qu'apporte BERT à l'analyse syntaxique en constituants discontinus ? Une suite de tests pour évaluer les prédictions de structures syntaxiques discontinues en anglais (What does BERT contribute to discontinuous constituency parsing ? A test suite to evaluate discontinuous constituency structure predictions in English)

Qu'apporte BERT à l'analyse syntaxique en constituants discontinus ? Une suite de tests pour évaluer les prédictions de structures syntaxiques discontinues en anglais (What does BERT contribute to discontinuous constituency parsing ? A test suite to evaluate discontinuous constituency structure predictions in English)

Summary

The paper examines how the pretrained language model BERT affects the parsing of English sentences with discontinuous constituents. These are structures in which a single tree node spans several non-adjacent fragments of the sentence. The author built and released a test suite based on the validation section of the Discontinuous Penn Treebank. He selected all sentences with discontinuities, except those where the discontinuity is caused only by punctuation, and manually annotated which syntactic phenomenon causes it. The same transition-based parser of Coavoux & Cohen (2019) was then compared in two variants: without BERT and with BERT vectors fine-tuned during training. With BERT, the parser reaches an F1 of about 95. This was the best result at the time for discontinuous parsing of the PTB. BERT helps most with extrapositions, but the model still recognizes some of the difficult phenomena unreliably. The value of the paper lies in providing an automatic per-phenomenon evaluation instead of a single overall F1. Previously, a similar test suite existed only for German (Discosuite on the Tiger corpus).

Key points

  • Test suite: 266 sentences, i.e. approximately 16% of the validation section of the corpus. Discontinuities are divided into six types: long-distance extractions, quotations with an inserted main clause, left extraposition of a quotation, it-extraposition, other extrapositions, and subject–verb inversion. 21 sentences contain two phenomena each, and one more contains three. The annotations and evaluation scripts are published at gitlab.com/mcoavoux/disco-eval-ptb.
  • Model: the output of the bert-base-cased model from the transformers library was added to the parser's input representations (word embeddings and a character-level bi-LSTM). The BERT parameters were fine-tuned jointly with the parser.
  • Overall results on the test section (discodop evaluator): F1 rose from 90.9 to 94.8, which matches the result of Corro (2020). F1 for discontinuous constituents alone rose from 67.3 to 73.6.
  • Individual phenomena: the proportion of correctly recognized extrapositions rose from 23.1% to 59%, and of it-extrapositions from 41.7% to 75%.
  • Conclusion: BERT improves all metrics, but there is still room for improvement on difficult discontinuous constructions. As future work, the author plans to try other semi-supervised learning methods.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics Evaluation & benchmarks Syntax & grammar formalisms Interpretability

Cite

Cite as Maximin Coavoux. 2020. Qu'apporte BERT à l'analyse syntaxique en constituants discontinus ? Une suite de tests pour évaluer les prédictions de structures syntaxiques discontinues en anglais (What does BERT contribute to discontinuous constituency parsing ? A test suite to evaluate discontinuous constituency structure predictions in English). Actes de la 6e conférence conjointe Journées d'Études sur la Parole (JEP, 33e édition), Traitement Automatique des Langues Naturelles (TALN, 27e édition), Rencontre des Étudiants Chercheurs en Informatique pour le Traitement Automatique des Langues (RÉCITAL, 22e édition). Volume 2 : Traitement Automatique des Langues Naturelles.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{coavoux2020qu,
  title = {{Qu'apporte BERT à l'analyse syntaxique en constituants discontinus ? Une suite de tests pour évaluer les prédictions de structures syntaxiques discontinues en anglais (What does BERT contribute to discontinuous constituency parsing ? A test suite to evaluate discontinuous constituency structure predictions in English)}},
  author = {Maximin Coavoux},
  booktitle = {Actes de la 6e conférence conjointe Journées d'Études sur la Parole (JEP, 33e édition), Traitement Automatique des Langues Naturelles (TALN, 27e édition), Rencontre des Étudiants Chercheurs en Informatique pour le Traitement Automatique des Langues (RÉCITAL, 22e édition). Volume 2 : Traitement Automatique des Langues Naturelles},
  year = {2020},
  url = {https://aclanthology.org/2020.jeptalnrecital-taln.17/}
}

References

16 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. B RANTS S., D IPPER S., E ISENBERG P., H ANSEN -S CHIRRA S., K ÖNIG E., L EZIUS W., R OHRER C., S MITH G. & U SZKOREIT H. (2004). Tiger : Linguistic interpretation of a german corpus. Research on language and computation, 2(4), 597–620. find
  2. C OAVOUX M. & C OHEN S. B. (2019). Discontinuous constituency parsing with a stack-free transition system and a dynamic oracle. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics : Human Language Technologies, Volume 1 (Long and Short Papers), p. 204–217, Minneapolis, Minnesota : Association for Computational Linguistics. doi : 10.18653/v1/N19-1018. ↗
  3. C OAVOUX M., C RABBÉ B. & C OHEN S. B. (2019). Unlexicalized transition-based discontinuous constituency parsing. Transactions of the Association for Computational Linguistics, 7, 73–89. doi : 10.1162/tacl_a_00255. ↗
  4. C ORRO C. (2020). Span-based discontinuous constituency parsing : a family of exact chart-based algorithms with time complexities from O(n 6 ) down to O(n 3 ). arXiv preprint : 2003.13785. ↗
  5. D EVLIN J., C HANG M.-W., L EE K. & T OUTANOVA K. (2019). BERT : Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics : Human Language Technologies, Volume 1 (Long and Short Papers), p. 4171–4186, Minneapolis, Minnesota : Association for Computational Linguistics. doi : 10.18653/v1/N19-1423. ↗ our card
  6. E VANG K. (2011). Parsing discontinuous constituents in English. Mémoire de master, University of Tübingen. find
  7. E VANG K. & K ALLMEYER L. (2011). PLCFRS parsing of English discontinuous constituents. In Proceedings of the 12th International Conference on Parsing Technologies, p. 104–116, Dublin, Ireland : Association for Computational Linguistics. find
  8. K ITAEV N., C AO S. & K LEIN D. (2019). Multilingual constituency parsing with self-attention and pre-training. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, p. 3499–3505, Florence, Italy : Association for Computational Linguistics. doi : 10.18653/v1/P19-1340. ↗ our card
  9. K ITAEV N. & K LEIN D. (2018). Constituency parsing with a self-attentive encoder. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1 : Long Papers), p. 2676–2686, Melbourne, Australia : Association for Computational Linguistics. doi : 10.18653/v1/P18-1249. ↗ our card
  10. M AIER W. (2015). Discontinuous incremental shift-reduce parsing. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1 : Long Papers), p. 1202–1212, Beijing, China : Association for Computational Linguistics. doi : 10.3115/v1/P15-1116. ↗
  11. M AIER W., K AESHAMMER M., B AUMANN P. & K ÜBLER S. (2014). Discosuite - a parser test suite for German discontinuous structures. In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC’14), p. 2905–2912, Reykjavik, Iceland : European Language Resources Association (ELRA). find
  12. S TANOJEVI Ć M. & G. A LHAMA R. (2017). Neural discontinuous constituency parsing. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, p. 1666– 1676, Copenhagen, Denmark : Association for Computational Linguistics. doi : 10.18653/v1/D17-1174. ↗
  13. VAN C RANENBURGH A., S CHA R. & B OD R. (2016). Data-oriented parsing with discontinuous constituents and function tags. Journal of Language Modelling, 4(1), 57–111. find
  14. V IJAY -S HANKER K., W EIR D. J. & J OSHI A. K. (1987). Characterizing structural descriptions produced by various grammatical formalisms. In 25th Annual Meeting of the Association for Computational Linguistics, p. 104–111, Stanford, California, USA : Association for Computational Linguistics. doi : 10.3115/981175.981190. ↗ our card
  15. W OLF T., D EBUT L., S ANH V., C HAUMOND J., D ELANGUE C., M OI A., C ISTAC P., R AULT T., L OUF R., F UNTOWICZ M. & B REW J. (2019). Huggingface’s transformers : State-of-the-art natural language processing. arXiv preprint : 1910.03771. ↗ our card
  16. Z HOU J. & Z HAO H. (2019). Head-driven phrase structure grammar parsing on Penn treebank. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, p. 2396– 2408, Florence, Italy : Association for Computational Linguistics. doi : 10.18653/v1/P19-1230. ↗ our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/coavoux-2020-qu-apporte-bert-l-analyse/