Elamur Research⌕
☰
home / papers / 2023 / Constituency Parsing using LLMs

Constituency Parsing using LLMs

Summary

The paper asks whether modern large language models can produce phrase-structure trees, a task where supervised parsers reach about 96% while unsupervised methods remain below 65%. The authors cast constituency parsing as sequence generation. They serialize trees in three ways: bracket strings, transition-action sequences, and natural-language span statements. They then test a range of closed- and open-source LLMs in zero-shot, few-shot and fully fine-tuned modes, against strong graph-based parsers, on one in-domain and five out-of-domain test sets. LLMs clearly beat older sequence-based parsers but still trail graph-based systems after supervised training. Much of the gap comes from outputs that are malformed or do not match the input sentence. To address this, the authors add two fixes: Learning from Erroneous Samples (LES) and Parsing with Multi-agent Collaboration (PMC). Both reduce broken trees and raise F1 in all settings. The work matters because it shows a practical route to annotation-light syntactic parsing with general-purpose LLMs.

Key points

  • Trees are linearized three ways: bracket-based (following Vinyals et al.), transition-based (following Liu and Zhang) and span-based. The span-based format reads most like natural language but yields much longer sequences.
  • Evaluation covers zero-shot, few-shot and full fine-tuning on six datasets. The treebanks referenced include the Penn Treebank (PTB) and the Multi-domain Constituent Treebank (MCTB).
  • The core weakness of LLM parsers is the generation of invalid (ill-formed) trees and unfaithful trees, meaning trees whose words diverge from the input sentence.
  • LES places faulty example parses with error annotations in the prompt. PMC uses two extra agents that check validity and faithfulness and send feedback so the parsing agent can revise its tree.
  • In the zero-shot setting the enhanced approach reaches an F1 of 76.87, which the authors report as a new best for end-to-end unsupervised constituency parsing.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing · AI & machine learning

Topics: Syntax & grammar formalisms

Institutions

🎓University of CambridgeCambridge · United Kingdom119 works · 110 950 citations🎓Harbin Institute of TechnologyHarbin · China58 works · 25 250 citations🎓Westlake UniversityHangzhou · China22 works · 6 635 citations🎓Soochow UniversitySuzhou · China34 works · 3 917 citations🎓Southeast UniversityNanjing · China2 works · 149 citations

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.1109/taslpro.2025.3600867 · arXiv 2310.19462

@misc{bai2023constituency,
  title = {{Constituency Parsing using LLMs}},
  author = {Bai, Xuefeng and Wu, Jialong and Chen, Yulong and Wang, Zhongqing and Chen, Kehai and Zhang, Min and Zhang, Yue},
  year = {2023},
  doi = {10.1109/taslpro.2025.3600867},
  eprint = {2310.19462},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.1109/taslpro.2025.3600867}
}

References

76 references, 31 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. M. P. Marcus, B. Santorini, and M. A. Marcinkiewicz, “Building a large annotated corpus of English: The Penn Treebank,” Computational Linguistics, vol. 19, no. 2, pp. 313–330, 1993. find our card
  2. M. Collins, “Three generative, lexicalised models for statistical parsing,” in Proc. of ACL. Association for Computational Linguistics, 1997, pp. 16–23. find our card
  3. E. Charniak, “A maximum-entropy-inspired parser,” in Proc. of NAACL. Association for Computational Linguistics, 2000, p. 132–139. find our card
  4. S. Petrov and D. Klein, “Improved inference for unlexicalized parsing,” in Proc. of NAACL. Association for Computational Linguistics, 2007, pp. 404–411. find our card
  5. X. Wang, H. Pham, P. Yin, and G. Neubig, “A tree-based decoder for neural machine translation,” in Proc. of EMNLP. Association for Computational Linguistics, 2018, pp. 4772–4777. find
  6. M. Jiang and J. Diesner, “A constituency parsing tree based method for relation extraction from abstracts of scholarly publications,” in Proc. of TextGraphs. Association for Computational Linguistics, 2019, pp. 186–191. find
  7. J. Xu and G. Durrett, “Neural extractive text summarization with syntactic compression,” in Proc. of EMNLP. Association for Computational Linguistics, 2019, pp. 3292–3303. find
  8. N. Kitaev, S. Cao, and D. Klein, “Multilingual constituency parsing with self-attention and pre-training,” in Proc. of ACL. Association for Computational Linguistics, 2019, pp. 3499–3505. find our card
  9. X. Gu, Y. Shen, J. Shen, J. Shang, and J. Han, “Phrase-aware unsupervised constituency parsing,” in Proc. of ACL. Association for Computational Linguistics, 2022, pp. 6406–6415. find
  10. Y. Chen, Y. Liu, L. Dong, S. Wang, C. Zhu, M. Zeng, and Y. Zhang, “AdaPrompt: Adaptive model training for prompt-based NLP,” in Findings of ACL: EMNLP 2022. Association for Computational Linguistics, 2022, pp. 6057–6068. find
  11. X. Sun, X. Li, J. Li, F. Wu, S. Guo, T. Zhang, and G. Wang, “Text classification via large language models,” in Findings of ACL: EMNLP 2023. Association for Computational Linguistics, 2023, pp. 8990–9005. find our card
  12. Z. Zhang, S. Wang, Y. Xu, Y. Fang, W. Yu, Y. Liu, H. Zhao, C. Zhu, and M. Zeng, “Task compass: Scaling multi-task pre-training with task prefix,” in Findings of ACL: EMNLP 2022. Association for Computational Linguistics, 2022, pp. 5671–5685. find
  13. A. Toroghi, W. Guo, A. Pesaranghader, and S. Sanner, “Verifiable, debuggable, and repairable commonsense logical reasoning via LLM-based theory resolution,” in Proc. of EMNLP. Association for Computational Linguistics, 2024, pp. 6634–6652. find
  14. T. Goyal, J. J. Li, and G. Durrett, “News summarization and evaluation in the era of gpt-3,” ArXiv preprint, vol. abs/2209.12356, 2022. find
  15. Y. Chen, Y. Liu, R. Xu, Z. Yang, C. Zhu, M. Zeng, and Y. Zhang, “UniSumm and SummZoo: Unified model and diverse benchmark for few-shot summarization,” in Proc. of ACL. Association for Computational Linguistics, 2023, pp. 12 833–12 855. find
  16. T. Zhang, F. Ladhak, E. Durmus, P. Liang, K. McKeown, and T. B. Hashimoto, “Benchmarking large language models for news summarization,” Transactions of the Association for Computational Linguistics, vol. 12, pp. 39–57, 2024. find our card
  17. W. Jiao, W. Wang, J.-t. Huang, X. Wang, and Z. Tu, “Is chatgpt a good translator? a preliminary study,” ArXiv preprint, vol. abs/2301.08745, 2023. find our card
  18. C. Qin, A. Zhang, Z. Zhang, J. Chen, M. Yasunaga, and D. Yang, “Is ChatGPT a general-purpose natural language processing task solver?” in Proc. of EMNLP. Association for Computational Linguistics, 2023, pp. 1339–1384. find our card
  19. M. Collins, J. Hajic, L. Ramshaw, and C. Tillmann, “A statistical parser for Czech,” in Proc. of ACL. Association for Computational Linguistics, 1999, pp. 505–512. find our card
  20. G. Durrett and D. Klein, “Neural CRF parsing,” in Proc. of ACL. Association for Computational Linguistics, 2015, pp. 302–312. find
  21. H. Yamada and Y. Matsumoto, “Statistical dependency analysis with support vector machines,” in Proceedings of the Eighth International Conference on Parsing Technologies, 2003, pp. 195–206. find our card
  22. K. Sagae and A. Lavie, “A classifier-based parser with linear run-time complexity,” in Proceedings of the Ninth International Workshop on Parsing Technology. Association for Computational Linguistics, 2005, pp. 125–132. find our card
  23. J. Nivre and R. McDonald, “Integrating graph-based and transition-based dependency parsers,” in Proc. of ACL. Association for Computational Linguistics, 2008, pp. 950–958. find our card
  24. Y. Zhang and S. Clark, “Transition-based parsing of the Chinese treebank using a global discriminative model,” in Proc. of IWPT. Association for Computational Linguistics, 2009, pp. 162–171. find our card
  25. O. Vinyals, L. Kaiser, T. Koo, S. Petrov, I. Sutskever, and G. E. Hinton, “Grammar as a foreign language,” in Proc. of NeurIPS, 2015, pp. 2773– 2781. find
  26. H. Kamigaito, K. Hayashi, T. Hirao, H. Takamura, M. Okumura, and M. Nagata, “Supervised attention for sequence-to-sequence constituency parsing,” in Proc. of IJCNLP (Volume 2: Short Papers). Asian Federation of Natural Language Processing, 2017, pp. 7–12. find
  27. L. Liu, M. Zhu, and S. Shi, “Improving sequence-to-sequence constituency parsing,” in Proc. of AAAI. AAAI Press, 2018, pp. 4873–4880. find
  28. N. Kitaev and D. Klein, “Constituency parsing with a self-attentive encoder,” in Proc. of ACL. Association for Computational Linguistics, 2018, pp. 2676–2686. find our card
  29. Y. Zhang, H. Zhou, and Z. Li, “Fast and accurate neural CRF constituency parsing,” in Proc. of IJCAI. ijcai.org, 2020, pp. 4046–4053. find
  30. C. Dyer, A. Kuncoro, M. Ballesteros, and N. A. Smith, “Recurrent neural network grammars,” in Proc. of NAACL-HLT. Association for Computational Linguistics, 2016, pp. 199–209. find
  31. J. Cross and L. Huang, “Span-based constituency parsing with a structure-label system and provably optimal dynamic oracles,” in Proc. of EMNLP. Association for Computational Linguistics, 2016, pp. 1–11. find our card
  32. J. Liu and Y. Zhang, “In-order transition-based constituent parsing,” Transactions of the Association for Computational Linguistics, vol. 5, pp. 413–424, 2017. find our card
  33. D. Fernández-González and C. Gómez-Rodríguez, “Enriched in-order linearization for faster sequence-to-sequence constituent parsing,” in Proc. of ACL. Association for Computational Linguistics, 2020, pp. 4092–4099. find
  34. T.-T. Nguyen, X.-P. Nguyen, S. Joty, and X. Li, “A conditional splitting framework for efficient constituency parsing,” in Proc. of ACL. Association for Computational Linguistics, 2021, pp. 5795–5807. find
  35. S. Yang and K. Tu, “Bottom-up constituency parsing and nested named entity recognition with pointer networks,” in Proc. of ACL. Association for Computational Linguistics, 2022, pp. 2403–2416. find
  36. Y. Tian, F. Xia, and Y. Song, “Large language models are no longer shallow parsers,” in Proc. of ACL. Association for Computational Linguistics, 2024, pp. 7131–7142. find
  37. T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. M. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Ber… find our card
  38. A. Chowdhery, S. Narang, J. Devlin, M. Bosma, G. Mishra, A. Roberts, P. Barham, H. W. Chung, C. Sutton, S. Gehrmann et al., “Palm: Scaling language modeling with pathways,” J. Mach. Learn. Res., vol. 24, pp. 240:1–240:113, 2023. find our card
  39. H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar et al., “Llama: Open and efficient foundation language models,” ArXiv preprint, vol. abs/2302.13971, 2023. find our card
  40. A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskever et al., “Language models are unsupervised multitask learners,” OpenAI blog, vol. 1, no. 8, p. 9, 2019. find
  41. V. Sanh, A. Webson, C. Raffel, S. H. Bach, L. Sutawika, Z. Alyafeai, A. Chaffin, A. Stiegler, A. Raja, M. Dey, M. S. Bari, C. Xu, U. Thakker, S. S. Sharma, E. Szczechla, T. Kim, G. Chhablani, N. V. Nayak, D. Datta, J. Chang, M. T. Jiang, H. Wang, M. Manica, S. Shen, Z. X. Yong, H. Pandey, R. Bawden,… find our card
  42. A. Webson and E. Pavlick, “Do prompt-based models really understand the meaning of their prompts?” in Proc. of NAACL. Association for Computational Linguistics, 2022, pp. 2300–2344. find our card
  43. P. Liang, R. Bommasani, T. Lee, D. Tsipras, D. Soylu, M. Yasunaga, Y. Zhang, D. Narayanan, Y. Wu, A. Kumar et al., “Holistic evaluation of language models,” Trans. Mach. Learn. Res., vol. 2023, 2023. find
  44. S. Zhang, S. Roller, N. Goyal, M. Artetxe, M. Chen, S. Chen, C. Dewan, M. Diab, X. Li, X. V. Lin et al., “Opt: Open pre-trained transformer language models,” ArXiv preprint, vol. abs/2205.01068, 2022. find our card
  45. L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. L. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, J. Schulman, J. Hilton, F. Kelton, L. Miller, M. Simens, A. Askell, P. Welinder, P. F. Christiano, J. Leike, and R. Lowe, “Training language models to follow instructions with human feedback… find our card
  46. N. Lambert, L. Castricato, L. von Werra, and A. Havrilla, “Illustrating reinforcement learning from human feedback (rlhf),” HuggingFace Blog, vol. 2022, 2022. find
  47. Y. Bai, A. Jones, K. Ndousse, A. Askell, A. Chen, N. DasSarma, D. Drain, S. Fort, D. Ganguli, T. Henighan et al., “Training a helpful and harmless assistant with reinforcement learning from human feedback,” ArXiv preprint, vol. abs/2204.05862, 2022. find
  48. L. Gao, J. Schulman, and J. Hilton, “Scaling laws for reward model overoptimization,” in Proc. of ICML, vol. 202. PMLR, 2023, pp. 10 835–10 866. find our card
  49. D. Hendrycks, C. Burns, S. Basart, A. Zou, M. Mazeika, D. Song, and J. Steinhardt, “Measuring massive multitask language understanding,” in Proc. of ICLR. OpenReview.net, 2021. find our card
  50. D. Xu, W. Chen, W. Peng, C. Zhang, T. Xu, X. Zhao, X. Wu, Y. Zheng, Y. Wang, and E. Chen, “Large language models for generative information extraction: a survey,” Frontiers Comput. Sci., vol. 18, no. 6, p. 186357, 2024. find
  51. Z. Li, Y. Zeng, Y. Zuo, W. Ren, W. Liu, M. Su, Y. Guo, Y. Liu, X. Li, Z. Hu, L. Bai, W. Li, Y. Liu, P. Yang, X. Jin, J. Guo, and X. Cheng, “Knowcoder: Coding structured knowledge into llms for universal information extraction,” in Proc. of ACL. Association for Computational Linguistics, 2024, pp. 87… find
  52. É. Simon, H. Olsen, H. You, S. Touileb, L. Øvrelid, and E. Velldal, “Generative approaches to event extraction: Survey and outlook,” in Proc. of FuturED. Association for Computational Linguistics, 2024, pp. 73–86. find
  53. Z. Meng, T. Liu, H. Zhang, K. Feng, and P. Zhao, “CEAN: Contrastive event aggregation network with LLM-based augmentation for event extraction,” in Proc. of EACL. Association for Computational Linguistics, 2024, pp. 321–333. find
  54. Y. Zhu, X. Wang, J. Chen, S. Qiao, Y. Ou, Y. Yao, S. Deng, H. Chen, and N. Zhang, “Llms for knowledge graph construction and reasoning: recent capabilities and future opportunities,” World Wide Web (WWW), vol. 27, no. 5, p. 58, 2024. find
  55. P. Schneider, M. Klettner, K. Jokinen, E. Simperl, and F. Matthes, “Evaluating large language models in semantic parsing for conversational question answering over knowledge graphs,” in Proc. of ICAART. SCITEPRESS, 2024, pp. 807–814. find
  56. E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen, “Lora: Low-rank adaptation of large language models,” in Proc. of ICLR. OpenReview.net, 2022. find our card
  57. S. Yang, L. Cui, R. Ning, D. Wu, and Y. Zhang, “Challenges to open-domain constituency parsing,” in Findings of ACL: ACL 2022. Association for Computational Linguistics, 2022, pp. 112–127. find
  58. Y. Tian, Y. Song, F. Xia, and T. Zhang, “Improving constituency parsing with span attention,” in Findings of ACL: EMNLP 2020. Association for Computational Linguistics, 2020, pp. 1691–1703. find
  59. K. Yang and J. Deng, “Strongly incremental constituency parsing with graph neural networks,” in Proc. of NeurIPS, 2020, pp. 21 687–21 698. find
  60. A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, “Attention is all you need,” in Proc. of NeurIPS, 2017, pp. 5998–6008. find our card
  61. S. Cao, N. Kitaev, and D. Klein, “Unsupervised parsing via constituency tests,” in Proc. of EMNLP. Association for Computational Linguistics, 2020, pp. 4798–4808. find
  62. R. OpenAI, “Gpt-4 technical report,” arXiv, pp. 2303–08 774, 2023. find our card
  63. W.-L. Chiang, Z. Li, Z. Lin, Y. Sheng, Z. Wu, H. Zhang, L. Zheng, S. Zhuang, Y. Zhuang, J. E. Gonzalez, I. Stoica, and E. P. Xing, “Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality,” LMSYS blog, vol. 2023, 2023. find
  64. L. Cui, S. Yang, and Y. Zhang, “Investigating non-local features for neural constituency parsing,” in Proc. of ACL. Association for Computational Linguistics, 2022, pp. 2065–2075. find
  65. C. Xu, D. Guo, N. Duan, and J. McAuley, “Baize: An open-source chat model with parameter-efficient tuning on self-chat data,” in Proc. of EMNLP. Association for Computational Linguistics, 2023, pp. 6268– 6278. find our card
  66. J. Wei, M. Bosma, V. Y. Zhao, K. Guu, A. W. Yu, B. Lester, N. Du, A. M. Dai, and Q. V. Le, “Finetuned language models are zero-shot learners,” in Proc. of ICLR. OpenReview.net, 2022. find
  67. Y. Ge, W. Hua, K. Mei, J. Ji, J. Tan, S. Xu, Z. Li, and Y. Zhang, “Openagi: When LLM meets domain experts,” in Proc. of NeurIPS, 2023, pp. 5539–5568. find
  68. C. Zhou, P. Liu, P. Xu, S. Iyer, J. Sun, Y. Mao, X. Ma, A. Efrat, P. Yu, L. Yu, S. Zhang, G. Ghosh, M. Lewis, L. Zettlemoyer, and O. Levy, “LIMA: less is more for alignment,” in Proc. of NeurIPS, 2023, pp. 55 006–55 021. find
  69. S. Gururangan, A. Marasovic, S. Swayamdipta, K. Lo, I. Beltagy, D. Downey, and N. A. Smith, “Don’t stop pretraining: Adapt language models to domains and tasks,” in Proc. of ACL. Association for Computational Linguistics, 2020, pp. 8342–8360. find
  70. W. Lu, R. K. Luu, and M. J. Buehler, “Fine-tuning large language models for domain adaptation: Exploration of training strategies, scaling, model merging and synergistic capabilities,” npj Computational Materials, vol. 11, no. 1, p. 84, 2025. find
  71. F. C. N. Pereira and D. H. D. Warren, “Parsing as deduction,” in Proc. of ACL. Association for Computational Linguistics, 1983, pp. 137–144. find
  72. J. Maynez, S. Narayan, B. Bohnet, and R. McDonald, “On faithfulness and factuality in abstractive summarization,” in Proc. of ACL. Association for Computational Linguistics, 2020, pp. 1906–1919. find
  73. Y. Bang, S. Cahyawijaya, N. Lee, W. Dai, D. Su, B. Wilie, H. Lovenia, Z. Ji, T. Yu, W. Chung, Q. V. Do, Y. Xu, and P. Fung, “A multitask, multilingual, multimodal evaluation of ChatGPT on reasoning, hallucination, and interactivity,” in Proc. of IJCNLP. Association for Computational Linguistics, 202… find our card
  74. A. Holtzman, J. Buys, L. Du, M. Forbes, and Y. Choi, “The curious case of neural text degeneration,” in Proc. of ICLR. OpenReview.net, 2020. find
  75. N. Kornell, M. J. Hays, and R. A. Bjork, “Unsuccessful retrieval attempts enhance subsequent learning.” Journal of experimental psychology. Learning, memory, and cognition, vol. 35 (4), pp. 989–98, 2009. find
  76. J. Metcalfe, “Learning from errors,” Annual review of psychology, vol. 68, p. 465—489, 2017. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/bai-2023-constituency-parsing-using-llms/