Elamur Research⌕
☰
home / papers / 2023 / COLLIE: Systematic Construction of Constrained Text Generation Tasks

COLLIE: Systematic Construction of Constrained Text Generation Tasks

Paper · 2023 · Shunyu Yao, Howard Chen, Austin W. Hanjie et al.

Summary

Existing benchmarks for constrained text generation usually test one fixed kind of constraint, such as including given words, and models like GPT-4 now pass them easily. COLLIE is a grammar for building constraints from count and position primitives. These apply at several text levels (character, word, sentence, paragraph, passage) and can be combined with logical and/or. A user writes a constraint structure with the target values left open. The framework then finds matching values and reference texts in real corpora, turns the constraint into a natural-language instruction, and checks model outputs automatically. With it, the authors built COLLIE-v1 from Wikipedia, CC-News and Project Gutenberg, covering 13 constraint structures. They tested five instruction-tuned LLMs zero-shot. Even the strongest model, GPT-4, met the constraints only about half the time on average. Difficulty depends mainly on the constraint's logic rather than the data source. Constraints on the first unit are easy, while those on the last or arbitrary positions are hard. Exact counts and logical combinations add further difficulty. The pipeline is lightweight and extensible, so the benchmark can grow harder as models improve.

Key points

  • Dataset size and sources: The abstract gives COLLIE-v1 as 2,080 instances across 13 constraint structures, drawn from Wikipedia, CC-News and Project Gutenberg. The conclusion instead cites 1,132 constraints, so check the released data for the exact figure.
  • Pipeline: COLLIE has four stages: constraint specification via grammar, extraction from corpora, instruction rendering, and automatic evaluation. Code and data are at collie-benchmark.github.io.
  • Overall results: Five models were evaluated zero-shot, including GPT-4 and PaLM. GPT-4 performed best but averaged only 50.9% constraint satisfaction.
  • Position effect: Starting a sentence with a given word gave GPT-4 a 100% success rate. Ending a sentence with a given word gave only about 40–60%. Results were consistent across the three corpora.
  • Feedback rounds: COLLIE can generate automatic natural-language feedback on a model's output for further attempts. This raised GPT-4 by about 20% after the second round, but performance leveled off near 66%, roughly matching pass@5. Tasks word02, sent01 and sent02 stayed difficult.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Princeton UniversityPrinceton · United States90 works · 39,109 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.Text generation · 202 works

Cite

Cite as Shunyu Yao, Howard Chen, Austin W. Hanjie, Runzhe Yang and Karthik Narasimhan. 2023. COLLIE: Systematic Construction of Constrained Text Generation Tasks. arXiv:2307.08689.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2307.08689 · arXiv 2307.08689

@misc{yao2023collie,
  title = {{COLLIE: Systematic Construction of Constrained Text Generation Tasks}},
  author = {Shunyu Yao, Howard Chen, Austin W. Hanjie, Runzhe Yang and Karthik Narasimhan},
  year = {2023},
  doi = {10.48550/arxiv.2307.08689},
  eprint = {2307.08689},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2307.08689}
}

References

38 references, 26 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. R. Anil, A. M. Dai, O. Firat, M. Johnson, D. Lepikhin, A. Passos, S. Shakeri, E. Taropa, P. Bailey, Z. Chen, et al. Palm 2 technical report. arXiv preprint arXiv:2305.10403, 2023.
  2. S. Bird. Nltk: the natural language toolkit. In Proceedings of the COLING/ACL 2006 Interactive Presentation Sessions, pages 69–72, 2006.
  3. J. Brooke, A. Hammond, and G. Hirst. Gutentag: an nlp-driven tool for digital humanities research in the project gutenberg corpus. In Proceedings of the Fourth Workshop on Computational Linguistics for Literature, pages 42–47, 2015. find
  4. T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. M. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Ber… — Language Models are Few-Shot Learners
  5. H. Chen, H. Li, D. Chen, and K. Narasimhan. Controllable text generation with language constraints. In preprint, 2022. URL https://arxiv.org/abs/2212.10466. ↗
  6. M. Chen, J. Tworek, H. Jun, Q. Yuan, H. Ponde de Oliveira Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman, A. Ray, R. Puri, G. Krueger, M. Petrov, H. Khlaaf, G. Sastry, P. Mishkin, B. Chan, S. Gray, N. Ryder, M. Pavlov, A. Power, L. Kaiser, M. Bavarian, C. Winter, P. Tillet, F. Petros… — Evaluating Large Language Models Trained on Code
  7. M. Chevalier-Boisvert, D. Bahdanau, S. Lahlou, L. Willems, C. Saharia, T. H. Nguyen, and Y. Bengio. Babyai: A platform to study the sample efficiency of grounded language learning. In International Conference on Learning Representations (ICLR), 2019. URL https://arxiv.org/abs/1810.08272. ↗
  8. W.-L. Chiang, Z. Li, Z. Lin, Y. Sheng, Z. Wu, H. Zhang, L. Zheng, S. Zhuang, Y. Zhuang, J. E. Gonzalez, I. Stoica, and E. P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023. URL https://lmsys.org/blog/ 2023-03-30-vicuna/. ↗
  9. S. Dathathri, A. Madotto, J. Lan, J. Hung, E. Frank, P. Molino, J. Yosinski, and R. Liu. Plug and play language models: A simple approach to controlled text generation. In International Conference on Learning Representations (ICLR), 2020. URL https://openreview.net/ pdf?id=H1edEyBKDS. ↗
  10. G. Dinu, P. Mathur, M. Federico, and Y. Al-Onaizan. Training neural machine translation to apply terminology constraints. In Association for Computational Linguistics (ACL), 2019.
  11. W. Foundation. Wikimedia downloads, 2022. URL https://dumps.wikimedia.org. ↗
  12. F. Hamborg, N. Meuschke, C. Breitinger, and B. Gipp. news-please: A generic news crawler and extractor. In Proceedings of the 15th International Symposium of Information Science, pages 218–223, March 2017. doi: 10.5281/zenodo.4120316. — [doi:10.5281/zenodo.4120316](https://doi.org/10.5281/zenodo.4120316) find
  13. E. Hasler, A. de Gispert, G. Iglesias, and B. Byrne. Neural machine translation decoding with terminology constraints. In North American Association for Computational Linguistics (NAACL), 2018.
  14. D. Hendrycks, C. Burns, S. Basart, A. Zou, M. Mazeika, D. Song, and J. Steinhardt. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300, 2020.
  15. C. Hokamp and Q. Liu. Lexically constrained decoding for sequence generation using grid beam search. In Association for Computational Linguistics (ACL), 2017.
  16. J. E. Hu, H. Khayrallah, R. Culkin, P. Xia, T. Chen, M. Post, and B. Van Durme. Improved lexically constrained decoding for translation and monolingual rewriting. In North American Association for Computational Linguistics (NAACL), 2019.
  17. Z. Hu, Z. Yang, X. Liang, R. Salakhutdinov, and E. P. Xing. Toward controlled generation of text. In International Conference on Machine Learning (ICML), 2017.
  18. J. Johnson, B. Hariharan, L. van der Maaten, L. Fei-Fei, C. L. Zitnick, and R. Girshick. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. In Conference on computer vision and pattern recognition (CVPR), 2017. URL https://ieeexplore.ieee.org/document/8099698. ↗
  19. N. S. Keskar, B. McCann, L. R. Varshney, C. Xiong, and R. Socher. CTRL: A conditional transformer language model for controllable generation. In preprint, 2019. URL https://arxiv.org/abs/1909.05858. ↗
  20. B. Krause, A. D. Gotmare, B. McCann, N. S. Keskar, S. Joty, R. Socher, and N. F. Rajani. GeDi: Generative discriminator guided sequence generation. In Findings of the Empirical Methods in Natural Language Processing (EMNLP Findings), 2021. URL https://aclanthology.org/2021.findings-emnlp.424. ↗
  21. Q. Lhoest, A. Villanova del Moral, Y. Jernite, A. Thakur, P. von Platen, S. Patil, J. Chaumond, M. Drame, J. Plu, L. Tunstall, J. Davison, M. Šaško, G. Chhablani, B. Malik, S. Brandeis, T. Le Scao, V. Sanh, C. Xu, N. Patry, A. McMillan-Major, P. Schmid, S. Gugger, C. Delangue, T. Matussière, L. Debu… — Datasets: A Community Library for Natural Language Processing
  22. B. Z. Li, J. Yu, M. Khabsa, L. Zettlemoyer, A. Halevy, and J. Andreas. Quantifying adaptability in pre-trained language models with 500 tasks. In North American Association for Computational Linguistics (NAACL), 2022. URL https://arxiv.org/abs/2112.03204. ↗
  23. X. L. Li, J. Thickstun, I. Gulrajani, P. Liang, and T. B. Hashimoto. Diffusion-lm improves controllable text generation. In Advances in Neural Information Processing Systems (NeurIPS), 2022. URL https://openreview.net/forum?id=3s9IrEsjLyk. ↗
  24. P. Liang, R. Bommasani, T. Lee, D. Tsipras, D. Soylu, M. Yasunaga, Y. Zhang, D. Narayanan, Y. Wu, A. Kumar, et al. Holistic evaluation of language models. arXiv preprint arXiv:2211.09110, 2022. find
  25. B. Y. Lin, W. Zhou, M. Shen, P. Zhou, C. Bhagavatula, Y. Choi, and X. Ren. Commongen: A constrained text generation challenge for generative commonsense reasoning. In Findings of the Empirical Methods in Natural Language Processing (EMNLP Findings), 2020. find
  26. X. Lu, P. West, R. Zellers, R. L. Bras, C. Bhagavatula, and Y. Choi. Neurologic decoding: (un)supervised neural text generation with predicate logic constraints. In North American Association for Computational Linguistics (NAACL), 2021. URL https://aclanthology.org/2021.naacl-main.339.pdf. ↗
  27. X. Lu, S. Welleck, P. West, L. Jiang, J. Kasai, D. Khashabi, R. L. Bras, L. Qin, Y. Yu, R. Zellers, N. A. Smith, and Y. Choi. Neurologic a*esque decoding: Constrained text generation with lookahead heuristics. In North American Association for Computational Linguistics (NAACL), 2022. URL https://acl…
  28. OpenAI. Gpt-4 technical report. In preprint, 2023. URL https://arxiv.org/abs/ 2303.08774. ↗
  29. OpenAI. Gpt-4 technical report, 2023.
  30. L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. L. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, J. Schulman, J. Hilton, F. Kelton, L. Miller, M. Simens, A. Askell, P. Welinder, P. Christiano, J. Leike, and R. Lowe. Training language models to follow instructions with human feedback. In…
  31. L. Ruis, J. Andreas, M. Baroni, D. Bouchacourt, and B. M. Lake. A benchmark for systematic generalization in grounded language understanding. In Advances in Neural Information Processing Systems (NeurIPS), 2020. URL https://arxiv.org/abs/2003.05161. — [arXiv:2003.05161](https://arxiv.org/abs/2003.05161) ↗
  32. T. Schick, J. Dwivedi-Yu, R. Dessì, R. Raileanu, M. Lomeli, L. Zettlemoyer, N. Cancedda, and T. Scialom. Toolformer: Language models can teach themselves to use tools. In preprint, 2023. URL https://arxiv.org/abs/2302.04761. ↗
  33. A. Srivastava, A. Rastogi, A. Rao, A. A. M. Shoeb, A. Abid, A. Fisch, A. R. Brown, A. Santoro, A. Gupta, A. Garriga-Alonso, et al. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. arXiv preprint arXiv:2206.04615, 2022.
  34. R. Taori, I. Gulrajani, T. Zhang, Y. Dubois, X. Li, C. Guestrin, P. Liang, and T. B. Hashimoto. Stanford alpaca: An instruction-following llama model. https://github.com/tatsu-lab/stanford_alpaca, 2023. ↗
  35. A. Wang, A. Singh, J. Michael, F. Hill, O. Levy, and S. R. Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. In Empirical Methods in Natural Language Processing (EMNLP), 2018. URL https://openreview.net/pdf? id=rJ4km2R5t7. ↗
  36. Y. Wang, S. Mishra, P. Alipoormolabashi, Y. Kordi, A. Mirzaei, A. Naik, A. Ashok, A. S. Dhanasekaran, A. Arunkumar, D. Stap, et al. Super-naturalinstructions: Generalization via declarative instructions on 1600+ nlp tasks. In Proceedings of the 2022 Conference on Empirical Methods in Natural Languag… find
  37. J. Weston, A. Bordes, S. Chopra, A. M. Rush, B. Van Merriënboer, A. Joulin, and T. Mikolov. Towards ai-complete question answering: A set of prerequisite toy tasks. arXiv preprint arXiv:1502.05698, 2015.
  38. S. Yao, J. Zhao, D. Yu, N. Du, I. Shafran, K. Narasimhan, and Y. Cao. React: Synergizing reasoning and acting in language models. In International Conference on Learning Representations (ICLR), 2022. URL https://arxiv.org/abs/2210.03629. Code Availability The code, data, and additional details are a… ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/yao-2023-collie-systematic-construction-constrained-text/