Elamur Research⌕
☰
home / papers / 2023 / Bridging the Gap: A Survey on Integrating (Human) Feedback for Natural Language Generation

Bridging the Gap: A Survey on Integrating (Human) Feedback for Natural Language Generation

Summary

Large language models trained by maximum likelihood on web-scraped text can produce toxic, wrong or unhelpful output. Common automatic metrics, often based on surface overlap with references, frequently miss these failures. This survey (published in TACL, 2023) reviews how human judgments can steer natural language generation closer to what people actually want. The authors first define feedback as a function that maps an input and one or more outputs to some signal. They then propose a taxonomy with four axes: format, objective, use and modeling. The survey covers two families of methods, applied during training or at decoding time:

  • Direct use: optimizing on the feedback itself, for example reinforcement learning with human rewards.
  • Learned proxies: training feedback models that stand in for human judges.

The paper also reviews datasets, problems in collecting annotations, and the new area of AI-generated feedback. Its main conclusion is that the gains come mostly from the feedback signal, not from the particular algorithm used. The authors also find that richer feedback, such as natural-language critiques, remains underused.

Key points

  • Feedback formats: numerical scores (e.g., 1–5 stars for translations, binary good/bad), rankings (used in InstructGPT-style instruction tuning and summarization work), natural-language comments, multi-aspect ratings, post-edits and MQM error annotations.
  • Data quantity: Gao et al. (2022) found that training preference models on fewer than 1,000 comparisons gave only chance-level improvements.
  • Annotation concerns: annotators disagree, especially on toxicity and ethical judgments. Judgments are affected by anchoring and positivity biases. The survey also cites a TIME report of annotators paid about $2/hour or less to label toxic content for RLHF.
  • AI feedback comes in two forms:
    • Self-feedback: the same LLM critiques and revises its own output, as in Constitutional-AI-style revision and few-shot refinement with GPT-3.5/GPT-4.
    • External feedback: a separate, possibly small model critiques the task model.
  • Resource: a companion website is available at feedback-gap-survey.info.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cited by

40 papers of our library cite it (of the 140673 whose bibliographies we read); 3 of them have a page here, 37 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.1162/tacl_a_00626 · arXiv 2305.00955

@article{fernandes2023bridging,
  title = {{Bridging the Gap: A Survey on Integrating (Human) Feedback for Natural Language Generation}},
  author = {Fernandes, Patrick and Madaan, Aman and Liu, Emmy and Farinhas, António and Martins, Pedro Henrique and Bertsch, Amanda and Souza, José G. C. de and Zhou, Shuyan and Wu, Tongshuang and Neubig, Graham and Martins, André F. T.},
  journal = {Transactions of the Association for Computational Linguistics},
  year = {2023},
  doi = {10.1162/tacl\_a\_00626},
  eprint = {2305.00955},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.1162/tacl_a_00626}
}

References

160 references, 33 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Afra Feyza Akyürek, Ekin Akyürek, Aman Madaan, Ashwin Kalyan, Peter Clark, Derry Wijaya, and Niket Tandon. 2023. Rl4f: Generating natural language feedback with reinforcement learning for repairing model outputs. https://doi.org/10.18653/v1/2023.acl-long.427 ↗
  2. Saleema Amershi, Maya Cakmak, William Bradley Knox, and Todd Kulesza. 2014. Power to the people: The role of humans in interactive machine learning. AI Magazine, 35(4):105–120. https://doi.org/10.1609/aimag.v35i4.2513 ↗
  3. Dario Amodei, Chris Olah, Jacob Steinhardt, Paul F. Christiano, John Schulman, and Dan Mané. 2016. Concrete problems in AI safety. CoRR, abs/1606.06565. https://doi.org/10.48550/arXiv.1606.06565 ↗ our card
  4. Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, Carol Chen, Catherine Olsson, Christopher Olah, Danny Hernandez, Dawn Drain, Deep Ganguli, Dustin Li, Eli Tran-Johnson, Ethan Perez, Jamie Kerr, Jared Mueller, Jeffrey Ladish, Joshua Landau, Kamal Ndousse, Kamile Lukosuite, Liane Lovitt, M… find our card
  5. Chantal Amrhein and Rico Sennrich. 2022. Identifying weaknesses in machine translation metrics through minimum Bayes risk decoding: A case study for COMET. In Proceedings of the 2nd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 12th International Joi… find
  6. Kushal Arora, Kurt Shuster, Sainbayar Sukhbaatar, and Jason Weston. 2022. Director: Generator-classifiers for supervised language modeling. In Proceedings of the 2nd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 12th International Joint Conference on… find
  7. Amanda Askell, Yuntao Bai, Anna Chen, Dawn Drain, Deep Ganguli, Tom Henighan, Andy Jones, Nicholas Joseph, Ben Mann, Nova DasSarma, et al. 2021. A general language assistant as a laboratory for alignment. arXiv preprint arXiv:2112.00861. find
  8. Karl Johan Åström and Richard M. Murray. 2021. Feedback systems: An introduction for scientists and engineers. Princeton University Press. find
  9. Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, Nicholas Joseph, Saurav Kadavath, Jackson Kernion, Tom Conerly, Sheer El-Showk, Nelson Elhage, Zac Hatfield-Dodds, Danny Hernandez, Tristan Hume, Scott Johnston, Sh… find
  10. Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Gagan Bansal, Tongshuang Wu, Joyce Zhou, Raymond Fok, Besmira Nushi, Ece Kamar, Marco Tulio Ribeiro, and Daniel Weld. 2021. Does the whole exceed its parts? The effect of AI explanations on complementary team pe… — [doi:10.1145/3411764.3445717](https://doi.org/10.1145/3411764.3445717) find
  11. Emily M. Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. 2021. On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, FAccT ’21, pages 610–623, New York, NY, USA. Associa… — [doi:10.1145/3442188.3445922](https://doi.org/10.1145/3442188.3445922) find
  12. Pushpak Bhattacharyya, Rajen Chatterjee, Markus Freitag, Diptesh Kanojia, Matteo Negri, and Marco Turchi. 2022. Findings of the WMT 2022 shared task on automatic post-editing. In Proceedings of the Seventh Conference on Machine Translation (WMT), pages 109–117, Abu Dhabi, United Arab Emirates (Hybri… find
  13. Florian Böhm, Yang Gao, Christian M. Meyer, Ori Shapira, Ido Dagan, and Iryna Gurevych. 2019. Better rewards yield better summaries: Learning to summarise without references. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Co… find
  14. Tolga Bolukbasi, Kai-Wei Chang, James Y. Zou, Venkatesh Saligrama, and Adam T. Kalai. 2016. Man is to computer programmer as woman is to homemaker? Debiasing word embeddings. Advances in neural information processing systems, 29. find our card
  15. Rishi Bommasani, Drew A. Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S. Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, Erik Brynjolfsson, S. Buch, Dallas Card, Rodrigo Castellon, Niladri S. Chatterji, Annie S. Chen, Kathleen A. Creel, Jared Davis, Dora Demsz… find our card
  16. Nick Bostrom. 2014. Superintelligence: Paths, Dangers, Strategies, 1st edition. Oxford University Press, Inc., USA. find
  17. Samuel R. Bowman, Jeeyoon Hyun, Ethan Perez, Edwin Chen, Craig Pettit, Scott Heiner, Kamile Lukošūtė, Amanda Askell, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, Christopher Olah, Daniela Amodei, Dario Amodei, Dawn Drain, Dustin Li, Eli Tran-Johnson, Jackson Kernion, J… find
  18. Stephen Casper, Xander Davies, Claudia Shi, Thomas Krendl Gilbert, Jérémy Scheurer, Javier Rando, Rachel Freedman, Tomasz Korbak, David Lindner, Pedro Freire, Tony Wang, Samuel Marks, Charbel-Raphaël Segerie, Micah Carroll, Andi Peng, Phillip Christoffersen, Mehul Damani, Stewart Slocum, Usman An… find
  19. Arun Chaganty, Stephen Mussmann, and Percy Liang. 2018. The price of debiasing automatic metrics in natural language evaluation. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 643–653, Melbourne, Australia. Association for Co… find
  20. Xinyun Chen, Maxwell Lin, Nathanael Schärli, and Denny Zhou. 2023. Teaching large language models to self-debug. arXiv preprint arXiv:2304.05128. find
  21. Elizabeth Clark, Tal August, Sofia Serrano, Nikita Haduong, Suchin Gururangan, and Noah A. Smith. 2021. All that’s ‘human’ is not gold: Evaluating human evaluation of generated text. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International… find
  22. Jacob Cohen. 1960. A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20:37–46. https://doi.org/10.1177/001316446002000104 — [doi:10.1177/001316446002000104](https://doi.org/10.1177/001316446002000104) ↗
  23. Gonçalo M. Correia and André F. T. Martins. 2019. A simple and effective approach to automatic post-editing with transfer learning. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3050–3056, Florence, Italy. Association for Computational Linguistic… find
  24. Michael Denkowski, Chris Dyer, and Alon Lavie. 2014. Learning from post-editing: Online model adaptation for statistical machine translation. In Proceedings of the 14th Conference of the European Chapter of the Association for Computational Linguistics, pages 395–404, Gothenburg, Sweden. Association… find
  25. Yi Ding, Jacob You, Tonja-Katrin Machulla, Jennifer Jacobs, Pradeep Sen, and Tobias Höllerer. 2022. Impact of annotator demographics on sentiment dataset labeling. Proceedings of the ACM on Human Computer Interaction, 6(CSCW2). https://doi.org/10.1145/3555632 — [doi:10.1145/3555632](https://doi.org/10.1145/3555632) ↗
  26. Jesse Dodge, Maarten Sap, Ana Marasović, William Agnew, Gabriel Ilharco, Dirk Groeneveld, Margaret Mitchell, and Matt Gardner. 2021. Documenting large webtext corpora: A case study on the colossal clean crawled corpus. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language P… find
  27. Domenic Donato, Lei Yu, Wang Ling, and Chris Dyer. 2022. Mad for robust reinforcement learning in machine translation. https://doi.org/10.48550/arXiv.2207.08583 — [doi:10.48550/arxiv.2207.08583](https://doi.org/10.48550/arxiv.2207.08583) ↗
  28. Ahmed Elgohary, Christopher Meek, Matthew Richardson, Adam Fourney, Gonzalo Ramos, and Ahmed Hassan Awadallah. 2021. Nl-edit: Correcting semantic parse errors through natural language interaction. arXiv preprint arXiv: 2103.14540. https://doi.org/10.18653/v1/2021.naacl-main.444 ↗
  29. Kawin Ethayarajh, Heidi Zhang, Yizhong Wang, and Dan Jurafsky. 2023. Stanford human preferences dataset. find
  30. Patrick Fernandes, António Farinhas, Ricardo Rei, José De Souza, Perez Ogayo, Graham Neubig, and Andre Martins. 2022. Quality-aware decoding for neural machine translation. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human L… find
  31. J. L. Fleiss. 1971. Measuring nominal scale agreement among many raters. Psychological Bulletin, 76:378–382. https://doi.org/10.1037/h0031619 — [doi:10.1037/h0031619](https://doi.org/10.1037/h0031619) ↗
  32. Markus Freitag, George Foster, David Grangier, Viresh Ratnakar, Qijun Tan, and Wolfgang Macherey. 2021. Experts, errors, and context: A large-scale study of human evaluation for machine translation. Transactions of the Association for Computational Linguistics, 9:1460–1474. https://doi.org/10.1162 /… find our card
  33. Markus Freitag, David Grangier, Qijun Tan, and Bowen Liang. 2022a. High quality rather than high model probability: Minimum Bayes risk decoding with neural metrics. Transactions of the Association for Computational Linguistics, 10:811–825. https://doi.org/10.1162/tacl a 00491 ↗
  34. Markus Freitag, Ricardo Rei, Nitika Mathur, Chi-kiu Lo, Craig Stewart, Eleftherios Avramidis, Tom Kocmi, George Foster, Alon Lavie, and André F. T. Martins. 2022b. Results of WMT22 metrics shared task: Stop using BLEU – neural metrics are better and more robust. In Proceedings of the Seventh Confer… find our card
  35. Leo Gao, John Schulman, and Jacob Hilton. 2022. Scaling laws for reward model overoptimization. https://doi.org/10.48550/arXiv .2210.1076 ↗
  36. Yang Gao, Christian M. Meyer, and Iryna Gurevych. 2018. APRIL: Interactively learning to summarise by combining active preference learning and reinforcement learning. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4120–4130, Brussels, Belgium. Associ… find
  37. Sebastian Gehrmann, Abhik Bhattacharjee, Abinaya Mahendiran, Alex Wang, Alexandros Papangelis, Aman Madaan, Angelina McMillan-Major, Anna Shvets, Ashish Upadhyay, Bingsheng Yao, Bryan Wilie, Chandra Bhagavatula, Chaobin You, Craig Thomson, Cristina Garbacea, Dakuo Wang, Daniel Deutsch, Deyi Xiong, D… find
  38. Sebastian Gehrmann, Elizabeth Clark, and Thibault Sellam. 2023. Repairing the cracked foundation: A survey of obstacles in evaluation practices for generated text. Journal of Artificial Intelligence Research, 77:103–166. https://doi.org/10.1613/jair.1.13715 ↗
  39. Bhavya Ghai, Q. Vera Liao, Yunfeng Zhang, Rachel Bellamy, and Klaus Mueller. 2021. Explainable active learning (xal) toward AI explanations as interfaces for machine teachers. Proceedings of the ACM on Human-Computer Interaction, 4(CSCW3):1–28. https://doi.org/10.1145/3432934 — [doi:10.1145/3432934](https://doi.org/10.1145/3432934) ↗
  40. Gaurav R. Ghosal, Matthew Zurek, Daniel S. Brown, and Anca D. Dragan. 2023. The effect of modeling human rationality level on learning rewards from multiple feedback types. https://doi.org/10.1609/aaai.v37i5 .25740 — [doi:10.1609/aaai.v37i5](https://doi.org/10.1609/aaai.v37i5) ↗
  41. Dan Gillick and Yang Liu. 2010. Non-expert evaluation of summarization systems is risky. In Proceedings of the NAACL HLT 2010 Workshop on Creating Speech and Language Data with Amazon’s Mechanical Turk, pages 148–151, Los Angeles. Association for Computational Linguistics. find
  42. Amelia Glaese, Nat McAleese, Maja Tre , bacz, John Aslanides, Vlad Firoiu, Timo Ewalds, Maribeth Rauh, Laura Weidinger, Martin Chadwick, Phoebe Thacker, Lucy Campbell-Gillingham, Jonathan Uesato, Po-Sen Huang, Ramona Comanescu, Fan Yang, Abigail See, Sumanth Dathathri, Rory Greig, Charlie Chen, Doug… find our card
  43. Taisiya Glushkova, Chrysoula Zerva, Ricardo Rei, and André F. T. Martins. 2021. Uncertainty-aware machine translation evaluation. In Findings of the Association for Computational Linguistics: EMNLP 2021. Association for Computational Linguistics. https://doi.org/10.18653/v1/2021.findings-emnlp.330 ↗
  44. C. A. E. Goodhart. 1984. Problems of Monetary Management: The UK Experience. Macmillan Education UK, London. https://doi.org/10.1007/978-1-349-17295-5_4 — [doi:10.1007/978-1-349-17295-5_4](https://doi.org/10.1007/978-1-349-17295-5_4) ↗
  45. Mitchell L. Gordon, Michelle S. Lam, Joon Sung Park, Kayur Patel, Jeff Hancock, Tatsunori Hashimoto, and Michael S. Bernstein. 2022. Jury learning: Integrating dissenting voices into machine learning models. In CHI Conference on Human Factors in Computing Systems. ACM. https://doi.org/10.1145 /34911… find
  46. Yvette Graham, Timothy Baldwin, Alistair Moffat, and Justin Zobel. 2013. Continuous measurement scales in human evaluation of machine translation. In Proceedings of the 7th Linguistic Annotation Workshop and Interoperability with Discourse, pages 33–41, Sofia, Bulgaria. Association for Computational… find
  47. Lewis D. Griffin, Bennett Kleinberg, Maximilian Mozes, Kimberly T. Mai, Maria Vau, Matthew Caldwell, and Augustine Marvor-Parker. 2023. Susceptibility to influence of large language models. https://doi.org/10.48550/arXiv.2303.06074 ↗
  48. Braden Hancock, Antoine Bordes, Pierre-Emmanuel Mazare, and Jason Weston. 2019. Learning from dialogue after deployment: Feed yourself, chatbot! In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3667–3684. https://doi.org/10.18653/v1/P19 -1358 ↗
  49. Jochen Hartmann, Jasper Schwenzow, and Maximilian Witte. 2023. The political ideology of conversational AI: Converging evidence on chatgpt’s pro-environmental, left-libertarian orientation. https://doi.org/10.2139/ssrn.4316084 ↗
  50. Lynn Hasher, David Goldstein, and Thomas Toppino. 1977. Frequency and the conference of referential validity. Journal of Verbal Learning and Verbal Behavior, 16(1):107–112. https://doi.org/10.1016/S0022-5371 (77)80012-1 — [doi:10.1016/s0022-5371](https://doi.org/10.1016/s0022-5371) ↗
  51. A. F. Hayes and K. Krippendorff. 2007. Answering the call for a standard reliability measure for coding data. Communication Methods and Measures, 1:77–89. https://doi.org/10 .1080/19312450709336664 find
  52. Dan Hendrycks, Collin Burns, Steven Basart, Andrew Critch, Jerry Li, Dawn Song, and Jacob Steinhardt. 2020. Aligning AI with shared human values. arXiv preprint arXiv:2008.02275. find our card
  53. Lars Hornuf and Daniel Vrankar. 2022. Hourly wages in crowdworking: A meta-analysis. Business & Information Systems Engineering, 64(5):553–573. https://doi.org/10.1007/s12599-022-00769-5 ↗
  54. Maurice Jakesch, Advait Bhat, Daniel Buschek, Lior Zalmanson, and Mor Naaman. 2023. Co-writing with opinionated language models affects users’ views. ArXiv, abs/2302.00560. https://doi.org/10.1145/3544548 .3581196 — [doi:10.1145/3544548](https://doi.org/10.1145/3544548) ↗
  55. Natasha Jaques, Asma Ghandeharioun, Judy Hanwen Shen, Craig Ferguson, Àgata Lapedriza, Noah Jones, Shixiang Gu, and Rosalind W. Picard. 2019. Way off-policy batch deep reinforcement learning of implicit human preferences in dialog. CoRR, abs/1907. 00456. find
  56. Liwei Jiang, Jena D. Hwang, Chandra Bhagavatula, Ronan Le Bras, Jenny Liang, Jesse Dodge, Keisuke Sakaguchi, Maxwell Forbes, Jon Borchardt, Saadia Gabriel, Yulia Tsvetkov, Oren Etzioni, Maarten Sap, Regina Rini, and Yejin Choi. 2022. Can machines learn morality? The delphi experiment. https://doi.o… ↗
  57. Zachary Kenton, Tom Everitt, Laura Weidinger, Iason Gabriel, Vladimir Mikulik, and Geoffrey Irving. 2021. Alignment of language agents. CoRR, abs/2103.14659. https://doi.org/10.48550/arXiv.2103.14659 ↗
  58. W. Bradley Knox and Peter Stone. 2013. Learning non-myopically from human-generated reward. In Proceedings of the 2013 International Conference on Intelligent User Interfaces, IUI ’13, pages 191–202, New York, NY, USA. Association for Computing Machinery. https://doi.org/10.1145/2449396 .2449422 — [doi:10.1145/2449396](https://doi.org/10.1145/2449396) ↗
  59. Tomasz Korbak, Kejian Shi, Angelica Chen, Rasika Bhalerao, Christopher L. Buckley, Jason Phang, Samuel R. Bowman, and Ethan Perez. 2023. Pretraining language models with human preferences. https://doi.org/10 .48550/arXiv.2302.08582 find
  60. Julia Kreutzer, Shahram Khadivi, Evgeny Matusov, and Stefan Riezler. 2018. Can neural machine translation be improved with user feedback? In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 3 (Indus… find
  61. Anand Kulkarni, Prayag Narula, David Rolnitzky, and Nathan Kontny. 2014. Wish: Amplifying creative ability with expert crowds. In Proceedings of the AAAI Conference on Human Computation and Crowdsourcing, volume 2, pages 112–120. https://doi.org/10 .1609/hcomp.v2i1.13166 find
  62. Shankar Kumar and William Byrne. 2002. Minimum bayes-risk word alignments of bilingual texts. In Proceedings of the ACL-02 Conference on Empirical Methods in Natural Language Processing - Volume 10, EMNLP ’02, pages 140–147, USA. Association for Computational Linguistics. https://doi .org/10.3115/11… find
  63. Nathan Lambert, Lewis Tunstall, Nazneen Rajani, and Tristan Thrush. 2023. Hugging-Face h4 stack exchange preference dataset. find
  64. Florian Lehmann, Niklas Markert, Hai Dang, and Daniel Buschek. 2022. Suggestion lists vs. continuous generation: Interaction design for writing with generative models on mobile devices affect text length, wording and perceived authorship. Proceedings of Mensch und Computer 2022. https://doi.org/10.1… find
  65. Jan Leike, David Krueger, Tom Everitt, Miljan Martic, Vishal Maini, and Shane Legg. 2018. Scalable agent alignment via reward modeling: A research direction. ArXiv, abs/1811.07871. find
  66. Piyawat Lertvittayakumjorn and Francesca Toni. 2021. Explanation-based human debugging of nlp models: A survey. Transactions of the Association for Computational Linguistics, 9:1508–1528. https://doi.org/10.1162/tacl a 00440 ↗
  67. Jiwei Li, Alexander H Miller, Sumit Chopra, Marc’Aurelio Ranzato, and Jason Weston. 2017. Dialogue learning with human-in-the-loop. In International Conference on Learning Representations. https://doi.org/10 .48550/arXiv.1611.09823 find
  68. Zichao Li, Prakhar Sharma, Xing Han Lu, Jackie Cheung, and Siva Reddy. 2022. Using interactive feedback to improve the accuracy and explainability of question answering systems post-deployment. In Findings of the Association for Computational Linguistics: ACL 2022. Association for Computational Ling… find
  69. Q. Vera Liao and Kush R. Varshney. 2021. Human-centered explainable AI (xAI): From algorithms to user experiences. arXiv preprint arXiv:2110.10790. — [arXiv:2110.10790](https://arxiv.org/abs/2110.10790) find
  70. Bing Liu, Gokhan Tür, Dilek Hakkani-Tür, Pararth Shah, and Larry Heck. 2018. Dialogue learning with human teaching and feedback in end-to-end trainable task-oriented dialogue systems. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguisti… find
  71. Hao Liu, Carmelo Sferrazza, and Pieter Abbeel. 2023. Languages are rewards: Hindsight fine-tuning using human feedback. arXiv preprint arXiv:2302.02676. find
  72. Yixin Liu, Pengfei Liu, Dragomir Radev, and Graham Neubig. 2022. BRIO: Bringing order to abstractive summarization. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2890–2903, Dublin, Ireland. Association for Computational Ling… find our card
  73. Arle Lommel, Hans Uszkoreit, and Aljoscha Burchardt. 2014. Multidimensional quality metrics (MQM): A framework for declaring and describing translation quality metrics. Revista Tradumàtica: Tecnologies de la traducció. https://evistes.uab.cat/tradumatica/article/view/n12-lommel-uzskoreit -burchar… ↗
  74. Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Sean Welleck, Bodhisattwa Prasad Majumder, Shashank Gupta, Amir Yazdanbakhsh, and Peter Clark. 2023. Self-refine: Iterative refinement with self-feedback. htt… find
  75. Aman Madaan, Niket Tandon, Dheeraj Rajagopal, Peter Clark, Yiming Yang, and Eduard Hovy. 2021. Think about it! Improving defeasible reasoning by first modeling the question scenario. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 6291–6310, Online an… find
  76. Nitika Mathur, Timothy Baldwin, and Trevor Cohn. 2020. Tangled up in BLEU: Reevaluating the evaluation of automatic machine translation evaluation metrics. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4984–4997, Online. Association for Computation… find our card
  77. Nikhil Mehta and Dan Goldwasser. 2019. Improving natural language interaction with robots using advice. arXiv preprint arXiv:1905.04655. https://doi.org/10 .18653/v1/N19-1195 find
  78. Steven Moore, Huy A. Nguyen, and John Stamper. 2020. Towards crowdsourcing the identification of knowledge components. In Proceedings of the Seventh ACM Conference on Learning@ Scale, pages 245–248. https://doi.org/10.1145/3386527.3405940 — [doi:10.1145/3386527.3405940](https://doi.org/10.1145/3386527.3405940) ↗
  79. Richard Ngo. 2022. The alignment problem from a deep learning perspective. arXiv preprint arXiv:2209.00626. find
  80. Duy-Hung Nguyen, Nguyen Viet Dung Nghiem, Bao-Sinh Nguyen, Dung Tien Tien Le, Shahab Sabahi, Minh-Tien Nguyen, and Hung Le. 2022. Make the most of prior data: A solution for interactive text summarization with preference feedback. In Findings of the Association for Computational Linguistics: NAACL 2… find
  81. Yixin Nie, Xiang Zhou, and Mohit Bansal. 2020. What can we learn from collective human opinions on natural language inference data? arXiv preprint arXiv:2010.03532. https://doi.org/10.18653/v1/2020.emnlp-main .734 ↗ our card
  82. Erik Nijkamp, Bo Pang, Hiroaki Hayashi, Lifu Tu, Huan Wang, Yingbo Zhou, Silvio Savarese, and Caiming Xiong. 2022. A conversational paradigm for program synthesis. arXiv e-prints, pages arXiv–2203. find
  83. OpenAI. 2023a. Gpt-4 technical report. https://doi.org/10.48550/arXiv.2303.08774 ↗ our card
  84. OpenAI. 2023b. Model index for researchers. Accessed: 2023-05-01. find
  85. Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. 2022. Train… find
  86. Glaese, Nat McAleese, and Geoffrey Irving. 2022a. Red teaming language models with language models. https://doi.org/10 .18653/v1/2022.emnlp-main.225 find our card
  87. Santanu Pal, Sudip Kumar Naskar, Mihaela Vela, and Josef van Genabith. 2016. A neural network based approach to automatic post-editing. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 281–286, Berlin, Germany. Association for… find
  88. Ethan Perez, Sam Ringer, Kamile Lukošūtė, Karina Nguyen, Edwin Chen, Scott Heiner, Craig Pettit, Catherine Olsson, Sandipan Kundu, Saurav Kadavath, Andy Jones, Anna Chen, Ben Mann, Brian Israel, Bryan Seethor, Cameron McKinnon, Christopher Olah, Da Yan, Daniela Amodei, Dario Amodei, Dawn Drain, D… find our card
  89. Mihir Parmar, Swaroop Mishra, Mor Geva, and Chitta Baral. 2023. Don’t blame the annotator: Bias already starts in the annotation instructions. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 1779–1789, Dubrovnik, Croatia. Associat… find
  90. Tatiana Passali, Alexios Gidiotis, Efstathios Chatzikyriakidis, and Grigorios Tsoumakas. 2021. Towards human-centered summarization: A case study on financial news. In Proceedings of the First Workshop on Bridging Human–Computer Interaction and Natural Language Processing, pages 21–27, Online. Assoc… find
  91. Debjit Paul, Mete Ismayilzada, Maxime Peyrard, Beatriz Borges, Antoine Bosselut, Robert West, and Boi Faltings. 2023. Refiner: Reasoning feedback on intermediate representations. arXiv preprint arXiv:2304.01904. find our card
  92. Romain Paulus, Caiming Xiong, and Richard Socher. 2017. A deep reinforced model for abstractive summarization. CoRR, abs/1705 .04304. https://doi.org/10.48550/arXiv .1705.04304 ↗ our card
  93. Baolin Peng, Michel Galley, Pengcheng He, Hao Cheng, Yujia Xie, Yu Hu, Qiuyuan Huang, Lars Lidén, Zhou Yu, Weizhu Chen, and Jianfeng Gao. 2023. Check your facts and try again: Improving large language models with external knowledge and automated feedback. ArXiv, abs/2302.12813. find our card
  94. Ethan Perez, Saffron Huang, Francis Song, Trevor Cai, Roman Ring, John Aslanides, Amelia Neil Perry, Megha Srivastava, Deepak Kumar, and Dan Boneh. 2022. Do users write more insecure code with AI assistants? https://doi.org/10.48550/arXiv.2211.03622 — [doi:10.48550/arxiv.2211.03622](https://doi.org/10.48550/arxiv.2211.03622) ↗
  95. Maxime Peyrard, Teresa Botschen, and Iryna Gurevych. 2017. Learning to score system summaries for better content selection evaluation. In Proceedings of the Workshop on New Frontiers in Summarization, pages 74–84, Copenhagen, Denmark. Association for Computational Linguistics. https://doi.org /10.18… ↗
  96. Barbara Plank. 2022. The ‘problem’ of human label variation: On ground truth in data, modeling and evaluation. https://doi.org/10 .18653/v1/2022.emnlp-main.731 find our card
  97. Vinodkumar Prabhakaran, Aida Mostafazadeh Davani, and Mark Diaz. 2021. On releasing annotator-level labels and information in datasets. find our card
  98. In Proceedings of the Joint 15th Linguistic Annotation Workshop (LAW) and 3rd Designing Meaning Representations (DMR) Workshop, pages 133–138, Punta Cana, Dominican Republic. Association for Computational Linguistics. https://doi.org/10.18653/v1/2021.law-1.14 ↗ our card
  99. Yiwei Qin, Weizhe Yuan, Graham Neubig, and Pengfei Liu. 2022. T5score: Discriminative fine-tuning of generative evaluation metrics. https://doi.org/10.48550/arXiv .2212.05726 ↗
  100. Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea Finn. 2023. Direct preference optimization: Your language model is secretly a reward model. find
  101. Ricardo Rei, Craig Stewart, Ana C. Farinha, and Alon Lavie. 2020a. COMET: A neural framework for MT evaluation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2685–2702, Online. Association for Computational Linguistics. https://doi.org /10.1… ↗ our card
  102. Ricardo Rei, Craig Stewart, Ana C. Farinha, and Alon Lavie. 2020b. Unbabel’s participation in the WMT(20 metrics shared task). In Proceedings of the Fifth Conference on Machine Translation, pages 911–920, Online. Association for Computational Linguistics. https://doi.org/10.18653/v1/2020.emnlp -mai… — [doi:10.18653/v1/2020.emnlp](https://doi.org/10.18653/v1/2020.emnlp) ↗
  103. Machel Reid and Graham Neubig. 2022. Learning to model editing processes. arXiv preprint arXiv:2205.12374. https://doi.org/10 .18653/v1/2022.findings-emnlp.280 find
  104. Christopher Riesbeck. 1981. Failure-driven reminding for incremental learning. In IJCAI, pages 115–120. Citeseer. find
  105. Arturo Rosenblueth, Norbert Wiener, and Julian Bigelow. 1943. Behavior, purpose and teleology. Philosophy of Science, 10(1):18–24. https://doi.org/10.1086/286788 — [doi:10.1086/286788](https://doi.org/10.1086/286788) ↗
  106. Annabel Rothschild, Justin Booker, Christa Davoll, Jessica Hill, Venise Ivey, Carl DiSalvo, Ben Rydal Shapiro, and Betsy DiSalvo. 2022. Towards fair and pro-social employment of digital pieceworkers for sourcing machine learning training data. In CHI Conference on Human Factors in Computing Systems… — [doi:10.1145/3491101.3516384](https://doi.org/10.1145/3491101.3516384) find
  107. Ananya B. Sai, Akash Kumar Mohankumar, and Mitesh M. Khapra. 2022. A survey of evaluation metrics used for nlg systems. ACM Computing Surveys, 55(2). https://doi.org/10 .1145/3485766 find
  108. Shibani Santurkar, Esin Durmus, Faisal Ladhak, Cinoo Lee, Percy Liang, and Tatsunori Hashimoto. 2023. Whose opinions do language models reflect? https://doi.org/10.48550/arXiv.2303.17548 ↗ our card
  109. Maarten Sap, Swabha Swayamdipta, Laura Vianna, Xuhui Zhou, Yejin Choi, and Noah A. Smith. 2022. Annotators with attitudes: How annotator beliefs and identities bias toxic language detection. https://doi.org/10 .18653/v1/2022.naacl-main.431 find our card
  110. William Saunders, Catherine Yeh, Jeff Wu, Steven Bills, Long Ouyang, Jonathan Ward, and Jan Leike. 2022. Self-critiquing models for assisting human evaluators. https://doi.org/10.48550/arXiv.2206.05802 ↗ our card
  111. Roger C. Schank. 1983. Dynamic Memory: A Theory of Reminding and Learning in Computers and People. Cambridge University Press. find
  112. Jérémy Scheurer, Jon Ander Campos, Jun Shern Chan, Angelica Chen, Kyunghyun Cho, and Ethan Perez. 2022. Training language models with language feedback. https://doi.org/10.48550/arXiv.2204.14146 — [doi:10.48550/arxiv.2204.14146](https://doi.org/10.48550/arxiv.2204.14146) ↗
  113. Jérémy Scheurer, Jon Ander Campos, Tomasz Korbak, Jun Shern Chan, Angelica Chen, Kyunghyun Cho, and Ethan Perez. 2023. Training language models with language feedback at scale. https://doi.org/10 .48550/arXiv.2303.16755 find
  114. Timo Schick, Jane Dwivedi-Yu, Zhengbao Jiang, Fabio Petroni, Patrick Lewis, Gautier Izacard, Qingfei You, Christoforos Nalmpantis, Edouard Grave, and Sebastian Riedel. 2022. Peer: A collaborative language model. ArXiv, abs/2208.11663. find
  115. Natalie Schluter. 2017. The limits of automatic summarisation according to ROUGE. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers, pages 41–45, Valencia, Spain. Association for Computational Linguistics. https://d… find
  116. John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347. find
  117. Thibault Sellam, Dipanjan Das, and Ankur Parikh. 2020. BLEURT: Learning robust metrics for text generation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7881–7892, Online. Association for Computational Linguistics. https://doi.org/10.18653 /v1/202… find
  118. Shiqi Shen, Yong Cheng, Zhongjun He, Wei He, Hua Wu, Maosong Sun, and Yang Liu. 2016. Minimum risk training for neural machine translation. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1683–1692, Berlin, Germany. Associatio… find our card
  119. Emily Sheng, Kai-Wei Chang, Premkumar Natarajan, and Nanyun Peng. 2019. The woman worked as a babysitter: On biases in language generation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Process… find our card
  120. Victor S. Sheng, Foster Provost, and Panagiotis G. Ipeirotis. 2008. Get another label? Improving data quality and data mining using multiple, noisy labelers. In Proceedings of the 14th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD ’08, pages 614–622, New York, NY, U… — [doi:10.1145/1401890](https://doi.org/10.1145/1401890) find
  121. Weiyan Shi, Yu Li, Saurav Sahay, and Zhou Yu. 2021. Refine and imitate: Reducing repetition and inconsistency in persuasion dialogues via reinforcement learning and human demonstration. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 3478–3492, Punta Cana, Dominican R… find
  122. Noah Shinn, Beck Labash, and Ashwin Gopinath. 2023. Reflexion: An autonomous agent with dynamic memory and self-reflection. https://doi.org/10.48550/arXiv.2303.11366 ↗ our card
  123. Boaz Shmueli, Jan Fell, Soumya Ray, and Lun-Wei Ku. 2021. Beyond fair pay: Ethical implications of NLP crowdsourcing. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3758–3769, Online. Associatio… find
  124. Michel Simard, Nicola Ueffing, Pierre Isabelle, and Roland Kuhn. 2007. Rule-based translation with statistical phrase-based post-editing. In Proceedings of the Second Workshop on Statistical Machine Translation, pages 203–206. https://doi.org/10.3115/1626355 .1626383 ↗
  125. Rion Snow, Brendan O’Connor, Daniel Jurafsky, and Andrew Ng. 2008. Cheap and fast – but is it good? Evaluating non-expert annotations for natural language tasks. In Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing, pages 254–263, Honolulu, Hawaii. Association fo… find
  126. Teerachart Soratana, Yili Liu, and X Jessie Yang. 2022. Effects of payment rate and country’s income level on attitude toward a crowdsourcing task. In Proceedings of the Human Factors and Ergonomics Society Annual Meeting, volume 66, pages 2220–2224. SAGE Publications Sage CA: Los Angeles, CA. https… — [doi:10.1177/1071181322661532](https://doi.org/10.1177/1071181322661532) find
  127. Miriah Steiger, Timir J. Bharucha, Sukrit Venkatagiri, Martin J. Riedl, and Matthew Lease. 2021. The psychological well-being of content moderators: The emotional labor of commercial moderation and avenues for improving support. In Proceedings of the 2021 CHI Conference on Human Factors in Computing… — [doi:10.1145/3411764](https://doi.org/10.1145/3411764) find
  128. Nisan Stiennon, Long Ouyang, Jeff Wu, Daniel M. Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul Christiano. 2020. Learning to summarize from human feedback. https://doi.org/10.48550/arXiv.2009.01325 ↗
  129. Simone Stumpf, Vidya Rajaram, Lida Li, Margaret Burnett, Thomas Dietterich, Erin Sullivan, Russell Drummond, and Jonathan Herlocker. 2007. Toward harnessing user feedback for machine learning. In Proceedings of the 12th International Conference on Intelligent User Interfaces, pages 82–91. https://do… — [doi:10.1145/1216295.1216316](https://doi.org/10.1145/1216295.1216316) find
  130. Alane Suhr, Clara Vania, Nikita Nangia, Maarten Sap, Mark Yatskar, Samuel Bowman, and Yoav Artzi. 2021. Crowdsourcing beyond annotation: Case studies in benchmark data collection. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing: Tutorial Abstracts, pages 1–6… find
  131. Zeerak Talat, Hagen Blix, Josef Valvoda, Maya Indira Ganesh, Ryan Cotterell, and Adina Williams. 2022. On the machine learning of ethical judgments from natural language. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Langu… find
  132. Alon Talmor, Oyvind Tafjord, Peter Clark, Yoav Goldberg, and Jonathan Berant. 2020. Teaching pre-trained models to systematically reason over implicit knowledge. arXiv preprint arXiv:2006.06609, 4(6). find
  133. Niket Tandon, Aman Madaan, Peter Clark, and Yiming Yang. 2022. Learning to repair: Repairing model output errors after deployment using a dynamic memory of feedback. In Findings of the Association for Computational Linguistics: NAACL 2022, pages 339–352, Seattle, United States. Association for Compu… find
  134. Andrea L. Thomaz and Cynthia Breazeal. 2008. Teachable robots: Understanding human teaching behavior to build more effective robot learners. Artificial Intelligence, 172(6):716–737. https://doi.org/10 .1016/j.artint.2007.09.009 find
  135. Craig Thomson and Ehud Reiter. 2021. Generation challenges: Results of the accuracy evaluation shared task. In Proceedings of the 14th International Conference on Natural Language Generation, pages 240–248, Aberdeen, Scotland, UK. Association for Computational Linguistics. find
  136. Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoorv Kulshreshtha, Heng-Tze Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du, YaGuang Li, Hongrae Lee, Huaixiu Steven Zheng, Amin Ghafouri, Marcelo Menegali, Yanping Huang, Maxim Krikun, Dmitry Lepikhin, James Qin, Dehao Chen, Yuanzho… find our card
  137. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller,… find
  138. Carlos Toxtli, Siddharth Suri, and Saiph Savage. 2021. Quantifying the invisible labor in crowd work. Proceedings of the ACM on Human-Computer Interaction, 5(CSCW2):1–26. https://doi.org/10.1145/3476060 — [doi:10.1145/3476060](https://doi.org/10.1145/3476060) ↗
  139. Alexander Matt Turner, Aseem Saxena, and Prasad Tadepalli. 2022. Formalizing the problem of side effect regularization. In NeurIPS ML Safety Workshop. find
  140. Peter Vamplew, Richard Dazeley, Cameron Foale, Sally Firmin, and Jane Mummery. 2018. Human-aligned artificial intelligence is a mul-tiobjective problem. Ethics and Information Technology, 20:27–40. https://doi.org/10.1007/s10676-017-9440-6 — [doi:10.1007/s10676-017-9440-6](https://doi.org/10.1007/s10676-017-9440-6) ↗
  141. Zijie J. Wang, Dongjin Choi, Shenyu Xu, and Diyi Yang. 2021. Putting humans in the natural language processing loop: A survey. arXiv preprint arXiv:2103.04044. find
  142. Christopher J. C. H. Watkins and Peter Dayan. 1992. Q-learning. Machine Learning, 8:279–292. https://doi.org/10.1023/A:1022676722315 ↗
  143. Sean Welleck, Ximing Lu, Peter West, Faeze Brahman, Tianxiao Shen, Daniel Khashabi, and Yejin Choi. 2022. Generating sequences by learning to self-correct. arXiv preprint arXiv: 2211.00053. find our card
  144. Jason Weston, Sumit Chopra, and Antoine Bordes. 2014. Memory networks. arXiv preprint arXiv:1410.3916. find
  145. Jason E. Weston. 2016. Dialog-based language learning. Advances in Neural Information Processing Systems. https://doi.org/10 .48550/arXiv.1604.06045 find
  146. Norbert Wiener. 1948. Cybernetics; or control and communication in the animal and the machine. find
  147. Ronald J. Williams. 1992. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine Learning, 8(3–4):229–256. https://doi.org/10.1007/BF00992696 ↗
  148. Bin Wu, Chenyan Xiong, Maosong Sun, and Zhiyuan Liu. 2018. Query suggestion with feedback memory network. In Proceedings of the 2018 World Wide Web Conference, pages 1563–1571. https://doi.org/10 .1145/3178876.3186068 find
  149. Jeff Wu, Long Ouyang, Daniel M. Ziegler, Nisan Stiennon, Ryan Lowe, Jan Leike, and Paul Christiano. 2021. Recursively summarizing books with human feedback. https://doi.org/10.48550/arXiv.2109.10862 ↗ our card
  150. Jing Xu, Megan Ung, Mojtaba Komeili, Kushal Arora, Y-Lan Boureau, and Jason Weston. 2022. Learning new skills after deployment: Improving open-domain internet-driven dialogue with human feedback. find
  151. Taku Yamagata, Ryan McConville, and Raul Santos-Rodriguez. 2021. Reinforcement learning with feedback from multiple humans with diverse skills. https://doi.org/10 .48550/arXiv.2111.08596 find
  152. Kevin Yang, Nanyun Peng, Yuandong Tian, and Dan Klein. 2022. Re3: Generating longer stories with recursive reprompting and revision. In Conference on Empirical Methods in Natural Language Processing. https://doi.org/10.18653/v1/2022.emnlp-main.296 ↗
  153. Michihiro Yasunaga and Percy Liang. 2020. Graph-based, self-supervised program repair from diagnostic feedback. 37th International Conference on Machine Learning ICML 2020, PartF168147-14:10730–10739. https://doi.org/10.48550/arXiv.2005.10636 ↗
  154. Zheng Yuan, Hongyi Yuan, Chuanqi Tan, Wei Wang, Songfang Huang, and Fei Huang. 2023. Rrhf: Rank responses to align language models with human feedback without tears. arXiv preprint arXiv:2304.05302. find our card
  155. Chrysoula Zerva, Taisiya Glushkova, Ricardo Rei, and André F. T. Martins. 2022. Disentangling uncertainty in machine translation evaluation. https://doi.org/10.18653/v1/2022 .emnlp-main.591 ↗
  156. Yao Zhao, Rishabh Joshi, Tianqi Liu, Misha Khalman, Mohammad Saleh, and Peter J. Liu. 2023. Slic-hf: Sequence likelihood calibration with human feedback. arXiv preprint arXiv:2305.10425. find our card
  157. Rui Zheng, Shihan Dou, Songyang Gao, Yuan Hua, Wei Shen, Binghai Wang, Yan Liu, Senjie Jin, Qin Liu, Yuhao Zhou, Limao Xiong, Lu Chen, Zhiheng Xi, Nuo Xu, Wenbin Lai, Minghao Zhu, Cheng Chang, Zhangyue Yin, Rongxiang Weng, Wensen Cheng, Haoran Huang, Tianxiang Sun, Hang Yan, Tao Gui, Qi Zhang, Xipen… find our card
  158. Chunting Zhou, Pengfei Liu, Puxin Xu, Srini Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, Susan Zhang, Gargi Ghosh, Mike Lewis, Luke Zettlemoyer, and Omer Levy. 2023. Lima: Less is more for alignment. https://doi.org/10.48550/arXiv.2305.11206 ↗
  159. Daniel M. Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B. Brown, Alec Radford, Dario Amodei, Paul F. Christiano, and Geoffrey Irving. 2019. Fine-tuning language models from human preferences. CoRR, abs/1909.08593. https://doi.org/10.48550/arXiv .1909.08593 ↗
  160. Markus Zopf. 2018. Estimating summary quality with pairwise preferences. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1687–1696, New Orleans, Louisiana. Association for… find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/fernandes-2023-bridging-gap-survey-integrating-human/