Elamur Research⌕
☰
home / papers / 2024 / ChatQA: Surpassing GPT-4 on Conversational QA and RAG

ChatQA: Surpassing GPT-4 on Conversational QA and RAG

Paper · 2024 · Zihan Liu, Wei Ping, Rajarshi Roy et al.

Summary

The paper asks whether open-weight LLMs can match proprietary systems like GPT-4 at answering multi-turn questions grounded in supplied or retrieved documents, including tables, arithmetic, and questions the context cannot answer. NVIDIA researchers present ChatQA, a model family trained in two stages. The first stage is ordinary supervised fine-tuning on a mix of instruction and dialogue data. The second stage, "context-enhanced" tuning, adds single-turn and conversational QA datasets in which answers depend on a given passage. For retrieval, they fine-tune a single-turn dense retriever on multi-turn conversational data, either human-annotated or synthetic. This avoids the extra cost of rewriting each query with an LLM. They also release ChatRAGBench, a ten-dataset evaluation suite. ChatQA-1.0-70B, built on Llama2, slightly beats GPT-4 variants on average, and the Llama3-based version widens the margin. No OpenAI-generated training data is used. Model weights, tuning data, the benchmark and the retriever are publicly released, which gives researchers an open baseline for conversational RAG.

Key points

  • ChatRAGBench scores: ChatQA-1.0-70B averages 54.14, versus 53.90 for GPT-4-0613 and 54.03 for GPT-4-Turbo-2024-04-09. Llama3-ChatQA-1.5-70B exceeds GPT-4-Turbo by about 4.4% and leads in every benchmark category. Llama3-ChatQA-1.5-8B performs roughly on par with GPT-4.
  • Training data: Stage 1 uses 128K samples drawn from Soda, ELI5, FLAN/chain-of-thought, Self-Instruct, Unnatural Instructions, OpenAssistant, Dolly and a private crowd-sourced set. Stage 2 adds NarrativeQA, DROP, Quoref, ROPES, SQuAD, NewsQA, TAT-QA and conversational QA data.
  • Retriever: The fine-tuned single-turn retriever reaches zero-shot quality comparable to GPT-3.5-Turbo query rewriting, without the rewriting step's latency or API cost.
  • ChatRAGBench design: Five of its ten datasets involve long documents that require retrieval, and three involve tables and calculation.
  • Unanswerable questions and single-turn RAG: A small number of "unanswerable" training examples markedly reduces hallucination, leaving ChatQA-1.0-70B about 3.5% behind GPT-4-0613 on this task. On NQ, TriviaQA and HotpotQA, using the Dragon retriever, Llama3-ChatQA-1.5-8B outperforms RA-DIT (65B).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Vision and Pattern Recognition · 2,869 works

🧭 Topics 3Research subjects and languages the work is about.Language models & pretraining · 3,320 worksDialogue systems · 624 worksQuestion answering · 441 works

Cite

Cite as Zihan Liu, Wei Ping, Rajarshi Roy, Peng Xu, Chankyu Lee, Mohammad Shoeybi and Bryan Catanzaro. 2024. ChatQA: Surpassing GPT-4 on Conversational QA and RAG. arXiv:2401.10225.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2401.10225 · arXiv 2401.10225

@misc{liu2024chatqa,
  title = {{ChatQA: Surpassing GPT-4 on Conversational QA and RAG}},
  author = {Zihan Liu, Wei Ping, Rajarshi Roy, Peng Xu, Chankyu Lee, Mohammad Shoeybi and Bryan Catanzaro},
  year = {2024},
  doi = {10.48550/arxiv.2401.10225},
  eprint = {2401.10225},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2401.10225}
}

References

91 references, 53 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Adlakha, V., Dhuliawala, S., Suleman, K., de Vries, H., and Reddy, S. Topiocqa: Open-domain conversational question answering with topic switching. TACL, 2022.
  2. Albalak, A., Elazar, Y., Xie, S. M., Longpre, S., Lambert, N., Wang, X., Muennighoff, N., Hou, B., Pan, L., Jeong, H., et al. A survey on data selection for language models. arXiv preprint arXiv:2402.16827, 2024.
  3. Aliannejadi, M., Kiseleva, J., Chuklin, A., Dalton, J., and Burtsev, M. Building and evaluating open-domain dialogue corpora with clarifying questions. In EMNLP, 2021. find
  4. Anantha, R., Vakulenko, S., Tu, Z., Longpre, S., Pulman, S., and Chappidi, S. Open-domain question answering goes conversational via question rewriting. In NAACL, 2021.
  5. Anthropic. Introducing 100k context windows, 2023a. find
  6. Anthropic. Introducing Claude, 2023b. find
  7. Brabant, Q., Lecorvé, G., and Barahona, L. M. R. Coqar: Question rewriting on coqa. In LREC, 2022. find
  8. Campos, J. A., Otegi, A., Soroa, A., Deriu, J. M., Cieliebak, M., and Agirre, E. Doqa-accessing domain-specific faqs via conversational qa. In ACL, 2020. find
  9. Chen, Z., Li, S., Smiley, C., Ma, Z., Shah, S., and Wang, W. Y. Convfinqa: Exploring the chain of numerical reasoning in conversational finance question answering. In EMNLP, 2022a.
  10. Chen, Z., Zhao, J., Fang, A., Fetahu, B., Rokhlenko, O., and Malmasi, S. Reinforced question rewriting for conversational question answering. In EMNLP, 2022b. find
  11. Choi, E., He, H., Iyyer, M., Yatskar, M., Yih, W.-t., Choi, Y., Liang, P., and Zettlemoyer, L. Quac: Question answering in context. In EMNLP, 2018.
  12. Chu, Z., Chen, M., Chen, J., Wang, M., Gimpel, K., Faruqui, M., and Si, X. How to ask better questions? a large-scale multi-domain dataset for rewriting ill-formed questions. In AAAI, 2020. find
  13. Chung, H. W., Hou, L., Longpre, S., Zoph, B., Tay, Y., Fedus, W., Li, Y., Wang, X., Dehghani, M., Brahma, S., Webson, A., Gu, S. S., Dai, Z., Suzgun, M., Chen, X., Chowdhery, A., Castro-Ros, A., Pellat, M., Robinson, K., Valter, D., Narang, S., Mishra, G., Yu, A., Zhao, V., Huang, Y., Dai, A., Yu, H… find
  14. Cohere. Introducing command r+: A scalable llm built for business, 2024. find
  15. Conover, M., Hayes, M., Mathur, A., Meng, X., Xie, J., Wan, J., Shah, S., Ghodsi, A., Wendell, P., Zaharia, M., et al. Free dolly: Introducing the world’s first truly open instruction-tuned llm, 2023a. find
  16. Conover, M., Hayes, M., Mathur, A., Xie, J., Wan, J., Shah, S., Ghodsi, A., Wendell, P., Zaharia, M., and Xin, R. Free Dolly: Introducing the world’s first truly open instruction-tuned llm, 2023b. find
  17. Dai, Z., Chaganty, A. T., Zhao, V., Amini, A., Green, M., Rashid, Q., and Guu, K. Dialog inpainting: Turning documents to dialogs. In ICML, 2022. find
  18. Dasigi, P., Liu, N. F., Marasović, A., Smith, N. A., and Gardner, M. Quoref: A reading comprehension dataset with questions requiring coreferential reasoning. In EMNLP, 2019.
  19. Del Tredici, M., Barlacchi, G., Shen, X., Cheng, W., and de Gispert, A. Question rewriting for open-domain conversational qa: Best practices and limitations. In CIKM, 2021. find
  20. Deng, Y., Lei, W., Zhang, W., Lam, W., and Chua, T.-S. Pacific: Towards proactive conversational question answering over tabular and textual data in finance. In EMNLP, 2022. find
  21. Du, Z., Qian, Y., Liu, X., Ding, M., Qiu, J., Yang, Z., and Tang, J. Glm: General language model pretraining with autoregressive blank infilling. In ACL, 2022.
  22. Dua, D., Wang, Y., Dasigi, P., Stanovsky, G., Singh, S., and Gardner, M. Drop: A reading comprehension benchmark requiring discrete reasoning over paragraphs. In NAACL, 2019.
  23. Elgohary, A., Peskov, D., and Boyd-Graber, J. Can you unpack that? learning to rewrite questions-in-context. In EMNLP, 2019.
  24. Fan, A., Jernite, Y., Perez, E., Grangier, D., Weston, J., and Auli, M. Eli5: Long form question answering. In ACL, 2019.
  25. Feng, S., Wan, H., Gunasekara, C., Patel, S., Joshi, S., and Lastras, L. doc2dial: A goal-oriented document-grounded dialogue dataset. In EMNLP, 2020.
  26. Galimzhanova, E., Muntean, C. I., Nardini, F. M., Perego, R., and Rocchietti, G. Rewriting conversational utterances with instructed large language models. In IEEE/WIC International Conference on Web Intelligence and Intelligent Agent Technology, 2023. find
  27. Gao, C., Zhang, W., and Lam, W. Unigdd: A unified generative framework for goal-oriented document-grounded dialogue. In ACL, 2022. find
  28. Gao, Y., Xiong, Y., Gao, X., Jia, K., Pan, J., Bi, Y., Dai, Y., Sun, J., and Wang, H. Retrieval-augmented generation for large language models: A survey. arXiv preprint arXiv:2312.10997, 2023.
  29. Google. Introducing bard, 2023. find
  30. Guo, M., Zhang, M., Reddy, S., and Alikhani, M. Abg-coqa: Clarifying ambiguity in conversational question answering. In AKBC, 2021. find
  31. Honovich, O., Scialom, T., Levy, O., and Schick, T. Unnatural instructions: Tuning language models with (almost) no human labor. arXiv preprint arXiv:2212.09689, 2022.
  32. Huang, J., Ping, W., Xu, P., Shoeybi, M., Chang, K. C.-C., and Catanzaro, B. Raven: In-context learning with retrieval augmented encoder-decoder language models. arXiv preprint arXiv:2308.07922, 2023. — [arXiv:2308.07922](https://arxiv.org/abs/2308.07922) find
  33. Ishii, E., Xu, Y., Cahyawijaya, S., and Wilie, B. Can question rewriting help conversational question answering? In Proceedings of the Third Workshop on Insights from Negative Results in NLP, 2022. find
  34. Iyer, S., Lin, X. V., Pasunuru, R., Mihaylov, T., Simig, D., Yu, P., Shuster, K., Wang, T., Liu, Q., Koura, P. S., et al. Opt-iml: Scaling language model instruction meta learning through the lens of generalization. arXiv preprint arXiv:2212.12017, 2022.
  35. Izacard, G. and Grave, É. Leveraging passage retrieval with generative models for open domain question answering. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics, 2021.
  36. Izacard, G., Caron, M., Hosseini, L., Riedel, S., Bojanowski, P., Joulin, A., and Grave, E. Unsupervised dense information retrieval with contrastive learning. Transactions on Machine Learning Research, 2022.
  37. Izacard, G., Lewis, P., Lomeli, M., Hosseini, L., Petroni, F., Schick, T., Dwivedi-Yu, J., Joulin, A., Riedel, S., and Grave, E. Atlas: Few-shot learning with retrieval augmented language models. Journal of Machine Learning Research, 2023.
  38. Joshi, M., Choi, E., Weld, D. S., and Zettlemoyer, L. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. In ACL, 2017.
  39. Kim, H., Hessel, J., Jiang, L., Lu, X., Yu, Y., Zhou, P., Bras, R. L., Alikhani, M., Kim, G., Sap, M., et al. Soda: Million-scale dialogue distillation with social commonsense contextualization. arXiv preprint arXiv:2212.10465, 2022.
  40. Kočiskỳ, T., Schwarz, J., Blunsom, P., Dyer, C., Hermann, K. M., Melis, G., and Grefenstette, E. The narrativeqa reading comprehension challenge. TACL, 2018.
  41. Köpf, A., Kilcher, Y., von Rütte, D., Anagnostidis, S., Tam, Z.-R., Stevens, K., Barhoum, A., Duc, N. M., Stanley, O., Nagyfi, R., et al. Openassistant conversations–democratizing large language model alignment. arXiv preprint arXiv:2304.07327, 2023.
  42. Kwiatkowski, T., Palomaki, J., Redfield, O., Collins, M., Parikh, A., Alberti, C., Epstein, D., Polosukhin, I., Devlin, J., Lee, K., et al. Natural questions: A benchmark for question answering research. TACL, 2019.
  43. Köpf, A., Kilcher, Y., von Rütte, D., Anagnostidis, S., Tam, Z.-R., Stevens, K., Barhoum, A., Duc, N. M., Stanley, O., Nagyfi, R., ES, S., Suri, S., Glushkov, D., Dantuluri, A., Maguire, A., Schuhmann, C., Nguyen, H., and Mattick, A. Openassistant conversations - democratizing large language model a… — OpenAssistant Conversations — Democratizing Large Language Model Alignment
  44. Lin, K., Tafjord, O., Clark, P., and Gardner, M. Reasoning over paragraph effects in situations. In Proceedings of the 2nd Workshop on Machine Reading for Question Answering, 2019.
  45. Lin, S.-C., Asai, A., Li, M., Oguz, B., Lin, J., Mehdad, Y., Yih, W.-t., and Chen, X. How to train your dragon: Diverse augmentation towards generalizable dense retrieval. arXiv preprint arXiv:2302.07452, 2023a.
  46. Lin, X. V., Chen, X., Chen, M., Shi, W., Lomeli, M., James, R., Rodriguez, P., Kahn, J., Szilvasy, G., Lewis, M., et al. Ra-dit: Retrieval-augmented dual instruction tuning. arXiv preprint arXiv:2310.01352, 2023b.
  47. Lin, X. V., Chen, X., Chen, M., Shi, W., Lomeli, M., James, R., Rodriguez, P., Kahn, J., Szilvasy, G., Lewis, M., Zettlemoyer, L., and tau Yih, W. RA-DIT: Retrieval-augmented dual instruction tuning. In ICLR, 2024. URL https://openreview.net/forum?id=22OTbutug9. ↗
  48. Liu, N. F., Lin, K., Hewitt, J., Paranjape, A., Bevilacqua, M., Petroni, F., and Liang, P. Lost in the middle: How language models use long contexts. arXiv preprint arXiv:2307.03172, 2023.
  49. Longpre, S., Hou, L., Vu, T., Webson, A., Chung, H. W., Tay, Y., Zhou, D., Le, Q. V., Zoph, B., Wei, J., et al. The flan collection: Designing data and methods for effective instruction tuning. arXiv preprint arXiv:2301.13688, 2023.
  50. Mele, I., Muntean, C. I., Nardini, F. M., Perego, R., Tonellotto, N., and Frieder, O. Adaptive utterance rewriting for conversational search. Information Processing & Management, 2021. find
  51. Meta. Introducing meta llama 3: The most capable openly available llm to date, 2024. find
  52. Mishra, S., Khashabi, D., Baral, C., and Hajishirzi, H. Cross-task generalization via natural language crowdsourcing instructions. In ACL, 2022. find
  53. Mo, F., Mao, K., Zhu, Y., Wu, Y., Huang, K., and Nie, J.-Y. Convgqr: Generative query reformulation for conversational search. arXiv preprint arXiv:2305.15645, 2023. find
  54. Muennighoff, N., Wang, T., Sutawika, L., Roberts, A., Biderman, S., Scao, T. L., Bari, M. S., Shen, S., Yong, Z.-X., Schoelkopf, H., et al. Crosslingual generalization through multitask finetuning. arXiv preprint arXiv:2211.01786, 2022.
  55. Nakamura, K., Levy, S., Tuan, Y.-L., Chen, W., and Wang, W. Y. Hybridialogue: An information-seeking dialogue dataset grounded on tabular and textual data. In Findings of ACL, 2022. find
  56. Nguyen, T., Rosenberg, M., Song, X., Gao, J., Tiwary, S., Majumder, R., and Deng, L. Ms marco: A human generated machine reading comprehension dataset. choice, 2016.
  57. OpenAI. Introducing ChatGPT, 2022. find
  58. Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. Training language models to follow instructions with human feedback. NeurIPS, 2022.
  59. Pasupat, P. and Liang, P. Compositional semantic parsing on semi-structured tables. In ACL, 2015.
  60. Petroni, F., Piktus, A., Fan, A., Lewis, P., Yazdani, M., De Cao, N., Thorne, J., Jernite, Y., Karpukhin, V., Maillard, J., et al. Kilt: a benchmark for knowledge intensive language tasks. In NAACL, 2021.
  61. Qu, C., Yang, L., Chen, C., Qiu, M., Croft, W. B., and Iyyer, M. Open-retrieval conversational question answering. In SIGIR, 2020. find
  62. Rajpurkar, P., Zhang, J., Lopyrev, K., and Liang, P. Squad: 100,000+ questions for machine comprehension of text. In EMNLP, 2016.
  63. Rajpurkar, P., Jia, R., and Liang, P. Know what you don’t know: Unanswerable questions for squad. In ACL, 2018. — Know What You Don't Know: Unanswerable Questions for SQuAD
  64. Raposo, G., Ribeiro, R., Martins, B., and Coheur, L. Question rewriting? assessing its importance for conversational question answering. In ECIR, 2022. find
  65. Reddy, S., Chen, D., and Manning, C. D. Coqa: A conversational question answering challenge. TACL, 2019.
  66. Saeidi, M., Bartolo, M., Lewis, P., Singh, S., Rocktäschel, T., Sheldon, M., Bouchard, G., and Riedel, S. Interpretation of natural language rules in conversational machine reading. In EMNLP, 2018.
  67. Sanh, V., Webson, A., Raffel, C., Bach, S. H., Sutawika, L., Alyafeai, Z., Chaffin, A., Stiegler, A., Scao, T. L., Raja, A., et al. Multitask prompted training enables zero-shot task generalization. In ICLR, 2022.
  68. Shi, W., Min, S., Yasunaga, M., Seo, M., James, R., Lewis, M., Zettlemoyer, L., and Yih, W.-t. Replug: Retrieval-augmented black-box language models. arXiv preprint arXiv:2301.12652, 2023.
  69. Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023.
  70. Trischler, A., Wang, T., Yuan, X., Harris, J., Sordoni, A., Bachman, P., and Suleman, K. Newsqa: A machine comprehension dataset. In Proceedings of the 2nd Workshop on Representation Learning for NLP, 2017.
  71. Vakulenko, S., Longpre, S., Tu, Z., and Anantha, R. Question rewriting for conversational question answering. In WSDM, 2021a. find
  72. Vakulenko, S., Voskarides, N., Tu, Z., and Longpre, S. A comparison of question rewriting methods for conversational passage retrieval. In ECIR, 2021b. find
  73. Wang, B., Ping, W., McAfee, L., Xu, P., Li, B., Shoeybi, M., and Catanzaro, B. Instructretro: Instruction tuning post retrieval-augmented pretraining. In ICML, 2024. find
  74. Wang, L., Yang, N., Huang, X., Jiao, B., Yang, L., Jiang, D., Majumder, R., and Wei, F. Text embeddings by weakly-supervised contrastive pre-training. arXiv preprint arXiv:2212.03533, 2022a.
  75. Wang, Y., Kordi, Y., Mishra, S., Liu, A., Smith, N. A., Khashabi, D., and Hajishirzi, H. Self-instruct: Aligning language model with self generated instructions. arXiv preprint arXiv:2212.10560, 2022b. find
  76. Wang, Y., Mishra, S., Alipoormolabashi, P., Kordi, Y., Mirzaei, A., Naik, A., Ashok, A., Dhanasekaran, A. S., Arunkumar, A., Stap, D., et al. Super-naturalinstructions: Generalization via declarative instructions on 1600+ nlp tasks. In EMNLP, 2022c. find
  77. Wang, Y., Ivison, H., Dasigi, P., Hessel, J., Khot, T., Chandu, K. R., Wadden, D., MacMillan, K., Smith, N. A., Beltagy, I., et al. How far can camels go? exploring the state of instruction tuning on open resources. arXiv preprint arXiv:2306.04751, 2023.
  78. Wei, J., Bosma, M., Zhao, V. Y., Guu, K., Yu, A. W., Lester, B., Du, N., Dai, A. M., and Le, Q. V. Finetuned language models are zero-shot learners. In ICLR, 2022a.
  79. Wei, J., Wang, X., Schuurmans, D., Bosma, M., Xia, F., Chi, E., Le, Q. V., Zhou, D., et al. Chain-of-thought prompting elicits reasoning in large language models. NeurIPS, 2022b.
  80. Wu, Z., Luan, Y., Rashkin, H., Reitter, D., Hajishirzi, H., Ostendorf, M., and Tomar, G. S. Conqrr: Conversational query rewriting for retrieval with reinforcement learning. In EMNLP, 2022.
  81. Wu, Z., Parish, R., Cheng, H., Min, S., Ammanabrolu, P., Ostendorf, M., and Hajishirzi, H. Inscit: Information-seeking conversations with mixed-initiative interactions. TACL, 2023.
  82. Xu, C., Sun, Q., Zheng, K., Geng, X., Zhao, P., Feng, J., Tao, C., and Jiang, D. Wizardlm: Empowering large language models to follow complex instructions. arXiv preprint arXiv:2304.12244, 2023a.
  83. Xu, F., Shi, W., and Choi, E. RECOMP: Improving retrieval-augmented LMs with context compression and selective augmentation. In ICLR, 2024. URL https://openreview.net/forum?id= mlJLVigNHp. ↗
  84. Xu, P., Ping, W., Wu, X., McAfee, L., Zhu, C., Liu, Z., Subramanian, S., Bakhturina, E., Shoeybi, M., and Catanzaro, B. Retrieval meets long context large language models. arXiv preprint arXiv:2310.03025, 2023b.
  85. Yang, Z., Qi, P., Zhang, S., Bengio, Y., Cohen, W., Salakhutdinov, R., and Manning, C. D. Hotpotqa: A dataset for diverse, explainable multi-hop question answering. In EMNLP, 2018.
  86. Ye, F., Fang, M., Li, S., and Yilmaz, E. Enhancing conversational search: Large language model-aided informative query rewriting. In EMNLP, 2023. find
  87. Yu, S., Liu, J., Yang, J., Xiong, C., Bennett, P., Gao, J., and Liu, Z. Few-shot generative conversational query rewriting. In SIGIR, 2020. find
  88. Zhang, H., Diao, S., Lin, Y., Fung, Y. R., Lian, Q., Wang, X., Chen, Y., Ji, H., and Zhang, T. R-tuning: Teaching large language models to refuse unknown questions. arXiv preprint arXiv:2311.09677, 2023a. find
  89. Zhang, S., Dong, L., Li, X., Zhang, S., Sun, X., Wang, S., Li, J., Hu, R., Zhang, T., Wu, F., et al. Instruction tuning for large language models: A survey. arXiv preprint arXiv:2308.10792, 2023b.
  90. Zhou, C., Liu, P., Xu, P., Iyer, S., Sun, J., Mao, Y., Ma, X., Efrat, A., Yu, P., Yu, L., et al. Lima: Less is more for alignment. arXiv preprint arXiv:2305.11206, 2023. find
  91. Zhu, F., Lei, W., Huang, Y., Wang, C., Zhang, S., Lv, J., Feng, F., and Chua, T.-S. Tat-qa: A question answering benchmark on a hybrid of tabular and textual content in finance. In ACL, 2021.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/liu-2024-chatqa-surpassing-gpt-4-conversational/