Elamur Research⌕
☰
home / papers / 2023 / 问答ChatGPT之后: 超大预训练模型的机遇和挑战

问答ChatGPT之后: 超大预训练模型的机遇和挑战

Summary

This Chinese-language survey from the Institute of Automation, CAS, uses ChatGPT as a lens on very large pre-trained models (PTMs). It is a review and position paper: it introduces no new dataset, corpus, or experiments. The authors first trace the growth of language PTMs (ELMo, BERT, GPT-1/2/3) and multimodal PTMs (CLIP, Flamingo, Gato, and Chinese systems such as 盘古, 悟道, M6, 紫东太初, and 文心/ERNIE). They then explain the Transformer decoder behind GPT, including masked multi-head attention. Using OpenAI's blog and the InstructGPT paper, since ChatGPT itself was unpublished, they reconstruct its likely training pipeline. Next, they reinterpret ChatGPT through their own "parallel intelligence" framework, Hanoi, as a human-in-the-loop "social closed loop" linking natural humans, digital humans, and robots. The paper closes by naming open challenges and proposing directions, including decision/control PTMs and industrial deployment. Its value lies mainly in being an accessible, early (2023) overview that sets out the authors' research agenda.

Key points

  • Published in 自动化学报 (Acta Automatica Sinica) 49(4):705–717, 2023 (DOI 10.16383/j.aas.c230107).
  • ChatGPT's pipeline is described in four stages:
    • self-supervised GPT pre-training;
    • supervised fine-tuning on human demonstrations;
    • a reward model trained on human rankings of outputs;
    • RLHF optimisation with PPO, on top of GPT-3.5.
  • The authors see a shift in AI paradigms, from "algorithmic intelligence" (AlphaGo) to "linguistic intelligence" (ChatGPT). They predict the next stage will be "imaginative intelligence" based on multimodal models.
  • Listed challenges include:
    • rapidly growing compute costs;
    • unreliable or fabricated answers;
    • little diversity in generated text;
    • legal and ethical risks.
  • Decision/control PTMs are judged to lag behind for three reasons: hard dynamic-system optimisation, no large unlabelled datasets, and difficulty with self-supervision. The proposed fix is labelled data from simulated "artificial societies."
  • Highlighted applications are AIGC, industrial PTMs (combining Hanoi, scenario engineering, AI4Science, and distillation), and digital humans/assistants.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages · Natural language processing

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.16383/j.aas.c230107

@inproceedings{2023,
  title = {{问答ChatGPT之后: 超大预训练模型的机遇和挑战}},
  author = {卢经纬 and 郭超 and 戴星原 and 缪青海 and 王兴霞 and 杨静 and 王飞跃},
  booktitle = {自动化学报 (Acta Automatica Sinica)},
  year = {2023},
  doi = {10.16383/j.aas.c230107},
  url = {https://doi.org/10.16383/j.aas.c230107}
}

References

66 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Wang F Y, Miao Q H, Li X, Wang X X, Lin Y L. What does ChatGPT say: The DAO from algorithmic intelligence to linguistic intelligence. IEEE/CAA Journal of Automatica Sinica, 2023, 10(3): 575−579 find
  2. Han X, Zhang Z X, Ding N, Gu Y X, Liu X, Huo Y Q, et al. Pre-trained models: Past, present and future. AI Open, 2021, 2: 225−250 find
  3. Bommasani R, Hudson D A, Adeli E, Altman R, Arora S, von Arx S, et al. On the opportunities and risks of foundation models [Online], available: https://arxiv.org/abs/2108.07258, August 16, 2021 ↗ our card
  4. Lu J W, Wang X X, Cheng X, Yang J, Kwan O, Wang X. Parallel factories for smart industrial operations: From big AI models to field foundational models and scenarios engineering. IEEE/CAA Journal of Automatica Sinica, 2022, 9(12): 2079− 2086 find
  5. Wang X X, Cheng X, Lu J W, Kwan O, Li S X, Ping Z X. Metaverses-based parallel oil fields in CPSS: A framework and methodology. IEEE Transactions on Systems, Man, and Cybernetics: Systems, 2023, 53(4): 2138−2147 find
  6. Yang J, Li S M, Wang X X, Lu J W, Wu H Y, Wang X. De-FACT in ManuVerse for parallel manufacturing: Foundation models and parallel workers in smart factories. IEEE Transactions on Systems, Man, and Cybernetics: Systems, 2023, 53(4): 2188−2199 find
  7. Chen T, Kornblith S, Swersky K, Norouzi M, Hinton G. Big self-supervised models are strong semi-supervised learners. In: Proceedings of the 34th International Conference on Neural Information Processing Systems. Vancouver, Canada: Curran Associates Inc., 2020. find
  8. Ramesh A, Dhariwal P, Nichol A, Chu C, Chen M. Hierarchical text-conditional image generation with CLIP latents [Online], available: https://arxiv.org/abs/2204.06125, April 13, 2022 ↗ our card
  9. Rombach R, Blattmann A, Lorenz D, Esser P, Ommer B. High-resolution image synthesis with latent diffusion models. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). New Orleans, USA: IEEE, 2022. 10674−10685 find
  10. Vaswani A, Shazeer N, Parmar N, Uszkoreit U, Jones L, Gomez A N, et al. Attention is all you need. In: Proceedings of the 31st International Conference on Neural Information Processing Systems. Long Beach, USA: Curran Associates Inc., 2017. 6000− 6010 find
  11. Peters M E, Neumann M, Iyyer M, Gardner M, Clark C, Lee K, et al. Deep contextualized word representations. In: Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. New Orleans, USA: Association for Computational L… find
  12. Devlin J, Chang M W, Lee K, Toutanova K. BERT: Pre-training of deep bidirectional transformers for language understanding. In: Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Minneapolis, USA: Association for… find
  13. Radford A, Narasimhan K, Salimans T, Sutskever I. Improving language understanding by generative pre-training [Online], available: https://s3-us-west-2.amazonaws.com/openai-assets/re-search-covers/language-unsupervised/language_understanding_paper.pdf, April 13, 2022 ↗
  14. Radford A, Wu J, Child R, Luan D, Amodei D, Sutskever I. Language models are unsupervised multitask learners [Online], available: https://cdn.openai.com/better-language-models/langu-age_models_are_unsupervised_multitask_learners.pdf, April 13, 2022 ↗
  15. Brown T B, Mann B, Ryder N, Subbiah M, Kaplan J, Dhariwal P, et al. Language models are few-shot learners. In: Proceedings of the 34th International Conference on Neural Information Processing Systems. Vancouver, Canada: Curran Associates Inc., 2020. Article No. 159 find
  16. Raffel C, Shazeer N, Roberts A, Lee K, Narang S, Matena M, et al. Exploring the limits of transfer learning with a unified text-to-text transformer. The Journal of Machine Learning Research, 2020, 21(1): Article No. 140 find
  17. Zhong R Q, Lee K, Zhang Z, Klein D. Adapting language models for zero-shot learning by meta-tuning on dataset and prompt collections. In: Proceedings of the Findings of the Association for Computational Linguistics. Punta Cana, Dominican Republic: Association for Computational Linguistics, 2021. 285… find
  18. Lu J S, Batra D, Parikh D, Lee S. ViLBERT: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks. In: Proceedings of the 33rd International Conference on Neural Information Processing Systems. Vancouver, Canada: Curran Associates Inc., 2019. Article No. 2 find our card
  19. Radford A, Kim J W, Hallacy C, Ramesh A, Goh G, Agarwal S, et al. Learning transferable visual models from natural language supervision. In: Proceedings of the 38th International Conference on Machine Learning (ICML). PMLR, 2021. 8748−8763 find our card
  20. Alayrac J B, Donahue J, Luc P, Miech A, Barr I, Hasson Y, et al. Flamingo: A visual language model for few-shot learning. In: Proceedings of the 36th International Conference on Neural Information Processing Systems (NeurIPS). New Orleans, USA: 2022. find our card
  21. Reed S, Żołna K, Parisotto E, Colmenarejo S G, Novikov A, Barth-Maron G, et al. A generalist agent. Transactions on Machine Learning Research, 2022: 1−42 find
  22. Zeng W, Ren X, Su T, Wang H, Liao Y, Wang Z, et al. PanGu-α : Large-scale autoregressive pretrained Chinese language models with auto-parallel computation [Online], available: https://arxiv.org/abs/2104.12369, April 26, 2021 ↗
  23. Fei N Y, Lu Z W, Gao Y Z, Yang G X, Huo Y Q, Wen J Y, et al. Towards artificial general intelligence via a multimodal foundation model. Nature Communications, 2022, 13(1): Article No. 3094 find
  24. Lin J Y, Men R, Yang A, Zhou C, Ding M, Zhang Y C, et al. M6: A Chinese multimodal pretrainer [Online], available: https://arxiv.org/abs/2103.00823, May 29, 2021 — [arXiv:2103.00823](https://arxiv.org/abs/2103.00823) ↗
  25. Liu J, Zhu X X, Liu F, Guo L T, Zhao Z J, Sun M Z, et al. OPT: Omni-perception pre-trainer for cross-modal understanding and generation [Online], available: https://arxiv.org/abs/2107.00249, July 6, 2021 — [arXiv:2107.00249](https://arxiv.org/abs/2107.00249) ↗
  26. Zhang Z Y, Han X, Liu Z Y, Jiang X, Sun M S, Liu Q. ERNIE: Enhanced language representation with informative entities. In: Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. Florence, Italy: Association for Computational Linguistics, 2019. 1441−1451 find
  27. Wang S H, Sun Y, Xiang Y, Wu Z H, Ding S Y, Gong W B, et al. ERNIE 3.0 Titan: Exploring larger-scale knowledge enhanced pre-training for language understanding and generation [Online], available: https://arxiv.org/abs/2112.12731, December 23, 2021 ↗
  28. Hoffmann J, Borgeaud S, Mensch A, Buchatskaya E, Cai T, Rutherford E, et al. Training compute-optimal large language models [Online], available: https://arxiv.org/abs/2203.15556, March 29, 2022 ↗
  29. Wang Fei-Yue. Parallel system methods for management and control of complex systems. Control and Decision, 2004, 19(5): 485−489, 514 (王飞跃. 平行系统方法与复杂系统的管理和控制. 控制与决策, 2004, 19(5): 485−489, 514) find
  30. Wang Fei-Yue. Parallel control and digital twins: Control theory revisited and reshaped. Chinese Journal of Intelligent Science and Technology, 2020, 2(3): 293−300 (王飞跃. 平行控制与数字孪生: 经典控制理论的回顾与重铸. 智能 科学与技术学报, 2020, 2(3): 293−300) find
  31. Wang F Y. Parallel intelligence in metaverses: Welcome to Hanoi! IEEE Intelligent Systems, 2022, 37(1): 16−20 find
  32. Chen L, Zhang Y Q, Tian B, Ai Y F, Cao D P, Wang F Y. Parallel driving OS: A ubiquitous operating system for autonomous driving in CPSS. IEEE Transactions on Intelligent Vehicles, 2022, 7(4): 886−895 find
  33. Tian F Y, Li Z H, Wang F Y, Li L. Parallel learning-based steering control for autonomous driving. IEEE Transactions on Intelligent Vehicles, 2023, 8(1): 379−389 find
  34. Wang J G, Wang X, Shen T Y, Wang Y T, Li L, Tian Y L, et al. Parallel vision for long-tail regularization: Initial results from IVFC autonomous driving testing. IEEE Transactions on Intelligent Vehicles, 2022, 7(2): 286−299 find
  35. Wang K F, Gou C, Zheng N N, Rehg J M, Wang F Y. Parallel vision for perception and understanding of complex scenes: Methods, framework, and perspectives. Artificial Intelligence Review, 2017, 48(3): 299−329 find
  36. Ouyang L, Wu J, Jiang X, Almeida D, Wainwright C L, Mishkin P, et al. Training language models to follow instructions with human feedback. In: Proceedings of the 36th International Conference on Neural Information Processing Systems (NeurIPS). New Orleans, USA: 2022. find
  37. Tian Yong-Lin, Wang Yu-Tong, Wang Jian-Gong, Wang Xiao, Wang Fei-Yue. Key problems and progress of vision Transformers: The state of the art and prospects. Acta Automatica Sinica, 2022, 48(4): 957−979 (田永林, 王雨桐, 王建功, 王晓, 王飞跃. 视觉 Transformer 研究的 关键问题: 现状及展望. 自动化学报, 2022, 48(4): 957−979) find
  38. Schulman J, Wolski F, Dhariwal P, Radford A, Klimov O. Proximal policy optimization algorithms [Online], available: https://arxiv.org/abs/1707.06347, August 28, 2017 ↗
  39. Sutton R S, Barto A G. Reinforcement Learning: An Introduction (Second edition). Cambridge: MIT Press, 2018. find
  40. Lu Jing-Wei, Cheng Xiang, Wang Fei-Yue. Artificial intelligence and deep learning methods for solving differential equations: The state of the art and prospects. Chinese Journal of Intelligent Science and Technology, 2022, 4(4): 461−476 (卢经纬, 程相, 王飞跃. 求解微分方程的人工智能与深度学习方法: 现状及展望. 智能科学与技术学报, 2022, 4(4)… find
  41. Mnih V, Kavukcuoglu K, Silver D, Rusu A A, Veness J, Bellemare M G, et al. Human-level control through deep reinforcement learning. Nature, 2015, 518(7540): 529−533 find
  42. Wang Fei-Yue. Parallel management: The DAO to smart ecological technology for complexity management intelligence. Acta Automatica Sinica, 2022, 48(11): 2655−2669 (王飞跃. 平行管理: 复杂性管理智能的生态科技与智慧管理之 DAO. 自动化学报, 2022, 48(11): 2655−2669) find
  43. Li Li, Lin Yi-Lun, Cao Dong-Pu, Zheng Nan-Ning, Wang Fei-Yue. Parallel learning —— A new framework for machine learning. Acta Automatica Sinica, 2017, 43(1): 1−8 (李力, 林懿伦, 曹东璞, 郑南宁, 王飞跃. 平行学习 —— 机器学习 的一个新型理论框架. 自动化学报, 2017, 43(1): 1−8) find
  44. Wang X J, Kang M Z, Sun H Q, de Reffye P, Wang F Y. DeCASA in AgriVerse: Parallel agriculture for smart villages in Metaverses. IEEE/CAA Journal of Automatica Sinica, 2022, 9(12): 2055−2062 find
  45. Wang J G, Tian Y L, Wang Y T, Yang J, Wang X X, Wang S J, et al. A framework and operational procedures for metaverses-based industrial foundation models. IEEE Transactions on Systems, Man, and Cybernetics: Systems, 2023, 53(4): 2037−2046 find
  46. Li X, Tian Y L, Ye P J, Duan H B, Wang F Y. A novel scenarios engineering methodology for foundation models in metaverse. IEEE Transactions on Systems, Man, and Cybernetics: Systems, 2023, 53(4): 2148−2159 find
  47. Wang Y T, Wang J G, Cao Y S, Li S X, Kwan O. Integrated inspection on PCB manufacturing in cyber-physical-social systems. IEEE Transactions on Systems, Man, and Cybernetics: Systems, 2023, 53(4): 2098−2106 find
  48. Liu Y H, Shen Y, Tian Y L, Ai Y F, Tian B, Wu E, et al. RadarVerses in metaverses: A CPSI-based architecture for 6S radar systems in CPSS. IEEE Transactions on Systems, Man, and Cybernetics: Systems, 2023, 53(4): 2128−2137 find
  49. Lu J W, Wei Q L, Wang F Y. Parallel control for optimal tracking via adaptive dynamic programming. IEEE/CAA Journal of Automatica Sinica, 2020, 7(6): 1662−1674 find
  50. Wang Fei-Yue, Chen Jun-Long. Intelligent Control Method and Application. Beijing: Science and Technology of China Press, 2020. (王飞跃, 陈俊龙. 智能控制方法与应用. 北京: 中国科学技术出版 社, 2020.) find
  51. Wang Fei-Yue. Parallel philosophy and intelligent science: From Leibniz＇ s Monad to Blockchain＇ s DAO. Pattern Recognition and Artificial Intelligence, 2020, 33(12): 1055−1065 find
  52. (王飞跃. 平行哲学与智能科学: 从莱布尼茨的 Monad 到区块链 之 DAO. 模式识别与人工智能, 2020, 33(12): 1055−1065) find
  53. Lu J W, Wei Q L, Zhou T M, Wang Z Y, Wang F Y. Event-triggered near-optimal control for unknown discrete-time nonlinear systems using parallel control. IEEE Transactions on Cybernetics, 2023, 53(3): 1890−1904 find
  54. Zhang L M, Agrawala M. Adding conditional control to text-to-image diffusion models [Online], available: https://arxiv.org/abs/2302.05543, February 10, 2023 — [arXiv:2302.05543](https://arxiv.org/abs/2302.05543) ↗
  55. Guo C, Dou Y, Bai T X, Dai X Y, Wang C F, Wen Y. Art-Verse: A paradigm for parallel human-machine collaborative painting creation in metaverses. IEEE Transactions on Systems, Man, and Cybernetics: Systems, 2023, 53(4): 2200−2208 find
  56. Li X, Ye P J, Li J J, Liu Z M, Cao L B, Wang F Y. From features engineering to scenarios engineering for trustworthy AI: I&I, C&C, and V&V. IEEE Intelligent Systems, 2022, 37(4): 18−26 find
  57. Ye P J, Wang F Y. Parallel Population and Parallel Human —— A Cyber-Physical Social Approach. Hoboken: Wiley-IEEE Press, 2023. find
  58. Ye P J, Wang X, Zheng W B, Wei Q L, Wang F Y. Parallel cognition: Hybrid intelligence for human-machine interaction and management. Frontiers of Information Technology & Electronic Engineering, 2022, 23(12): 1765−1779 find
  59. Wang Fei-Yue, Jiang Zheng-Hua, Dai Ru-Wei. Population studies and artificial societies: A discussion of artificial population systems and their applications. Complex Systems and Complexity Science, 2005, 2(1): 1−9 (王飞跃, 蒋正华, 戴汝为. 人口问题与人工社会方法: 人工人口系 统的设想与应用. 复杂系统与复杂性科学, 2005, 2(1): 1−9) find
  60. Ye P J, Wang F Y. Parallel population and parallel human —— A cyber-physical social approach. IEEE Intelligent Systems, 2022, 37(5): 19−27 find
  61. 卢经纬 青岛智能产业技术研究院副 研究员. 2022 年获得中国科学院大学 计算机应用技术博士学位. 主要研究 方向为最优控制, 自适应动态规划, 深度强化学习和自动驾驶. E-mail: lujingweihh@gmail.com (LU Jing-Wei Associate professor at the Qingdao Academy of Intelligent Industries. He received his Ph.D. degree in computer application technology from University of Chinese Academy o… find
  62. 郭 超 中国科学院自动化研究所复 杂系统管理与控制国家重点实验室助 理研究员. 主要研究方向为机器艺术 创作, 智能机器人系统, 深度学习, 强 化学习. E-mail: guochao2014@ia.ac.cn (GUO Chao Assistant professor at the State Key Laboratory for Management and Control of Complex Systems, Institute of Automation, Chinese Academy of Sciences. His research interest covers AI ar… find
  63. 缪青海 中国科学院大学人工智能学 院副教授. 2007 年获得中国科学院自 动化研究所博士学位. 主要研究方向 为智能系统, 机器学习, 计算机视觉. E-mail: miaoqh@ucas.ac.cn (MIAO Qing-Hai Associate professor at the School of Artificial Intelligence, University of Chinese Academy of Sciences. He received his Ph.D. degree from the Institute of Automation, Chinese Academy… find
  64. 王兴霞 中国科学院自动化研究所复 杂系统管理与控制国家重点实验室博 士研究生. 2021 年获得南开大学工学 硕士学位. 主要研究方向为平行控制, 平行油田和多智能体系统. E-mail: wangxingxia2022@ia.ac.cn (WANG Xing-Xia Ph.D. candidate at the State Key Laboratory for Management and Control of Complex Systems, Institute of Automation, Chinese Academy of Sciences. She received her mas… find
  65. 杨 静 中国科学院自动化研究所复 杂系统管理与控制国家重点实验室博 士研究生. 2020 年获得北京化工大学 自动化学士学位. 主要研究方向为平 行制造, 社会制造, 人工智能和社会 物理信息系统. E-mail: yangjing2020@ia.ac.cn (YANG Jing Ph.D. candidate at the State Key Laboratory for Management and Control of Complex Systems, Institute of Automation, Chinese Academy of Sciences. She received h… find
  66. 王飞跃 中国科学院自动化研究所复 杂系统管理与控制国家重点实验室研 究员. 主要研究方向为智能系统, 复 杂系统建模, 分析与控制. 本文通信 作者. E-mail: feiyue.wang@ia.ac.cn (WANG Fei-Yue Professor at the State Key Laboratory for Management and Control of Complex Systems, Institute of Automation, Chinese Academy of Sciences. His research interest covers modeling, ana… find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/papers/lu-2023-questions-after-chatgpt/