Elamur Research⌕
☰
home / authors / Kaitao Song

Kaitao Song

Kaitao Song works on pre-training methods for neural language models and on systems built around large language models. Song's papers here ask how a model trained on unlabeled text can be made to serve both understanding and generation tasks, and later how a language model can direct other specialised models. The methods centre on Transformer architectures, masking and ordering schemes for pre-training objectives, and agent-style pipelines. The work falls under Language models & pretraining and Text generation.

The most cited paper here is MASS: Masked Sequence to Sequence Pre-training for Language Generation (2019). Encoder-only methods such as BERT suit generation poorly, so MASS trains an encoder and a decoder jointly. A contiguous stretch of a sentence is masked at the encoder, and the decoder must rebuild just that stretch. The approach was tested on multilingual WMT monolingual text and fine-tuned for several generation tasks. A year later, MPNet (2020) placed the masked objective of BERT and the permuted objective of XLNet in one shared framework. It then combined their strengths, predicting tokens in permuted order while keeping position information so the model sees a full-length sentence, as it does during fine-tuning.

The newest paper here, HuggingGPT (2023), moves from training objectives to coordination. ChatGPT acts as a controller that plans a multi-step job and calls expert models from the Hugging Face hub, using their natural-language descriptions as a common interface. This extends a text-only model to tasks involving images and speech.

Song's longest-running affiliation is Microsoft Research Asia in Beijing, with earlier bylines at Microsoft in Redmond and at Nanjing University of Science and Technology. More recent papers also list Fudan University and Apple. Frequent co-authors include Xu Tan, Tao Qin, Tie-Yan Liu and Jianfeng Lu.

Kaitao Song has 3 papers in our library, in 2019–2023, most of them on Language models & pretraining, Text generation; often with Xu Tan, Jianfeng Lu, Tao Qin.

3 papers in our library · 2019–2023

current affiliationFudan University, Shanghai, China; ShanghaiTech University, Shanghai, China; Microsoft Research Asia (China), Beijing, China; Apple (United States), Cupertino, United States

OpenAlex · ORCID · h-index 18 · 77 works · 9 672 citations OpenAlex, 2026-10-14

3 works in our library · 1 237 citations in our library · 4 more works in our library · ⭐ Paper Star rank #909

The profile is the library's match of the name; two people with one name are not always told apart.

Elsewhere: Google Scholar · Semantic Scholar · dblp · ACL Anthology

Affiliations

From the bylines of their papers (OpenAlex, 2026-10-14), ORCID and Wikidata; institutions on papers are matched by machine. 2 more institutions on single bylines are not listed.

Topics

Language models & pretraining2Text generation1

Works by year

2020s

20231 paper20201 paper

2010s

20191 paper

Papers

The most cited first (OpenAlex and our library).

  1. MASS: Masked Sequence to Sequence Pre-training for Language Generation (2019 · 1 109 citations)
    Pre-training methods like BERT work well for language understanding, but they train only an encoder. That makes them a poor fit for generation tasks, which use a full encoder-decoder. MASS addresses this with one pre-training objective that trains both halves together on unlabeled text. A contiguous span of a…
  2. MPNet: Masked and Permuted Pre-training for Language Understanding (2020 · 944 citations)
    BERT and XLNet each use a pre-training objective with its own weakness. BERT's masked language modeling (MLM) predicts each masked token independently of the others. XLNet's permuted language modeling (PLM) captures dependencies between predicted tokens, but it hides the positions of tokens that come later. As a…
  3. HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Face (2023 · 737 citations)
    Large language models handle text well, but they cannot directly see images, hear speech, or coordinate several specialised models to finish a multi-step job.

Also in our library

4 more works of theirs are in our library without a page here yet; they will appear as we publish them.

Co-authors here

6 co-authors with a page, the most papers together first.

Cited by

Papers of our library cite their works 1 237 times (of the 140 884 papers whose bibliographies we read); 95 citing papers with a page here →.

This page in Markdown → https://md.research.elamur.ai/free/en/authors/kaitao-song/