💭 Chatbots

About 16 minutes to read · every paper named here has its own page in this library

What it is

A chatbot is software you can talk to. You type or say something, and it answers in words. The exchange can continue over many turns, and each answer should fit what came before. Chatbots are the most visible part of the wider field of dialogue systems. That field also covers spoken assistants and the theory of how conversations are structured. It draws on natural language understanding, natural language generation and pragmatics, the study of what speakers mean in context.

The field usually separates chatbots by how they produce a reply and by what they are for.

• Rule-based chatbots match the user's words against hand-written patterns and fill in templated answers. The classic example is ELIZA (Weizenbaum, 1966), which imitated a psychotherapist by turning statements back into questions. PARRY (Colby, 1972) and A.L.I.C.E. (Wallace, 1995) belong to the same lineage.
• Retrieval-based chatbots keep a large pool of human-written replies. For each turn they pick the reply that best matches the conversation so far (Wu et al. 2017).
• Task-oriented systems help a user reach a goal, such as booking a restaurant or a train ticket. They represent what the user wants as intents (the kind of request) and slots (attributes such as area, price or time). A dialogue state records the slot values gathered so far (Budzianowski et al. 2018; Lei et al. 2018).
• Open-domain generative chatbots write new replies word by word on any topic. Their purpose is engaging conversation rather than completing a task (Vinyals & Le 2015; Adiwardana et al. 2020).
• Assistants built on large language models are general models tuned to follow instructions. They merge chit-chat, question answering and task help in one system (Ouyang et al. 2022; OpenAI et al. 2023).

Social chatbots form a further category. They are designed as long-term companions, and their success is judged by sustained engagement rather than single answers. XiaoIce is the best-documented case (Zhou et al. 2018).

How it works

The classical pipeline

Task-oriented systems were long built as a chain of modules:

1. Understanding turns the utterance into an intent plus tagged slots.
2. A state tracker updates the belief about the user's goal.
3. A policy chooses the next system act.
4. A generator renders that act as text.

A single neural network can jointly predict the intent of an utterance and its slot tags. The tags are written in IOB format, which marks each word as inside, outside or at the beginning of a slot span (Li et al. 2017). A policy can be warm-started by imitating a rule-based agent and then refined with reinforcement learning against a simulated user (Li et al. 2017). In that setting, errors in slots turned out to be more damaging than errors in intents, and wrong slot values were the most harmful of all (Li et al. 2017).

The generator was traditionally a set of templates. The Semantically Conditioned LSTM (SC-LSTM) replaced them with a learned model. An LSTM, or long short-term memory network, is a recurrent neural network that carries a memory cell from word to word. In SC-LSTM, a gate gradually "uses up" a vector that encodes the dialogue act and its slot values as those values are expressed in the output (Wen et al. 2015).

Neural generation as language modelling

Open-domain generative chatbots treat the reply as a sequence predicted from the context. Write x for the conversation history and y = (y₁, …, y_T) for the reply, split into T tokens (usually units produced by subword tokenization). A neural language model then factorises the probability of the reply as

P(y | x) = ∏ₜ₌₁ᵀ P(yₜ | y₁, …, yₜ₋₁, x)

Here each factor is the probability of the t-th token given the history x and the tokens already produced. Training minimises the cross-entropy of human replies under this model.

The most direct early version was a sequence-to-sequence (seq2seq) LSTM, which encodes the context and then decodes the reply. It was trained on helpdesk logs and movie subtitles (Vinyals & Le 2015). A hierarchical encoder-decoder (HRED) improved on it by adding a second recurrent network that tracks the conversation across turns (Serban et al. 2015).

Later systems put the whole conversation into a single Transformer. DialoGPT simply concatenates the turns of a Reddit thread and models them autoregressively, as GPT-2 does with ordinary text (Zhang et al. 2019).

Decoding strongly shapes quality. Choosing the single most probable reply favours bland answers like "I don't know" (Serban et al. 2015). One common remedy samples several candidates and reranks them with a backward model that asks how well the reply predicts its context. This is the maximum-mutual-information idea:

score(y) = log P(y | x) + λ · log P(x | y)

The first term rewards fluent, likely replies. The second rewards replies specific enough to point back to x. The weight λ ≥ 0 balances the two. DialoGPT uses this kind of reranking (Zhang et al. 2019).

The BlenderBot study, "Recipes for Building an Open-Domain Chatbot", showed that decoding choices change human ratings sharply even at equal perplexity. Forcing a minimum reply length in beam search mattered especially (Roller et al. 2021).

Modern assistants: instructions, history, retrieval and tools

A contemporary assistant starts from a pretrained large language model. It is then tuned in two steps:

• supervised fine-tuning on demonstrations of good answers;
• reinforcement learning from human feedback (RLHF), in which a reward model trained on human rankings guides further optimisation (Ouyang et al. 2022).

At inference time the model receives one long input. It contains:

• a system instruction that sets role and rules;
• the conversation history so far;
• the new user turn.

The model's memory of the conversation is just this context window, and the window is bounded (OpenAI et al. 2023). The same holds for earlier open-domain bots. BlenderBot kept only 128 tokens of history, counted in byte-pair encoding (BPE) subword units (Roller et al. 2021).

To get past what the model memorised in training, assistants call outside resources, an approach known as retrieval-augmented generation. Wizard of Wikipedia framed knowledge-grounded chat as conversation backed by retrieved encyclopedia passages (Dinan et al. 2018).

BlenderBot 3 goes further. A single network decides whether to search the web, writes the query, reads the results, and reads from and writes to a long-term memory. Control codes tell the network which module it is running at each step (Shuster et al. 2022).

Task-oriented systems built on pretrained models follow the same pattern with structured tools. GPT-2 can produce the dialogue state and the system action as intermediate text. An external module then queries a database before the model writes the reply (Ham et al. 2020). Key-value retrieval over a knowledge base is an earlier form of this tool use (Eric et al. 2017).

History

The idea of judging a machine by its conversation goes back to Turing (1950). ELIZA (Weizenbaum, 1966) showed how far shallow pattern matching could go in giving an impression of understanding. For decades afterwards, chatbots were mostly scripted, and serious dialogue research concentrated on spoken, task-oriented systems.

From the late 1990s, statistical methods, notably reinforcement learning, entered dialogue management. Later managers were based on partially observable Markov decision processes (POMDPs), which keep a probability distribution over what the user might want. Their policies need many dialogues to train, so a probabilistic agenda-based user simulator was proposed to bootstrap them. A policy learned purely from simulated users then completed 90.6% of 160 dialogues with 40 human subjects (Schatzmann et al. 2007).

A data-driven turn for open chat came in 2011. Ritter, Cherry and Dolan treated a reply to a Twitter post as a translation of that post. They trained a phrase-based statistical machine translation (MT) system, which learns how chunks of one text map onto chunks of another, on about 1.3 million conversations (Ritter et al. 2011).

In industry, a parallel thread began with Microsoft's XiaoIce, launched in China in May 2014. By the time of its 2018 report, it had reached more than 660 million active users (Zhou et al. 2018).

The neural era in research began in 2015:

• The sequence-to-sequence model of Vinyals and Le, described above, was preferred by human judges over the Cleverbot chatbot in 97 of 200 questions, against 60 the other way (Vinyals & Le 2015).
• HRED (Serban et al. 2015) appeared the same year.
• The Ubuntu Dialogue Corpus supplied about 930,000 two-person dialogues for training (Lowe et al. 2015).
• Cambridge researchers made generation and then whole task-oriented systems trainable end to end (Wen et al. 2015; Wen et al. 2017).

Pretraining reshaped the field from 2019 to 2022:

• DialoGPT (Zhang et al. 2019), with a 762M-parameter version as its largest.
• Google's Meena, with 2.6B parameters trained on 40B words of social-media conversation (Adiwardana et al. 2020).
• Facebook's BlenderBot, with generators up to 9.4B parameters (Roller et al. 2021).
• Baidu's PLATO-2, up to 1.6B parameters (Bao et al. 2021).
• The 175B-parameter BlenderBot 3, deployed publicly (Shuster et al. 2022).

The decisive step toward ChatGPT-style assistants was InstructGPT. Raters preferred the outputs of a 1.3B-parameter instruction-tuned model over those of the 175B GPT-3 (Ouyang et al. 2022). GPT-4 followed in 2023 as a multimodal model aligned with RLHF (OpenAI et al. 2023). From 2023, evaluation moved toward large-scale human preference voting between assistants (Zheng et al. 2023; Chiang et al. 2024).

Methods

Retrieval and response selection

Retrieval chatbots must score each candidate reply against a multi-turn history. The Sequential Matching Network compares the candidate with every utterance separately. It then accumulates the matches in dialogue order with a gated recurrent unit (GRU), a simplified recurrent cell. It gained more than 6% in R10@1 on Ubuntu over the best baseline (Wu et al. 2017). R10@1 is the share of test contexts in which the correct reply ranks first among ten candidates (see the metrics section below).

The Deep Attention Matching Network replaced recurrence with stacked self-attention and cross-attention, adding over 4% R10@1 on Ubuntu and Douban (Zhou et al. 2018). It targets replies that connect to the context only implicitly, for instance through a pronoun that refers back to something said earlier. Resolving such references is the task of coreference resolution (Zhou et al. 2018). Its errors show the limits of matching: candidates that are almost right but wrong in detail, and replies that contradict the context (Zhou et al. 2018).

Retrieval also persists inside generative systems. BlenderBot compared a poly-encoder retriever with generators and with hybrid retrieve-and-refine designs (Roller et al. 2021).

Task-oriented modelling

Research on task-oriented dialogue has steadily merged modules:

• Sequicity writes the dialogue state as a short "belief span" of text and generates the reply after it. One seq2seq model replaces a pipeline of per-slot classifiers and still handles out-of-vocabulary values (Lei et al. 2018).
• A transferable state generator was proposed to share state tracking across domains (Wu et al. 2019).
• SimpleTOD recast the whole task as plain language modelling (Hosseini-Asl et al. 2020).
• MinTL fine-tunes T5 or BART to emit only edits to the previous state. With 20% of the data it matches a strong baseline trained on all of it (Lin et al. 2020).
• TOD-BERT continues BERT pretraining on nine merged task-oriented corpora. It helps most when labels are scarce (Wu et al. 2020), a few-shot learning setting typical of new domains.

Generation for new domains can be bootstrapped with "counterfeit" data built by swapping slots of the same functional class. With about 10% of target data, this comes close to training from scratch on all of it (Wen et al. 2016). Tree-structured meaning representations combined with constrained decoding give control over discourse relations such as contrast. They also reduce omitted or invented content (Balakrishnan et al. 2019).

Training policies by interaction raises its own problems:

• Drift from the corpus. Supervised agents drift into states absent from their corpus. Letting users correct the agent turn by turn, before reinforcement learning, raised task success over supervised learning plus RL alone (Liu et al. 2018).
• Self-play. A simulator and a rule-based agent can talk to each other and produce annotated dialogue outlines, which crowd workers then rewrite into natural language (Shah et al. 2018).
• A medical case. An agent trained as a deep Q-network (DQN), a reinforcement-learning method that estimates the value of each possible action, asked patients about symptoms. It reached a success rate of 0.65 on simulated diagnosis dialogues, against 0.23 for a rule-based agent. The simulator drew its user goals from real patient records collected on an online health platform (Wei et al. 2018).

Persona, empathy and consistency

Early neural chatbots had no stable identity. One answered that it was a lawyer and, a turn later, a doctor (Vinyals & Le 2015). Consistency became a design target in its own right. XiaoIce designs a persona for each region, and an empathy module tracks the user's emotion and intent across a conversation (Zhou et al. 2018).

Blended Skill Talk asks whether persona, knowledge and empathy can be combined in one agent. Multi-task training followed by fine-tuning on blended conversations worked best (Smith et al. 2020). MIME models empathy by partly mimicking the speaker's emotion, with separate handling for positive and negative emotions (Majumder et al. 2020). PLATO-2 handles the one-to-many nature of dialogue with discrete latent variables. A separate model then selects the most coherent of the diverse candidates (Bao et al. 2021).

Persuasive dialogue adds strategy. Studies of persuasion for charity found that a strategy's effect can depend on the personality of the person being persuaded (Wang et al. 2019).

Alignment, safety and faithfulness

Because chatbots learn from human conversation, they absorb its toxicity and bias; see bias, fairness and safety. Facebook researchers compared several remedies (Xu et al. 2020):

• detecting unsafe utterances and filtering them in a second stage;
• filtering training data and controlling generation;
• steering away from sensitive topics;
• reducing gender bias.

They also introduced "baked-in" safety, where unsafe training targets are rewritten so the generator itself learns safe replies (Xu et al. 2020).

Bot-Adversarial Dialogue recruited crowdworkers to provoke models deliberately. Even the best combined system still failed on about 5.6% of adversarial turns (Xu et al. 2021). Fairness tests that swap gendered or dialect words in otherwise identical contexts found significant differences in how generative and retrieval models responded (Liu et al. 2020). ProsocialDialog trains agents to push back constructively against problematic statements, guided by explicit rules of thumb, rather than agreeing or deflecting (Kim et al. 2022).

Hallucination, meaning fluent but unsupported claims, is the other main risk. On OpenDialKG, a dataset of chat grounded in a knowledge graph, a critic can mask suspicious entity mentions and replace them with entities retrieved from the graph. In human evaluation, this cut hallucinations by 42.05% for the GPT2-KB variant (Dziri et al. 2021).

At the level of whole assistants, RLHF reduced fabrication on closed-domain tasks from 41% to 21% of cases (Ouyang et al. 2022). Hallucination remains a stated limitation of GPT-4 (OpenAI et al. 2023). Domain assistants such as the medical MedAlpaca models face the same risk, and their authors warn about confabulation (Han et al. 2023).

Datasets and benchmarks

Corpora

Dialogue data falls mainly into three families: scraped human conversation, crowd-collected Wizard-of-Oz dialogues, and adversarial or norm-focused sets.

Other resources extend the field in specific directions. Image-grounded conversation adds a shared photo as context, which ties chatbots to research on multimodal learning (Mostafazadeh et al. 2017). MovieTriples supplied three-turn exchanges from movie scripts (Serban et al. 2015).

Pretraining corpora are far larger. DialoGPT used about 147 million Reddit dialogue instances (Zhang et al. 2019). BlenderBot used 1.5 billion Reddit comments (Roller et al. 2021).

Automatic metrics and their limits

Response selection is scored with Recall@k. Given n candidates of which one is correct, Rₙ@k is the fraction of test contexts where the correct reply ranks among the top k. This is the R10@1 reported for matching networks (Lowe et al. 2015).

Generated replies were long scored with word-overlap metrics borrowed from translation, such as BLEU. A careful study found that these metrics barely track human judgement: weakly on Twitter and almost not at all on Ubuntu (Liu et al. 2016). The cause is structural. A good reply may share no words with the single reference. Embedding-based metrics capture topic closeness more than appropriateness (Liu et al. 2016).

Two responses followed:

• Learned metrics. GRADE adds commonsense graph reasoning over ConceptNet to BERT features to score coherence (Huang et al. 2020).
• Strong language models as judges. Their agreement with human preference has been studied on multi-turn questions (Zheng et al. 2023).

Human ratings and task success

Human evaluation remains the reference standard. Meena introduced the Sensibleness and Specificity Average (SSA). Raters first ask whether a reply makes sense and then whether it is specific to the context, so a generic bot that always says "ok" is penalised. Across Meena variants, lower perplexity correlated with higher SSA, with R² = 0.93 (Adiwardana et al. 2020).

BlenderBot used pairwise ACUTE-Eval comparisons and beat Meena 75% to 25% on engagingness (Roller et al. 2021). Chatbot Arena turned pairwise comparison into an open platform where users vote between anonymous assistants (Chiang et al. 2024).

Task-oriented systems are judged by task success, ideally with real users. In the DSTC8 end-to-end track, the GPT-2 pipeline system ranked first with a 68.32% human-evaluated success rate (Ham et al. 2020). Social chatbots use engagement instead. XiaoIce reports conversation-turns per session averaged over months of logs, and its authors argue that bland replies cannot easily inflate this measure (Zhou et al. 2018).

See also evaluation and benchmarks.

People and institutions

Several groups shaped neural chatbots before the large-model era, and four of them stand out.

At Microsoft Research, Bill Dolan co-authored the translation-style response generator (Ritter et al. 2011). With Chris Brockett, Michel Galley and Jianfeng Gao, he worked on image-grounded conversation (Mostafazadeh et al. 2017). Galley and Brockett were also among the authors of DialoGPT (Zhang et al. 2019). Gao co-authored:

• the XiaoIce design paper (Zhou et al. 2018);
• work on end-to-end task completion (Li et al. 2017);
• a broad survey of neural conversational AI (Gao et al. 2018).

Alan Ritter, first author of the 2011 study, is at Georgia Institute of Technology.

In Cambridge, Steve Young led statistical spoken dialogue work, from agenda-based user simulation (Schatzmann et al. 2007) to neural generation (Wen et al. 2015; Wen et al. 2016). Milica Gašić and David Vandyke co-authored the neural generation papers and the end-to-end trainable task-oriented system (Wen et al. 2017). Gašić is now at Heinrich Heine University Düsseldorf.

In Montreal, Iulian Serban and Joelle Pineau worked on HRED (Serban et al. 2015). With Ryan Lowe they built the Ubuntu corpus (Lowe et al. 2015) and showed the weakness of overlap metrics (Liu et al. 2016).

At Facebook AI Research, Jason Weston co-authored:

• Wizard of Wikipedia (Dinan et al. 2018);
• the BlenderBot recipes (Roller et al. 2021);
• Blended Skill Talk (Smith et al. 2020);
• the safety studies (Xu et al. 2020; Xu et al. 2021).

He works at Meta and has also been affiliated with New York University. Eric Michael Smith contributed to Blended Skill Talk, the BlenderBot recipes and BlenderBot 3 (Shuster et al. 2022).

Other contributions cover the open-domain systems from Baidu and the newer safety and grounding work:

• Hua Wu co-authored PLATO-2 (Bao et al. 2021) and the Deep Attention Matching Network (Zhou et al. 2018).
• Nouha Dziri, now at the Allen Institute for Artificial Intelligence, studies hallucination in grounded dialogue (Dziri et al. 2021).
• Yejin Choi, now at Stanford University, co-authored ProsocialDialog (Kim et al. 2022).

Active institutions also include Carnegie Mellon University, Tsinghua University, Meta and Harbin Institute of Technology.

Open questions

Evaluation is still unresolved:

• Automatic overlap metrics fail for dialogue (Liu et al. 2016).
• Learned metrics struggle when replies contain few keywords (Huang et al. 2020).
• Using a language model as judge raises its own questions of bias and agreement with humans (Zheng et al. 2023).
• Human ratings also have weak spots. Crowd agreement on Meena's specificity judgements was only moderate (Adiwardana et al. 2020), and short evaluation chats say little about long-term behaviour (Roller et al. 2021).

Memory and consistency remain weak. Context windows are bounded, and models do not learn after deployment (OpenAI et al. 2023). BlenderBot 3's "continual learning" is batch retraining rather than online updating (Shuster et al. 2022). Long-term memory modules exist, but keeping a stable persona over months, as social chatbots aim to do (Zhou et al. 2018), has not been solved in a principled way.

Faithfulness to sources is a third concern. Grounding through retrieval and tools reduces fabrication but does not remove it. In OpenDialKG, 54.80% of responses already mention entities that have no support in either the dialogue history or the 1-hop subgraph. Models can therefore learn hallucination from the data itself (Dziri et al. 2021).

Safety and values are a fourth:

• Adversarial users still break the safest systems (Xu et al. 2021).
• Debiasing can trade fairness against response diversity (Liu et al. 2020).
• The norms built into safety data mostly reflect English-speaking, US-based annotators (Kim et al. 2022; Xu et al. 2020).
• Instruction tuning made models more truthful but did not measurably reduce social bias (Ouyang et al. 2022).

Coverage of tasks and languages is a fifth. Task-oriented research has leaned on English corpora, and cross-domain goals remain much harder to complete (Zhu et al. 2020). Persuasive and medical assistants raise questions of consent, transparency and clinical risk that benchmarks alone cannot settle (Wang et al. 2019; Han et al. 2023).

Related work appears under question answering, language models, speech act theory, information retrieval and prompt engineering.

The library in figures