ParlAI: A Dialog Research Software Platform

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/D17-2014.pdf

Summary

Dialog research has been split across many datasets with incompatible formats. This encourages models tuned to one benchmark's quirks, such as span prediction in SQuAD or supporting facts in bAbI, rather than skills that transfer. ParlAI, an open-source Python platform from Facebook AI Research, addresses this by placing every task behind one shared interface. Its design rests on three abstractions: worlds, agents and teachers. These exchange a single extensible Python dict carrying text, labels, candidate answers, rewards, episode boundaries and, optionally, images. Because every dataset uses this format, training one model on several tasks is as easy as training on one. The first release bundles over 20 tasks, automatic dataset download, batch and hogwild training, and PyTorch and Lua Torch examples. It also integrates Amazon Mechanical Turk for data gathering, human evaluation and online learning. As a demonstration, the authors train DrQA jointly on SQuAD and bAbI, a pairing they say had not been reported before. ParlAI matters because it lowers the cost of reproducible, multitask and human-in-the-loop dialog experiments.

Key points

• Tasks. The first release covers 20+ tasks, including SQuAD, bAbI tasks, MCTest, WikiQA, QACNN, QADailyMail, CBT, bAbI Dialog, Ubuntu, OpenSubtitles and VQA. They are grouped into five categories, among them question answering, sentence completion (cloze) and goal-oriented dialog.
• Bundled agents. These include DrQA (an attentive LSTM in PyTorch) and an end-to-end memory network (memnn, in Lua Torch). seq2seq is also mentioned. ZeroMQ connects ParlAI to non-Python code.
• Core library. It provides world classes for different settings: DialogPartnerWorld (two speakers), MultiAgentDialogWorld (more than two), BatchWorld (batch training) and HogwildWorld (multithreaded training). Model outputs are scored by the metrics.py module, which reports F1, exact-match accuracy and ranking-based measures.
• MTurk examples. Two are included: qa collector, which gathers question–answer pairs from a context paragraph, and model evaluator, which collects human ratings of a bot. A sandbox mode allows testing before paid launches.
• DrQA results. On SQuAD (dev, Exact Match), single-task training scores 66.4 and multitask 63.4. On the bAbI 10k subset, where answers appear verbatim in the text:
  • Task 2 drops from 98.1 to 54.3 under multitask training.
  • Task 3 rises from 45.4 to 58.1.
  • Several tasks reach 100 in both settings.