📄 ConvoKit: A Toolkit for the Analysis of Conversations

Summary

General NLP libraries such as NLTK, spaCy and scikit-learn are not designed for dialogue. Conversational data and code are also scattered across incompatible formats and one-off implementations, which hurts reproducibility. The authors present ConvoKit, an open-source Python package that treats conversations as structured social interactions rather than isolated text fragments. Its design rests on two abstractions. The Corpus is a hierarchy of Conversations, Utterances and Speakers, with reply links and user-defined metadata. The Transformer is a uniform operation that modifies a Corpus and follows the scikit-learn interface. Built-in Transformers handle text cleaning, parsing, feature extraction (politeness, hypergraph structure) and analysis (lexical comparison, classification, forecasting conversation trajectories). The package also ships ready-to-use datasets from very different settings: courtrooms, Wikipedia, Reddit and film scripts. The paper reports no benchmark results. Its contribution is a single shared representation and interface for conversational analysis. This makes such research easier to reproduce and opens it to social scientists who lack deep NLP expertise.

Key points

• Four core classes (Corpus, Conversation, Utterance, Speaker) carry unique IDs and a free-form meta table. Thread structure comes from each Utterance's reply_to pointer, which allows tree traversal and browsing of a Speaker's history.
• Transformers expose fit() and transform() like scikit-learn, plus a summarize() method for quick tables or plots. They can be chained in pipelines.
• Built-in Transformers are grouped into three roles:
  • preprocessing: TextCleaner, TextParser;
  • feature extraction: PolitenessStrategies, UserConvoDiversity, HyperConvo;
  • analysis: FightingWords, Classifier, Forecaster.
• Bundled corpora include supreme court transcripts, Wikipedia talk pages, a full Reddit dump and movie dialogs. Guidelines and code are provided for converting other datasets.
• A worked example compares the language of mixed-gender and single-gender movie dialogs with FightingWords. The toolkit is available at convokit.cornell.edu.