📄 Abstractive Summarization of Reddit Posts with Multi-level Memory Networks

Summary

Most summarization corpora come from news or other formal writing. In such text the key content usually sits near the beginning, and good summary sentences often already appear in the source. Systems can therefore score well by copying from predictable positions instead of truly abstracting. To counter this bias, the authors built Reddit TIFU, a corpus of informal personal stories crawled from the TIFU subreddit. That subreddit's rules require a descriptive title and a closing TL;DR. The post body serves as the input, the title as a short summary and the TL;DR as a long one, all written by the same user. The authors also propose multi-level memory networks (MMN). MMN replaces the usual RNN encoder with dilated convolutional memory that represents the text at word, sentence, paragraph and document scale. It also uses normalized gated tanh units and decodes without RNNs. MMN beats strong seq2seq and convolutional baselines on Reddit TIFU, Newsroom-Abs and XSum, and human judges generally preferred its outputs. The work provides a less extractive benchmark and evidence that memory-based encoders help with long, unstructured input.

Key points

• Reddit TIFU contains 122,933 post–summary pairs collected from January 2013 to March 2018. It is split into TIFU-short (79,949 pairs, titles averaging about 9 words) and TIFU-long (42,984 pairs, TL;DRs averaging about 23 words). Source posts average roughly 340–430 words.
• Gold-summary bigrams are spread across the whole source text rather than concentrated at the start. Even the Ext-Oracle extractive upper bound performs poorly, and the gap between abstractive and extractive methods is wider than on news datasets.
• MMN's novelties over earlier memory networks:
  • a multi-level memory structure;
  • a dilated convolutional memory write that links adjacent memory cells;
  • normalized gated tanh units to limit covariate shift;
  • an RNN-free decoder.
• On Newsroom-Abs, MMN reaches ROUGE-1/2/L of 17.5/4.7/14.2, compared with 14.7/2.2/11.4 for PG. On XSum it scores 32.0/12.1/26.0, slightly above the two-stage T-ConvS2S (31.9/11.5/25.8).
• In AMT tests (100 test samples, three turkers each), MMN was mostly preferred over s2s-att, PG, SEASS and DRGD but still lost to the gold summaries. Code and data are released at vision.snu.ac.kr/projects/reddit-tifu.