📄 Collective Content Selection for Concept-to-Text Generation

Summary

Concept-to-text generation systems must decide which of many database records belong in the output text. This content selection step has traditionally been hand-written per domain, which is costly and hard to scale. Barzilay and Lapata learn selection rules automatically from a parallel corpus of American football (NFL) game summaries and the matching statistics database. Training labels come from automatically aligning summary text with database entries. Rather than judging each entry independently, the authors cast selection as collective classification. Each entry carries its own preference to be kept or dropped, and links between related entries penalise giving them different labels. Link types are derived automatically from database relations. The resulting cost function is minimised exactly and globally with a min-cut algorithm. The collective model reaches an F-score of 60.15%, roughly 10 points above a context-agnostic classifier. It also compares favourably with earlier work done in a simpler biographical domain. The study shows that coherence constraints between related facts help data-driven content selection, even in a large database with many interconnected records.

Key points

• Data: 468 NFL game summaries paired with database entries, giving 105,792 instances.
  • 68 summaries were held out for testing.
  • 10 summaries (1,930 instances) were reserved for development.
• Automatic text–database alignment, checked against a gold standard, reached 94.0% precision and 90.4% recall.
• The collective classifier scored an F-score of 60.15%, a 10.4% F-score gain over the standard classifier (the same model with link scores set to zero). Paired t-tests showed significant gains in both precision and recall over that classifier and over a per-type majority baseline.
• For comparison, Duboue and McKeown (2003) reported a best F-score of 51.00% on simpler biographical summaries.
• The model favours recall and tends to overgenerate, leaving later pipeline components or a domain expert to filter its output.

Interesting ideas

• I1. Choose items jointly, not one at a time. Scoring each candidate alone cannot guarantee that the chosen set fits together. Adding pairwise penalties for splitting related items carries over to extractive summarisation, choosing facts for data-to-text reports, or picking related records in trade or geographic datasets.
• I2. Use min-cut for exact global inference. Individual preferences plus pairwise agreement terms form a two-class graph-cut problem, which is solved optimally and efficiently. This is a cheap alternative to approximate inference whenever constraints are pairwise and two-way.
• I3. Get links from the database schema. Candidate dependencies come from relational structure, such as entries sharing the same quarter, the same player, or the same action. This replaces heavy hand-crafted feature engineering, and the same approach works for any structured source with joinable attributes.
• I4. Build training labels by alignment. Entries that are verbalised in human-written texts count as positives and the rest as negatives. This yields supervision with no manual labelling, and checking the alignment against a small gold sample shows how noisy the labels are.
• I5. Make the baseline a switch inside the full model. Setting link weights to zero recovers the context-free classifier. The comparison therefore isolates exactly what the contextual constraints contribute, which is a clean ablation design worth reusing.