📄 Generating an Entailment Corpus from News Headlines

Summary

The Pascal RTE challenge supplied only 567 development pairs. That is too few for statistical models such as the MT-style alignment the MITRE authors wanted to apply to entailment. Their observation was that almost all TRUE RTE pairs are really partial paraphrases, and news headlines relate to their lead paragraphs in much the same way. Headline/lead pairs could therefore serve as a large, free source of training data. They hand-checked over 200 stories from 11 sources and chose Xinhua English (XIE) articles in Gigaword as the most productive source. They then built a fast yes/no/maybe annotation tool and used it to label several hundred documents. From those labels they trained an SVM-light classifier to pull out a cleaner 100,000-pair subset. About three in four of those pairs are true entailments, and a system trained on them ranked among the best at RTE. The paper also examines why human entailment judgments diverge. It argues that datasets should record this disagreement rather than force consensus.

Key points

• In the RTE dev2 set, 131 of 140 TRUE pairs (94%) were partial paraphrases of the text rather than looser entailments.
• MiTAP sources yielded few usable pairs. Op-eds, market summaries, obituaries, news-brief digests and first-person narratives were the main failures.
• About 900 Gigaword documents were annotated, 520 of them XIE. The estimated base rate of "lead entails headline" in XIE is 70%.
• The classifier was SVM-light with Boolean bag-of-words features over the whole article and a linear kernel, reaching 77% leave-one-out accuracy.
  • One round of active learning on 100 near-margin documents added little.
  • The top 100,000 of 679,000 XIE documents reached 74% purity. The 4-point gain later proved not statistically significant.
• Inter-judge agreement on 300 doubly judged documents:
  • 78% under strict labels, against 80% for RTE.
  • 90% when maybe counts as yes, against 83% for the MSR Paraphrase Corpus.
  • 15% of the first batch and 21% of the second stayed unreconciled.

Interesting ideas

• I1. Naturally occurring pairs, such as a headline and its lead, give abundant, almost free entailment examples where one side compresses the other.
• I2. Cheap surface filters remove whole unusable genres before any annotation: first-person pronouns flag narratives, and section types flag digests and obituaries.
• I3. The headline's meaning sometimes depends on the dateline, so a rule allowing place and date to complete the premise was needed.
• I4. Headlines fall into three types: truncated, proposition-less subject headings, and terse but well-formed statements.
• I5. Each pair gets three independent votes, kept as graded labels (TTT/TTF/TFF/FFF), and the disputed span is marked instead of forcing agreement. Systems can then be scored with human variation in mind.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (learn): headline/lead pairs could serve as silver data for checking that the world built from a lead supports the frames parsed from its headline.
• I2 (learn): explainable pre-filters of this kind could screen raw corpora before silver data is built, so that witness parsers are not spent on hopeless documents.
• I3 (world): attaching dateline location and time as default context for events in the text world would let entailment checks use them explicitly.
• I4 (frames): classifying a headline into one of these types before frame extraction would stop the system from inventing who-did-what-to-whom for headings that assert nothing.
• I5 (learn): disagreement among witness parsers could be stored the same way, with the conflicting arcs marked. Evaluation could then separate unanimous cases from contested ones, and gains could be tested for significance as the authors eventually did.