Elamur Research⌕
☰
home / papers / 2010 / A Simple Domain-Independent Probabilistic Approach to Generation

A Simple Domain-Independent Probabilistic Approach to Generation

Summary

The paper addresses data-to-text generation: producing a natural-language description from a world state given as a set of database records. Most earlier systems are tuned to one domain and treat content selection and surface realization as separate stages. Angeli, Liang and Klein instead propose one domain-independent pipeline that frames generation as a hierarchical sequence of small choices. The system first picks a record, then a subset of its fields, and finally a word template. Each choice is made by a discriminatively trained log-linear classifier using generic features that can look at the full history of earlier choices. Training needs only records paired with human text. Word-to-record alignments are induced automatically with the model of Liang et al. (2009), and templates are harvested from the aligned data. The approach was tested on ROBOCUP sportscasting, SUMTIME technical marine forecasts and WEATHERGOV public forecasts. It matched domain-specific state-of-the-art systems on BLEU and in human judgments, and on WEATHERGOV it more than doubled the BLEU of a baseline. This shows a single simple design can carry across quite different domains.

Key points

  • The three datasets differ widely in size and difficulty:
    • ROBOCUP (Chen and Mooney, 2008) has 1919 scenarios, averaging about 2.4 records and 5.7 words.
    • SUMTIME-METEO (Belz's 2008 version) has 469 scenarios, averaging 2.6 records and 16.2 words, and requires no content selection.
    • WEATHERGOV (Liang et al., 2009) has 29,528 scenarios, averaging 36 records and 28.7 words, of which only about 5 records are actually mentioned.
  • On WEATHERGOV the data were split into 25,000 training, 1,000 development and 3,528 test scenarios. The system was evaluated jointly on content selection and realization.
  • On WEATHERGOV the system clearly beat the baseline:
    • BLEU rose from 22.2 to 51.5.
    • F1 rose from 22.1 to 65.4.
    • Human-rated semantic correctness rose from 3.41 to 4.22, which even exceeds the human-written texts (3.85).
    • Fluency was on par with the human texts (4.12 vs 4.14).
  • Plain greedy decoding gave better results than beam search.
  • Most of the gain came from better content selection. The baseline tended to start with the most frequent record type (rainChance), even when no rain was forecast.

Interesting ideas

  • I1. One decision cascade for "what" and "how" to say. Treating record choice, field choice and template choice as stacked classification steps puts content selection and realization under a single learning framework. The same recipe could be tried on Ukrainian data-to-text tasks such as trade statistics or geographic summaries.
  • I2. Features over the whole decision history. Because the model does not require a tree or graph decomposition, features can refer to everything generated so far, for example "has this record type already been mentioned?" The authors found these long-range features essential for coherent output, and the idea transfers to any sequential structured-prediction task.
  • I3. Templates mined from automatically aligned data. An unsupervised alignment model links words to records, and reusable templates with field slots are then extracted from those links. This avoids hand-written grammars and makes porting to a new domain cheap.
  • I4. Base and coarse template variants as features. Each template also gets a more abstract version in which value-like words are replaced by slots. This lets rare templates share statistical strength, a useful backoff trick for small or sparse corpora.
  • I5. Number-tolerant BLEU. The WEATHERGOV evaluation counts numbers within 5 of each other as matching, so rounding in the references does not unfairly penalize correct outputs. Any numeric-heavy generation benchmark could adopt this kind of evaluation fix.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Topics: Machine translation

Cited by

89 papers of our library cite it (of the 140884 whose bibliographies we read); 7 of them have a page here, 82 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{angeli2010simple,
  title = {{A Simple Domain-Independent Probabilistic Approach to Generation}},
  author = {Angeli, Gabor and Liang, Percy and Klein, Dan},
  booktitle = {Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing},
  year = {2010},
  url = {https://aclanthology.org/D10-1049/}
}

References

21 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. R. Barzilay and L. Lee. 2004. Catching the drift: Probabilistic content models, with applications to generation and summarization. In Human Language Technology and North American Association for Computational Linguistics (HLT/NAACL). find our card
  2. A. Belz and E. Kow. 2009. System building cost vs. output quality in data-to-text generation. In European Workshop on Natural Language Generation, pages 16–24. find
  3. A. Belz. 2008. Automatic generation of weather forecast texts using comprehensive probabilistic generation-space models. Natural Language Engineering, 14(4):1–26. find
  4. Aoife Cahill and Josef van Genabith. 2006. Robust pcfg-based generation using automatically acquired LFG approximations. In Association for Computational Linguistics (ACL), pages 1033–1040, Morristown, NJ, USA. Association for Computational Linguistics. find
  5. D. L. Chen and R. J. Mooney. 2008. Learning to sportscast: A test of grounded language acquisition. In International Conference on Machine Learning (ICML), pages 128–135. find
  6. R. Dale, S. Geldof, and J. Prost. 2003. Coral: using natural language generation for navigational assistance. In Australasian computer science conference, pages 35– 44. find
  7. M. E. Foster and M. White. 2004. Techniques for text planning with XSLT. In Workshop on NLP and XML: RDF/RDFS and OWL in Language Technology, pages 1–8. find
  8. N. Green. 2006. Generation of biomedical arguments for lay readers. In International Natural Language Generation Conference, pages 114–121. find
  9. A. Koller and K. Striegnitz. 2002. Generation as dependency parsing. In Association for Computational Linguistics (ACL), pages 17–24. find
  10. P. Liang, M. I. Jordan, and D. Klein. 2009. Learning semantic correspondences with less supervision. In Association for Computational Linguistics and International Joint Conference on Natural Language Processing (ACL-IJCNLP). find our card
  11. D. C. Liu and J. Nocedal. 1989. On the limited memory method for large scale optimization. Mathematical Programming B, 45(3):503–528. find
  12. W. Lu, H. T. Ng, and W. S. Lee. 2009. Natural language generation with tree conditional random fields. In Empirical Methods in Natural Language Processing (EMNLP), pages 400–409. find
  13. Hiroko Nakanishi, Yusuke Miyao, and Jun’ichi Tsujii. 2005. Probabilistic models for disambiguation of an HPSG-based chart generator. In Parsing ’05: Proceedings of the Ninth International Workshop on Parsing Technology, pages 93–102, Morristown, NJ, USA. Association for Computational Linguistics. find
  14. K. Papineni, S. Roukos, T. Ward, and W. Zhu. 2002. BLEU: A method for automatic evaluation of machine translation. In Association for Computational Linguistics (ACL). find our card
  15. A. Ratnaparkhi. 1998. Maximum entropy models for natural language ambiguity resolution. Ph.D. thesis, University of Pennsylvania. find
  16. A. Ratnaparkhi. 2002. Trainable approaches to surface natural language generation and their application to conversational dialog systems. Computer, Speech & Language, 16:435–455. find
  17. E. Reiter, S. Sripada, J. Hunter, J. Yu, and I. Davy. 2005. Choosing words in computer-generated weather forecasts. Artificial Intelligence, 167:137–169. find
  18. R. Soricut and D. Marcu. 2006. Stochastic language generation using WIDL-expressions and its application in machine translation and summarization. In Association for Computational Linguistics (ACL), pages 1105–1112. find
  19. R. Turner, Y. Sripada, and E. Reiter. 2009. Generating approximate geographic descriptions. In European Workshop on Natural Language Generation, pages 42–49. find
  20. Michael White, Rajakrishnan Rajkumar, and Scott Martin. 2007. Towards broad coverage surface realization with CCG. In In Proceedings of the Workshop on Using Corpora for NLG: Language Generation and Machine Translation (UCNLG+MT). find our card
  21. Y. W. Wong and R. J. Mooney. 2007. Learning synchronous grammars for semantic parsing with lambda calculus. In Association for Computational Linguistics (ACL), pages 960–967. find our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/angeli-2010-simple-domain-independent-probabilistic-approach/