Elamur Research⌕
☰
home / papers / 1998 / Describing Complex Charts in Natural Language: A Caption Generation System

Describing Complex Charts in Natural Language: A Caption Generation System

Paper · 1998 · Vibhu O. Mittal, Johanna D. Moore, Giuseppe Carenini et al.

Summary

Automatic graphics systems can produce novel charts that combine many data attributes. Such charts are often hard to read without an explanation, so these systems either stick to familiar chart types or risk confusing users. Mittal, Moore, Carenini and Roth address this by building a system that writes explanatory natural-language captions for graphics designed by SAGE, an automatic presentation system.

The pipeline has three stages:

  • Content planning: the LONGBOW discourse planner selects and organizes caption content, using caption-specific plan operators. These draw on a model of the graphic's structure, its mapping to the underlying data, a measure of how perceptually complex each graphical element is, and the structure of the data itself.
  • Microplanning: separate modules handle clause ordering (with a new centering-based method for ordering within clauses), aggregation and referring expressions.
  • Realization: FUF/SURGE produces the English text.

The system covers 2-D charts and maps, illustrated with Pittsburgh housing-sales data. Apart from the lexicon, it is domain-independent and needs no rich semantic model of the domain. This makes it easy to port to new domains, though odd phrasings can appear when the database schema differs from how people normally talk about the data.

Key points

  • The architecture links SAGE to a generation chain of the LONGBOW text planner, a microplanner and the FUF/SURGE realizer. The authors deliberately chose the simplest adequate technique for each subtask.
  • The system needs three knowledge sources: graphical syntax (relations among graphical objects), graphical semantics (how data attributes map to visual properties), and a complexity mechanism that decides which elements need explaining.
  • The motivating example is a SAGE re-creation of Minard's 1861 chart of Napoleon's 1812 march, which encodes seven variables.
  • Porting the system to a new domain requires only a lexicon of new terms. Captions are built from SAGE's data characterization plus a minimal lexical representation.
  • The system has two main limitations:
    • It cannot coordinate captions with graphical keys or annotations, because its speech-act language allows no two-way exchange with SAGE.
    • It explains how a graphic encodes data, not which data matter for the user's task; the authors name that as future work.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Carnegie Mellon UniversityPittsburgh · United States598 works · 191,225 citations🎓University of PittsburghPittsburgh · United States65 works · 23,401 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksComputer Vision and Pattern Recognition · 3,486 works

Cite

Cite as Vibhu O. Mittal, Johanna D. Moore, Giuseppe Carenini and Steven Roth. 1998. Describing Complex Charts in Natural Language: A Caption Generation System. Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work.

@article{mittal1998describing,
  title = {{Describing Complex Charts in Natural Language: A Caption Generation System}},
  author = {Vibhu O. Mittal, Johanna D. Moore, Giuseppe Carenini and Steven Roth},
  journal = {Computational Linguistics},
  year = {1998},
  url = {https://aclanthology.org/J98-3004/}
}

References

49 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Andrg, Elisabeth and Thomas Rist. 1994. Referring to world objects with text and pictures. In Proceedings of COLING-94, pages 530-534. find
  2. Appelt, Douglas. 1985. Planning English Sentences. Studies in Natural Language Processing. Cambridge University Press. find
  3. Beattie, Vivien and Michael John Jones. 1994. Information design and manipulation: Financial graphs in corporate annual reports. Information Design Journal, 7(3):211-226. find
  4. Casner, Steven. 1991. A task-analytic approach to the automated design of graphic presentations. ACM Transactions on Graphics, 10(2):111-151, April. find
  5. Cleveland, William S, and Robert McGill. 1987. Graphical perception: The visual decoding of quantitative information on graphical displays of data. Journal of the Royal Statistical Society, 150 (Series A, Part 3):192-229. find
  6. Dale, Robert. 1992. Generating Referring Expressions. ACL-MIT Series in Natural Language Processing. The MIT Press. find
  7. Dale, Robert and Ehud Reiter. 1995. Computational interpretations of the Gricean maxims in the generation of referring expressions. Cognitive Science, 19:233–263.
  8. Dalianis, Hercules. 1996. Concise Natural Language Generation from Formal Specifications. Ph.D. thesis, Royal Institute of Technology/Stockholm University, June. find
  9. Derr, Marcia A. and Kathleen R. McKeown. 1984. Using focus to generate complex and simple sentences. In Proceedings off the Tenth International Conference on Computational Linguistics, COLING, pages 319-326. find
  10. Elhadad, Michael. 1992. Using Argumentation to Control Lexical Choice: A Functional Unification Implementation. Ph.D. thesis, Columbia University, New York, NY. (The SURGE grammar for FUF is available via anonymous FTP at: tip: / / www.cs.columbia.edu / fuf). find
  11. Elhadad, Michael and Jacques Robin. 1992. Controlling content realization with functional unification grammars. In R. Dale, E. Hovy, D. Rosner, and O. Stock, editors, Proceedings of the Sixth International Workshop on Natural Language Generation. find
  12. Fasciano, Massimo. 1996. Gdndration integrde de textes et de graphiques statistiques. Ph.D. thesis, University of Montreal, Montreal, Canada. find
  13. Fasciano, Massimo and Guy Lapalme. 1996. PostGraphe: A system for the generation of statistical graphics and text. In Donia Scott, editor, Proceedings of the Eighth International Workshop on Natural Language Generation, Brighton, U.K. find
  14. Feiner, Steven K. and Kathleen R. McKeown. 1991. Automating the generation of coordinated multi-media explanations. Computer, 24(10):33-42, October. find
  15. Gordon, Peter C., Barbara J. Grosz, and Laura A. Gilliom. 1993. Pronouns, names, and the centering of discourse. Cognitive Science, 17(3):311-349. find
  16. Grosz, Barbara J. 1977. The representation and use of focus in dialogue understanding. Technical Report 151, SRI International, Menlo Park, CA. find
  17. Grosz, Barbara J., Aravind K. Joshi, and Scott Weinstein. 1995. Centering: A framework for modelling the local coherence of discourse. Computational Linguistics, 21(2):203-226. find
  18. Grosz, Barbara J. and Candace Sidner. 1993. Lost intuitions and forgotten intentions. In Workshop on Centering Theory in Naturally-Occurring Discourse. find
  19. Hegarty, Mary and Marcel A. Just. 1993. Constructing mental models of machines from text and diagrams. Journal of Memory & Language, 32(6):717-742, December. find
  20. Hov N E. H. and K. F. McCoy. 1989. Focusing your RST: A step toward generating coherent multisentential text. In Proceedings of the Conference of the Cognitive Science Society, Aim Arbor, MI. find
  21. Hovy, Eduard and Marie Meteer, editors. 1990. Proceedings of the AAAI Workshop on Evaluating Natural Language Generators. AAAI, Boston, MA. find
  22. Huang, Xiaorong and Armin Fiedler. 1996. Paraphrasing and aggregating argumentative text using text structure. In Proceedings of the 8th International Workshop on Natural Language Generation. find
  23. Kerpedjiev, Stephan. 1992. Automatic generation of multimodal weather reports from datasets. In Proceedings of the 3rd International Conference on Applied Natural Language Processing, pages 48-54, Trento, Italy. find
  24. Kerpedjiev, Stephan, Giuseppe Carenini, Steven F. Roth, and Johanna D. Moore. 1997. Integrating planning and task-based design for multimedia presentation. In Proceedings of the Second International Conference on Intelligent User Interfaces (IUI "97), pages 145-152, Orlando, FL, January. find
  25. Large, Andrew, Jamshid BeheshG Alain Breuleux, and Andre Renaud. 1995. Multimedia and comprehension: The relationship among text, animation, and captions. Journal of the American Society for Information Science, 46(5):340-347, June. find
  26. Mackinlay, Jock D. 1986. Automating the design of graphical presentations of relational information. ACM Transactions on Graphics, 5(2):110-141, April. find
  27. Marks, Joe W. 1991. Automating the Design of Network Diagrams. Ph.D. thesis, Harvard University, Cambridge, MA. find
  28. Marks, Joe W. and Ehud Reiter. 1990. Avoiding unwanted conversational implicatures in text and graphics. In Proceedings of the Eighth National Conference on Artificial Intelligence (AAAI-90), pages 450-456, Boston, MA. AAAI, AAAI/MIT Press. find
  29. Maybury, Mark T. 1991. Topical' temporal, and spatial constraints on linguistic realization. Computational Intelligence, 7-4:266-275. find
  30. McKeown, Kathleen R. 1985. Text Generation: Using Discourse Strategies and Focus Constraints to Generate Natural Language Text. Cambridge University Press, Cambridge. find
  31. McKeown, Kathleen, Steve Feiner, Jacques Robin, Dor6e Seligmann, and Michael Tanenblatt. 1992. Generating cross-references for multimedia explanation. In Proceedings of Tenth National Conference on Artificial Intelligence, San Jose, CA, July 12-17. find
  32. Mittal, Vibhu O. and C6cile L. Paris. 1992. Generating object descriptions which integrate both text and examples. In Proceedings of the Ninth Canadian Artificial Intelligence Conference (AI/GI/V192), pages 1-8. Canadian Society for the Computational Studies of Intelligence (CSCSI), Morgan Kaufmann… find
  33. Mittal, Vibhu O. and C6cile L. Paris. 1993. Automatic documentation generation: The interaction between text and examples. In Proceedings of the Thirteenth International Joint Conference on Artificial Intelligence, pages 1158-1163, Chambery, France. IJCAI. find
  34. Nugent, Gwen C. 1983. Deaf students' learning from captioned instruction: The relationship between the visual and caption display. Journal of Special Education, 17(2):227-234. find
  35. Reiter, Ehud. 1990. The computational complexity of avoiding conversational implicatures. In Proceedings of the 28th Annual Meeting, pages 97-104. Association for Computational Linguistics.
  36. Roth, Steven F. and William E. Hefley. 1993. Intelligent multimedia presentation systems: Research and principles. In Intelligent Multimedia Interfaces. AAAI/MIT Press, Menlo Park CA, pages 13-59. find
  37. Roth, Steven F., John Kolojejchick, Joe Mattis, and Jade Goldstein. 1994. Interactive graphic design using automatic presentation knowledge. In Proceedings of CHI'94: Human Factors in Computing Systems, Boston, MA. ACM / SIGCHI. find
  38. Roth, Steven F. and Joe Mattis. 1990. Data characterization for intelligent graphics presentation. In Proceedings of the 1990 Conference on Human Factors in Computing Systems, pages 193-200, New Orleans, LA. ACM/SIGCHI. find
  39. Schmid, Calvin F. 1983. Statistical Graphics: Design Principles and Practices. John Wiley and Sons. find
  40. Shah, Priti. 1995. Cognitive Processes in Graph Comprehension. Ph.D. thesis, Carnegie Mellon University, Pittsburgh, PA. find
  41. Shaw, James. 1995. Conciseness through aggregation in text generation. In Proceedings of the 33rd Annual Meeting (Student Session), pages 329-331. Association of Computational Linguistics. find
  42. Sidner, Candace L. 1979. Towards a Computation Theory of Definite Anaphora Comprehension in English Discourse. Ph.D. thesis, MIT, Cambridge, MA. find
  43. Tufte, Edward R. 1983. The Visual Display of Quantitative Information. Graphics Press, Cheshire, Conn. find
  44. Wahlster, Wolfgang, Elisabeth Andr6, W. Finkler, H. J. Profitlich, and Thomas Rist. 1993. Plan-based integration of natural-language and graphics generation. Artificial Intelligence, 63(12):387-427, October. find
  45. Walker, Marilyn A. 1993. Initial contexts and shifting centers. In Proceedings of the Workshop on Centering Theory in Naturally-Occurring Discourse, Columbus, OH. ACL. find
  46. Walker, Marilyn, Masayo Iida, and Sharin Cote. 1994. Japanese discourse and the process of centering. Computational Linguistics, 20(2):193-232.
  47. Young, R. Michael. 1997. The LongBow Discourse Planner. Available at URL http://pogo.isp.pitt.edu/young/longbow/longbow.html. ↗
  48. Young, R. Michael and Johanna D. Moore. 1994. DPOCL: A principled approach to discourse planning. In Proceedings of the Seventh International Workshop on Natural Language Generation, pages 13-20, Kennebunkport, ME, July. find
  49. Young, R. Michael, Martha E. Pollack, and Johanna D. Moore. 1994. Decomposition and causality in partial-order planning. In Proceedings of the Second International Conference on AI and Planning Systems, pages 188-193, Chicago, July. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/mittal-1998-describing-complex-charts-natural-language/