Elamur Research⌕
☰
home / papers / 2023 / ChartSumm: A Comprehensive Benchmark for Automatic Chart Summarization of Long and Short Summaries

ChartSumm: A Comprehensive Benchmark for Automatic Chart Summarization of Long and Short Summaries

Paper · 2023 · Raian Rahman, Rizvi Hasan, Abdullah Al Farhad et al.

Summary

Turning charts into natural-language descriptions helps visually impaired readers and supports information retrieval. Progress has been held back by a lack of large, well-structured training data. The authors introduce ChartSumm, a benchmark of 84,363 charts, each with metadata (the underlying table) and a summary. The charts come from two sources. Knoema contributes line charts with brief, system-generated descriptions. Statista contributes bar, line and pie charts with longer human-written texts. Missing axis labels were inferred with heuristics and NER, and chart types were assigned with ChartReader. Separate test sets assess short (Knoema) and long (Statista) summary generation.

Baselines were fine-tuned and scored with BLEURT, CIDEr, CS and perplexity: three BART variants and one T5. Models trained on ChartSumm transferred better than those trained on Chart-To-Text, even on the Chart-To-Text test set. Error analysis still revealed several weaknesses:

  • hallucinated content,
  • omitted key values,
  • misread trends.

A Bengali pilot used machine-translated training data and a human-translated test set. It suggests the resource can be extended to other languages.

Key points

  • Size and sources: ChartSumm has 84,363 examples, roughly double Chart-To-Text (44,085), drawn from Knoema (all line charts) and Statista (64.70% bar, 33.76% line, 1.54% pie).
  • Splits: data is divided 80:10:10 per source. Training sets contain 34,503 (Knoema) and 32,985 (Statista) examples, and test sets contain 4,338 and 4,098.
  • Overlap with Chart-To-Text: only 5,338 Statista captions overlap, using a token-match threshold above 90%.
  • Baseline results: BART-Large-CNN and BART-Large-XSUM scored highest on BLEURT, CIDEr and CS, while T5 had the best perplexity. Training on Chart-To-Text cut the best T5 score on the Knoema test set by about 82.48%.
  • Bengali pilot: training data was translated with NLLB, and an mT5 model pre-trained on XL-SUM was fine-tuned. Evaluation used BLEU only, presented as the first chart-summarization study beyond English.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓York UniversityToronto · Canada13 works · 3,075 citations🎓Islamic University of TechnologyDhaka · Bangladesh10 works · 555 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Vision and Pattern Recognition · 2,869 works

🧭 Topics 2Research subjects and languages the work is about.Evaluation & benchmarks · 1,220 worksSummarization · 232 works

Cite

Cite as Raian Rahman, Rizvi Hasan, Abdullah Al Farhad, Md Tahmid Rahman Laskar, Md. Hamjajul Ashmafee and Abu Raihan Mostofa Kamal. 2023. ChartSumm: A Comprehensive Benchmark for Automatic Chart Summarization of Long and Short Summaries. arXiv:2304.13620.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.21428/594757db.0b1f96f6 · arXiv 2304.13620

@misc{rahman2023chartsumm,
  title = {{ChartSumm: A Comprehensive Benchmark for Automatic Chart Summarization of Long and Short Summaries}},
  author = {Raian Rahman, Rizvi Hasan, Abdullah {Al Farhad}, Md Tahmid Rahman Laskar, Md. Hamjajul Ashmafee and Abu Raihan Mostofa Kamal},
  year = {2023},
  doi = {10.21428/594757db.0b1f96f6},
  eprint = {2304.13620},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.21428/594757db.0b1f96f6}
}

References

42 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. David M. Blei, Andrew Y. Ng, and Michael I. Jordan. 2003. Latent dirichlet allocation. J. Mach. Learn. Res., 3(null):993–1022.
  2. Giuseppe Carenini, Cristina Conati, Enamul Hoque, and Ben Steichen. 2013. User task adaptation in multimedia presentations. In UMAP Workshops. Citeseer. find
  3. Chengliang Chai, Guoliang Li, Ju Fan, and Yuyu Luo. 2021. Crowdchart: Crowdsourced data extraction from visualization charts. IEEE Transactions on Knowledge and Data Engineering, 33(11):3537– 3549. find
  4. Marta R Costa-jussà, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, et al. 2022. No language left behind: Scaling human-centered machine translation. arXiv preprint arXiv:2207.04672. find
  5. Komal Dadhich, Siri Chandana Daggubati, and Jaya Sreevalsan-Nair. 2021. Barchartanalyzer: Digitizing images of bar charts. In BarChartAnalyzer. find
  6. Srushti Gajbhiye and Maria Lopes. 2021. Template-based nlg for tabular data using bert. pages 1–5. find
  7. Li Gong, Josep M Crego, and Jean Senellart. 2019. Enhanced transformer model for data-to-text generation. In Proceedings of the 3rd Workshop on Neural Generation and Translation, pages 148–156. find
  8. Tahmid Hasan, Abhik Bhattacharjee, Md. Saiful Islam, Kazi Mubasshir, Yuan-Fang Li, Yong-Bin Kang, M. Sohel Rahman, and Rifat Shahriyar. 2021. XL-sum: Large-scale multilingual abstractive summarization for 44 languages. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pa… find
  9. Enamul Hoque, Parsa Kavehzadeh, and Ahmed Masry. 2022. Chart question answering: State of the art and future directions. In Computer Graphics Forum, volume 41, pages 555–572. Wiley Online Library. find
  10. Ting-Yao Hsu, C Lee Giles, and Ting-Hao Huang. 2021. SciCap: Generating captions for scientific figures. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 3258–3264, Punta Cana, Dominican Republic. Association for Computational Linguistics. find
  11. Yi-Te Hsu, Sarthak Garg, Yi-Hsiu Liao, and Ilya Chatsv-iorkin. 2020. Efficient inference for neural machine translation. In Proceedings of SustaiNLP: Workshop on Simple and Efficient Natural Language Processing, pages 48–53, Online. Association for Computational Linguistics. find
  12. Shankar Kanthara, Rixie Tiffany Ko Leong, Xiang Lin, Ahmed Masry, Megh Thakkar, Enamul Hoque, and Shafiq Joty. 2022. Chart-to-text: A large-scale benchmark for chart summarization. arXiv preprint arXiv:2203.06486.
  13. Shankar Kantharaj, Xuan Long Do, Rixie Tiffany Ko Leong, Jia Qing Tan, Enamul Hoque, and Shafiq Joty. 2022a. Opencqa: Open-ended question answering with charts. arXiv preprint arXiv:2210.06628. find
  14. Shankar Kantharaj, Rixie Tiffany Leong, Xiang Lin, Ahmed Masry, Megh Thakkar, Enamul Hoque, and Shafiq Joty. 2022b. Chart-to-text: A large-scale benchmark for chart summarization. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), page…
  15. Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980.
  16. Md Tahmid Rahman Laskar, Cheng Chen, Xue-yong Fu, and Shashi Bhushan Tn. 2022a. Improving named entity recognition in telephone conversations via effective active learning with human in the loop. In Proceedings of the Fourth Workshop on Data Science with Human-in-the-Loop (Language Advances), pages… find
  17. Md Tahmid Rahman Laskar, Cheng Chen, Jonathan Johnston, Xue-Yong Fu, Shashi Bhushan TN, and Simon Corston-Oliver. 2022b. An auto encoder-based dimensionality reduction technique for efficient entity linking in business phone conversations. In Proceedings of the 45th International ACM SIGIR Conferenc… find
  18. Md Tahmid Rahman Laskar, Cheng Chen, Aliaksandr Martsinovich, Jonathan Johnston, Xue-Yong Fu, Shashi Bhushan Tn, and Simon Corston-Oliver. 2022c. BLINK with Elasticsearch for efficient entity linking in business conversations. In Proceedings of the 2022 Conference of the North American Chapter of th… find
  19. Md Tahmid Rahman Laskar, Enamul Hoque, and Jimmy Huang. 2020a. Query focused abstractive summarization via incorporating query relevance and transfer learning with transformer models. In Advances in Artificial Intelligence: 33rd Canadian Conference on Artificial Intelligence, Canadian AI 2020, Ottaw… find
  20. Md Tahmid Rahman Laskar, Enamul Hoque, and Jimmy Xiangji Huang. 2020b. WSL-DS: Weakly supervised learning with distant supervision for query focused multi-document abstractive summarization. In Proceedings of the 28th International Conference on Computational Linguistics, pages 5647–5654, Barcelona,… find
  21. Md Tahmid Rahman Laskar, Enamul Hoque, and Jimmy Xiangji Huang. 2022d. Domain adaptation with pre-trained transformers for query-focused abstractive text summarization. Computational Linguistics, 48(2):279–320.
  22. Md Tahmid Rahman Laskar, Xiangji Huang, and Enamul Hoque. 2020c. Contextualized embeddings based transformer encoder for sentence similarity modeling in answer selection task. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 5505–5514. find
  23. Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the…
  24. Zhuo Li, Matthew Stagitis, Sandra Carberry, and Kathleen F McCoy. 2013. Towards retrieving relevant information graphics. In Proceedings of the 36th international ACM SIGIR conference on research and development in information retrieval, pages 789–792. find
  25. Ilya Loshchilov and Frank Hutter. 2018. Decoupled weight decay regularization. In International Conference on Learning Representations. find
  26. Junyu Luo, Zekun Li, Jinpeng Wang, and Chin-Yew Lin. 2021. Chartocr: Data extraction from charts images via a deep hybrid framework. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. find
  27. Ahmed Masry, Do Xuan Long, Jia Qing Tan, Shafiq Joty, and Enamul Hoque. 2022. Chartqa: A benchmark for question answering about charts with visual and logical reasoning. arXiv preprint arXiv:2203.10244.
  28. Jason Obeid and Enamul Hoque. 2020. Chart-to-text: Generating natural language descriptions for charts by adapting the transformer model. In Proceedings of the 13th International Conference on Natural Language Generation, pages 138–147, Dublin, Ireland. Association for Computational Linguistics. find
  29. Matt Post. 2018. A call for clarity in reporting BLEU scores. In Proceedings of the Third Conference on Machine Translation: Research Papers, pages 186– 191, Brussels, Belgium. Association for Computational Linguistics.
  30. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9. find
  31. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2019. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv:1910.10683.
  32. Ehud Reiter. 2007. An architecture for data-to-text systems. In Proceedings of the Eleventh European Workshop on Natural Language Generation (ENLG 07), pages 97–104, Saarbrücken, Germany. DFKI GmbH. find
  33. Thibault Sellam, Dipanjan Das, and Ankur Parikh. 2020. BLEURT: Learning robust metrics for text generation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7881–7892, Online. Association for Computational Linguistics.
  34. Jaya Sreevalsan-Nair, Komal Dadhich, and Siri Chandana Daggubati. 2021. Tensor fields for data extraction from chart images: bar charts and scatter plots. In Topological Methods in Data Analysis and Visualization VI. Springer. find
  35. Ramakrishna Vedantam, C. Lawrence Zitnick, and Devi Parikh. 2015. Cider: Consensus-based image description evaluation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR).
  36. Sam Wiseman, Arturs Backurs, and Karl Stratos. 2021. Data-to-text generation by splicing together nearest neighbors. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 4283–4299, Online and Punta Cana, Dominican Republic. Association for Computational Li… find
  37. Sam Wiseman, Stuart M Shieber, and Alexander M Rush. 2017. Challenges in data-to-document generation. arXiv preprint arXiv:1707.08052.
  38. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lho… — Transformers: State-of-the-Art Natural Language Processing
  39. Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2021. mT5: A massively multilingual pre-trained text-to-text transformer. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Ling…
  40. Fangfang Zhou, Yong Zhao, Wenjiang Chen, Yijing Tan, Yaqi Xu, Yi Chen, Chao Liu, and Ying Zhao. 2021. Reverse-engineering bar charts using neural networks. Journal of Visualization, 24(2). find
  41. Jiawen Zhu, Jinye Ran, Roy Ka-wei Lee, Kenny Choo, and Zhi Li. 2021a. Autochart: A dataset for chart-to-text generation task. arXiv preprint arXiv:2108.06897. find
  42. Jiawen Zhu, Jinye Ran, Roy Ka-Wei Lee, Zhi Li, and Kenny Choo. 2021b. AutoChart: A dataset for chart-to-text generation task. In Proceedings of the International Conference on Recent Advances in Natural Language Processing (RANLP 2021), pages 1636– 1644, Held Online. INCOMA Ltd. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/rahman-2023-chartsumm-comprehensive-benchmark-automatic-chart/