Elamur Research⌕
☰
home / papers / 2021 / TableSense: Spreadsheet Table Detection with Convolutional Neural Networks

TableSense: Spreadsheet Table Detection with Convolutional Neural Networks

Paper · 2021 · Haoyu Dong, Shijie Liu, Shi Han et al.

Summary

Spreadsheet sheets often hold several tables. Their layouts are made for human readers, with blank separator rows, embedded heterogeneous blocks, missing values and tightly packed neighbouring tables. These features make table boundaries hard to find automatically, and the region-growth heuristics used in commercial spreadsheet software often fail on them. TableSense treats a sheet as a grid of cells, much like an image's pixel grid, and adapts convolutional object detection to it.

The framework has three parts:

  • Cell featurization: a scheme that encodes the rich information each cell carries.
  • Precise boundary module: a precise bounding-box regression (PBR) module, needed because missing even one row or column makes a detected table wrong.
  • Active learning: uncertainty-guided sampling that chose which web-crawled sheets to label next.

The resulting labeled set has 22,176 tables on 10,220 sheets. On a held-out test set, TableSense reaches 91.3% recall and 86.5% precision under a strict boundary-error criterion. This clearly beats region-growth and adapted vision detectors such as Mask R-CNN. Reliable table detection is a prerequisite for automated spreadsheet analytics in the cloud and on mobile devices.

Key points

  • New metric: Error-of-Boundary (EoB) is the largest offset, in cells, between any side of the predicted box and the ground-truth box. Results are reported at EoB-2 (tolerance of two cells) and EoB-0 (exact match), because IoU > 0.5 is too lenient for tables.
  • Data: WebSheet is a corpus of 4,290,022 crawled sheets. WebSheet10k (10,220 English sheets) was labeled over 7 active-learning iterations. WebSheet400 (400 sheets, 795 tables) is the test set. Table annotations for VEnron2, VEUSUS and VFUSE are released at github.com/microsoft/TableSense.
  • Baseline comparison: Excel-style region-growth scores only 58.5% recall and 55.2% precision at EoB-2. Mask R-CNN, YOLO-v3 and Faster R-CNN were compared using the same cell features.
  • Ablation: removing PBR lowers EoB-2 recall to 78.4% and precision to 68.1%.
  • Uncertainty metric: after the 7th iteration, it separated sheets with detection errors from correctly detected sheets with about 86% accuracy for both groups.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢MicrosoftRedmond · United States403 works · 162,838 citations🎓Beihang UniversityBeijing · China89 works · 9,227 citations

 Fields, topics and concepts

📚 Shelves 3The field of science this work belongs to (OpenAlex fields).Computer Science · 19,503 worksComputer Vision and Pattern Recognition · 3,621 worksSoftware · 150 works

Cite

Cite as Haoyu Dong, Shijie Liu, Shi Han, Zhouyu Fu and Dongmei Zhang. 2021. TableSense: Spreadsheet Table Detection with Convolutional Neural Networks. arXiv:2106.13500.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.1609/aaai.v33i01.330169 · arXiv 2106.13500

@misc{dong2021tablesense,
  title = {{TableSense: Spreadsheet Table Detection with Convolutional Neural Networks}},
  author = {Haoyu Dong, Shijie Liu, Shi Han, Zhouyu Fu and Dongmei Zhang},
  year = {2021},
  doi = {10.1609/aaai.v33i01.330169},
  eprint = {2106.13500},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.1609/aaai.v33i01.330169}
}

References

22 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Abadi, M.; Barham, P.; Chen, J.; Chen, Z.; Davis, A.; Dean, J.; Devin, M.; Ghemawat, S.; Irving, G.; Isard, M.; et al. 2016. Tensorflow: a system for large-scale machine learning. In OSDI, volume 16, 265–283.
  2. Doush, I. A., and Pontelli, E. 2010. Detecting and recognizing tables in spreadsheets. In Proceedings of the 9th IAPR International Workshop on Document Analysis Systems, 471–478. ACM. find
  3. Fang, J.; Mitra, P.; Tang, Z.; and Giles, C. L. 2012. Table header detection and classification. In AAAI, 599– 605. find
  4. Gatos, B.; Danatsas, D.; Pratikakis, I.; and Perantonis, S. J. 2005. Automatic table detection in document images. In International Conference on Pattern Recognition and Image Analysis, 609–618. Springer. find
  5. Girshick, R.; Donahue, J.; Darrell, T.; and Malik, J. 2014. Rich feature hierarchies for accurate object detection and semantic segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, 580–587.
  6. Girshick, R. 2015. Fast r-cnn. In Proceedings of the IEEE international conference on computer vision, 1440– 1448. find
  7. He, K.; Zhang, X.; Ren, S.; and Sun, J. 2016. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, 770–778.
  8. He, K.; Gkioxari, G.; Dollár, P.; and Girshick, R. 2017. Mask r-cnn. In Computer Vision (ICCV), 2017 IEEE International Conference on, 2980–2988. IEEE. find
  9. Hosang, J.; Benenson, R.; Dollár, P.; and Schiele, B. 2016. What makes for effective detection proposals? IEEE transactions on pattern analysis and machine intelligence 38(4):814–830. find
  10. Hu, J.; Kashi, R. S.; Lopresti, D. P.; and Wilfong, G. 1999. Medium-independent table detection. In Document Recognition and Retrieval VII, volume 3967, 291–303. International Society for Optics and Photonics. find
  11. Krishnakumar, A. 2007. Active learning literature survey. Technical report, Technical Report, University of California, Santa Cruz. find
  12. Krizhevsky, A.; Sutskever, I.; and Hinton, G. E. 2012. Imagenet classification with deep convolutional neural networks. In Advances in neural information processing systems, 1097–1105.
  13. LeCun, Y.; Boser, B.; Denker, J. S.; Henderson, D.; Howard, R. E.; Hubbard, W.; and Jackel, L. D. 1989. Backpropagation applied to handwritten zip code recognition. Neural computation 1(4):541–551. find
  14. Liu, Y.; Bai, K.; Mitra, P.; and Giles, C. L. 2007. Tableseer: automatic table metadata extraction and searching in digital libraries. In Proceedings of the 7th ACM/IEEE-CS joint conference on Digital libraries, 91–100. ACM. find
  15. Liu, Y.; Mitra, P.; and Giles, C. L. 2008. Identifying table boundaries in digital documents via sparse line detection. In Proceedings of the 17th ACM conference on Information and knowledge management, 1311–1320. ACM. find
  16. Ren, S.; He, K.; Girshick, R.; and Sun, J. 2015. Faster r-cnn: Towards real-time object detection with region proposal networks. In Advances in neural information processing systems, 91–99.
  17. Shafait, F., and Smith, R. 2010. Table detection in heterogeneous documents. In Proceedings of the 9th IAPR International Workshop on Document Analysis Systems, 65– 72. ACM. find
  18. Uijlings, J. R.; Van De Sande, K. E.; Gevers, T.; and Smeulders, A. W. 2013. Selective search for object recognition. International journal of computer vision 104(2):154–171. find
  19. Wang, Y., and Hu, J. 2002. A machine learning based approach for table detection on the web. In Proceedings of the 11th international conference on World Wide Web, 242–250. ACM. find
  20. Wang, J.; Wang, H.; Wang, Z.; and Zhu, K. Q. 2012. Understanding tables on the web. In International Conference on Conceptual Modeling, 141–155. Springer. find
  21. Zhai, Y., and Liu, B. 2005. Web data extraction based on partial tree alignment. In Proceedings of the 14th international conference on World Wide Web, 76–85. ACM. find
  22. Zuyev, K. 1997. Table image segmentation. In Document Analysis and Recognition, 1997., Proceedings of the Fourth International Conference on, volume 2, 705–708. IEEE. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/dong-2021-tablesense-spreadsheet-table-detection-convolutional/