Elamur Research⌕
☰
home / papers / 2021 / Trafilatura: A Web Scraping Library and Command-Line Tool for Text Discovery and Extraction

Trafilatura: A Web Scraping Library and Command-Line Tool for Text Discovery and Extraction

Paper · 2021 · Adrien Barbaresi

Summary

Building web corpora requires separating a page's useful text from surrounding noise such as navigation, ads, headers and footers, and finding relevant pages on a website. Existing open-source extractors did not meet the author's needs for accuracy, flexibility and ease of use. This was especially true for building text databases for a lexicographic platform at the ZDL/BBAW. The paper presents Trafilatura, an open-source Python library and command-line tool. It extracts main text, comments and metadata, and also provides components for text discovery and crawling, such as navigating sitemaps and web feeds. The author compares it with a range of other extraction packages on 500 real-world documents annotated at segment level, and also reports a simple, fast baseline. Trafilatura achieves the best balance of precision and recall and clearly outperforms the alternatives. Independent benchmarks confirm this result. For NLP and corpus linguistics, the tool offers a practical way to obtain cleaner web text, which helps with de-duplication, language modelling and close reading.

Key points

  • The benchmark covers 500 documents, with 1487 text and 1496 boilerplate segments.
  • Trafilatura 0.8.2 reached an F-score of 0.912 (precision 0.934, recall 0.890), and its "fast" mode reached 0.896.
  • The next-best tools, news-please (0.808) and readability-lxml (0.804), scored about 0.10 lower in F-score.
  • In the external evaluations:
    • an earlier version topped ScrapingHub's article extraction benchmark among open-source libraries;
    • it also gave the best macro-mean on the multilingual DANIEL corpus (about 1,600 pages in five languages);
    • it performed well for French and Swedish and for building Russian-Turkic parallel corpora.
  • The tool favours text quality over always returning output, so some pages yield no text. Planned work includes precision- and recall-oriented presets, evaluation of comment extraction, and a spider for websites that lack sitemaps or feeds.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

Also on the byline: Berlin-Brandenburg Academy of Sciences and Humanities.

📚 Shelves 7The field of science this work belongs to (OpenAlex fields).Computer Science · 9,086 worksArtificial Intelligence · 8,889 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 worksSocial Sciences · 744 worksInformation Systems · 523 worksGeneral Social Sciences · 87 works

Cite

Cite as Adrien Barbaresi. 2021. Trafilatura: A Web Scraping Library and Command-Line Tool for Text Discovery and Extraction. Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing: System Demonstrations.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2021.acl-demo.15

@inproceedings{barbaresi2021trafilatura,
  title = {{Trafilatura: A Web Scraping Library and Command-Line Tool for Text Discovery and Extraction}},
  author = {Adrien Barbaresi},
  booktitle = {Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing: System Demonstrations},
  year = {2021},
  doi = {10.18653/v1/2021.acl-demo.15},
  url = {https://doi.org/10.18653/v1/2021.acl-demo.15}
}

References

54 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Julian Alarte, Josep Silva, and Salvador Tamarit. 2019. What Web Template Extractor Should I Use?A Benchmarking and Comparison for Five Template Extractors. ACM Transactions on the Web (TWEB), 13(2):1–19. find
  2. Adrien Barbaresi. 2015. Ad hoc and general-purpose corpus construction from web sources. Ph.D. thesis, École Normale Supérieure de Lyon. find
  3. Adrien Barbaresi. 2016. Efficient construction of metadata-enhanced web corpora. In Proceedings of the 10th Web as Corpus Workshop, pages 7–16. Association for Computational Linguistics. find
  4. Marco Brunello. 2009. The creation of free linguistic corpora from the web. In Proceedings of the Fifth Web as Corpus Workshop (WAC5), pages 9–16. Elhuyar Fundazioa. find
  5. Christian Buck, Kenneth Heafield, and Bas Van Ooyen. 2014. N-gram Counts and Language Models from the Common Crawl. In Proceedings of the 9th International Conference on Language Resources and Evaluation (LREC’14). ELRA.
  6. Deng Cai, Shipeng Yu, Ji-Rong Wen, and Wei-Ying Ma. 2003. VIPS: a Vision-based Page Segmentation Algorithm. Technical report, Microsoft Technical Report (MSR-TR-2003-79). find
  7. Howard J. Carey and Milos Manic. 2016. HTML web content extraction using paragraph tags. In 25th International Symposium on Industrial Electronics (ISIE), pages 1099–1105. IEEE. find
  8. Adrien Barbaresi. 2020. htmldate: A Python package to extract publication dates from web pages. Journal of Open Source Software, 5(51):2439. find
  9. Adrien Barbaresi and Gaël Lejeune. 2020. Out-of-the-Box and Into the Ditch? Multilingual Evaluation of Generic Text Extraction Tools. In Proceedings of the 12th Web as Corpus Workshop, pages 5–13. find
  10. Adrien Barbaresi and Kay-Michael Würzner. 2014. For a fistful of blogs: Discovery and comparative benchmarking of republishable German content. In Proceedings of KONVENS 2014, NLP4CMC workshop, pages 2–10. Hildesheim University Press. find
  11. Marco Baroni, Silvia Bernardini, Adriano Ferraresi, and Eros Zanchetta. 2009. The WaCky Wide Web: a collection of very large linguistically processed web-crawled corpora. Language Resources and Evaluation, 43(3):209–226. find
  12. Marco Baroni, Francis Chantree, Adam Kilgarriff, and Serge Sharoff. 2008. Cleaneval: a Competition for Cleaning Web Pages. In Proceedings of the 6th Conference on Language Resources and Evaluation (LREC’08), pages 638–643. ELRA. find
  13. Daniel Bauer, Judith Degen, Xiaoye Deng, Priska Herger, Jan Gasthaus, Eugenie Giesbrecht, Lina Jansen, Christin Kalina, Thorben Kräger, Robert Märtin, Martin Schmidt, Simon Scholler, Johannes Steger, Egon Stemle, and Stefan Evert. 2007. FI-ASCO: Filtering the internet by automatic subtree classifi… find
  14. Vladimı́r Benko. 2016. wo Years of Aranea: Increasing Counts and Tuning the Pipeline. In Proceedings of the 10th International Conference on Language Resources and Evaluation (LREC’16), pages 4245– 4248. ELRA. find
  15. Chris Biemann, Gerhard Heyer, Uwe Quasthoff, and Matthias Richter. 2007. The Leipzig Corpora Collection: Monolingual Corpora of Standard Size. In Proceedings of the Corpus Linguistics Conference. find
  16. Isaac Caswell, Theresa Breiner, Daan van Esch, and Ankur Bapna. 2020. Language ID in the wild: Unexpected challenges on the path to a thousand-language web text corpus. In Proceedings of the 28th International Conference on Computational Linguistics, pages 6588–6608. International Committee on Compu…
  17. Valter Crescenzi, Giansalvatore Mecca, and Paolo Merialdo. 2001. Roadrunner: Towards Automatic Data Extraction From Large Web Sites. In Proceedings of the 27th VLDB Conference, pages 109–118. find
  18. Orphée De Clercq and Maribel Montero Perez. 2010. Data Collection and IPR in Multilingual Parallel Corpora. Dutch Parallel Corpus. In Proceedings of the 7th International Conference on Language Resources and Evaluation (LREC’10), pages 3383– 3388. ELRA. find
  19. Aidan Finn, Nicholas Kushmerick, and Barry Smyth. 2001. Fact or Fiction: Content Classification for Digital Libraries. In Joint DELOS-NSF Workshop: Personalization andRecommender Systems in Digital Libraries. find
  20. Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, et al. 2020. The Pile: An 800GB Dataset of Diverse Text for Language Modeling. arXiv preprint arXiv:2101.00027. find
  21. Alexander Geyken, Adrien Barbaresi, Jrg Didakowski, Bryan Jurish, Frank Wiegand, and Lothar Lemnitzer. 2017. Die Korpusplattform des ”Digitalen Wrter-buchs der deutschen Sprache” (DWDS). Zeitschrift fr germanistische Linguistik, 45(2):327–344. find
  22. Mohammad Ghasemisharif, Peter Snyder, Andrius Aucinas, and Benjamin Livshits. 2019. SpeedReader: Reader Mode Made Fast and Private. In Proceedings of the World Wide Web Conference, pages 526–537. find
  23. Natalie Glance, Matthew Hurst, and Takashi Tomokiyo. 2004. Blogpulse: Automated Trend Discovery for Weblogs. In WWW 2004 Workshop on the Weblogging Ecosystem: Aggregation, Analysis and Dynamics. find
  24. Yan Guo, Huifeng Tang, Linhai Song, Yu Wang, and Guodong Ding. 2010. ECON: an Approach to Extract Content from Web News Page. In Proceedings of 12th International Asia-Pacific Web Conference (APWEB), pages 314–320. IEEE. find
  25. Ivan Habernal, Omnia Zayed, and Iryna Gurevych. 2016. C4Corpus: Multilingual Web-size Corpus with Free License. In Proceedings of the 10th Conference on Language Resources and Evaluation (LREC’16), pages 914–922. find
  26. Felix Hamborg, Norman Meuschke, Corinna Breitinger, and Bela Gipp. 2017. news-please: A Generic News Crawler and Extractor. In Proceedings of the 15th International Symposium of Information Science, pages 218–223. find
  27. Hung-Yu Kao, Shian-Hua Lin, Jan-Ming Ho, and Ming-Syan Chen. 2004. Mining web informative structures and contents based on entropy analysis. IEEE Transactions on Knowledge and Data Engineering, 16(1):41–55. find
  28. Aidar Khusainov, Dzhavdet Suleymanov, Rinat Gilmullin, Alina Minsafina, Lenara Kubedinova, and Nilufar Abdurakhmonova. 2020. First Results of the TurkLang-7 Project: Creating Russian-Turkic Parallel Corpora and MT Systems. In Proceedings of the Computational Models in Language and Speech Workshop (C… find
  29. Adam Kilgarriff. 2007. Googleology is bad science. Computational Linguistics, 33(1):147–151. find
  30. Christian Kohlschütter, Peter Fankhauser, and Wolfgang Nejdl. 2010. Boilerplate detection using shallow text features. In Proceedings of the Third ACM International Conference on Web Search and Data Mining, WSDM 10, pages 441–450. find
  31. Julia Krasselt, Philipp Dressen, Matthias Fluor, Cerstin Mahlow, Klaus Rothenhäusler, and Maren Runte. 2020. Swiss-AL: A multilingual Swiss Web corpus for applied linguistics. In Proceedings of the 12th Conference on Language Resources and Evaluation (LREC’20), pages 4145–4151. ELRA. find
  32. JEP, TALN, RÉCITAL. Volume 4: Démonstrations et résumés d’articles internationaux, pages 46–49. Gaël Lejeune and Lichao Zhu. 2018. A New Proposal for Evaluating Web Page Cleaning Tools. Computación y Sistemas, 22(4). find
  33. Verena Lyding, Egon Stemle, Claudia Borghetti, Marco Brunello, Sara Castagnoli, Felice Dell’Orletta, Henrik Dittmann, Alessandro Lenci, and Vito Pirrelli. 2014. The PAISÁ Corpus of Italian Web Texts. In 9th Web as Corpus Workshop (WaC-9) @ EACL 2014, pages 36–43. European chapter of the Association… find
  34. Gilad Mishne and Natalie Glance. 2006. Leave a Reply: An Analysis of Weblog Comments. In Third Annual Workshop on the Weblogging Ecosystem, WWW 2006. find
  35. Tomoyuki Nanno, Toshiaki Fujiki, Yasuhiro Suzuki, and Manabu Okumura. 2004. Automatically Collecting, Monitoring, and Mining Japanese Weblogs. In Proceedings of the 13th international World Wide Web conference on Alternate track papers & posters, pages 320–321. ACM. find
  36. Christopher Olston and Marc Najork. 2010. Web Crawling. Foundations and Trends in Information Retrieval, 4(3):175–246. find
  37. Jeff Pasternack and Dan Roth. 2009. Extracting article text from the web with maximum subsequence segmentation. In Proceedings of the 18th international conference on World Wide Web, pages 971–980. find
  38. Matthew E. Peters and Dan Lecocq. 2013. Content extraction using diverse feature sets. In Proceedings of the 22nd International Conference on World Wide Web, pages 89–90. find
  39. Jan Pomikálek. 2011. Removing boilerplate and duplicate content from web corpora. Ph.D. thesis, Masaryk University. find
  40. Pir Abdul Rasool Qureshi and Nasrullah Memon. 2012. Hybrid model of content extraction. Journal of Computer and System Sciences, 78(4):1248–1257. find
  41. Kevin P. Scannell. 2007. The Crúbadán Project: Corpus building for under-resourced languages. In Proceedings of the 3rd Web as Corpus Workshop, volume 4, pages 5–15, Louvain-la-Neuve. find
  42. Veronika Laippala, Samuel Rönnqvist, Saara Hell-ström, Juhani Luotolahti, Liina Repo, Anna Salmela, Valtteri Skantsi, and Sampo Pyysalo. 2020. From Web Crawl to Clean Register-Annotated Corpora. In Proceedings of the 12th Web as Corpus Workshop, pages 14–22. find
  43. Roland Schäfer. 2016. CommonCOW: massively huge web corpora from CommonCrawl data and a method to distribute them freely under restrictive EU copyright laws. In Proceedings of the 10th International Conference on Language Resources and Evaluation (LREC’16), pages 4500–4504. ELRA. find
  44. Gaël Lejeune and Adrien Barbaresi. 2020. Bien choisir son outil d’extraction de contenu à partir du Web (Choosing the appropriate tool for Web Content Extraction). In Actes de la 6e conférence conjointe Roland Schäfer. 2017. Accurate and efficient general-purpose boilerplate detection for crawle… find
  45. Roland Schäfer, Adrien Barbaresi, and Felix Bildhauer. 2013. The Good, the Bad, and the Hazy: Design Decisions in Web Corpus Construction. In Proceedings of the 8th Web as Corpus Workshop, pages 7–15. find
  46. Roland Schäfer and Felix Bildhauer. 2012. Building Large Corpora from the Web Using a New Efficient Tool Chain. In Proceedings of the 8th Conference on Language Resources and Evaluation (LREC’12), pages 486–493. ELRA.
  47. Miroslav Spousta, Michal Marek, and Pavel Pecina. 2008. Victor: the Web-Page Cleaning Tool. In 4th Web as Corpus Workshop (WAC-4), pages 12–17. find
  48. Fei Sun, Dandan Song, and Lejian Liao. 2011. DOM-based content extraction via text density. In Proceedings of the 34th international ACM SIGIR conference on Research and development in Information Retrieval, pages 245–254. find
  49. Karane Vieira, Altigran S Da Silva, Nick Pinto, Edleno S De Moura, Joao MB Cavalcanti, and Juliana Freire. 2006. A Fast and Robust Method for Web Page TemplateDetection and Removal. In Proceedings of the 15th ACM International Conference on Information and Knowledge Management, pages 258–267. find
  50. Thijs Vogels, Octavian-Eugen Ganea, and Carsten Eickhoff. 2018. Web2text: Deep structured boilerplate removal. In European Conference on Information Retrieval, pages 167–179. Springer. find
  51. Junfeng Wang, Xiaofei He, Can Wang, Jian Pei, Jiajun Bu, Chun Chen, Ziyu Guan, and Gang Lu. 2009. News Article Extraction with Template-Independent Wrapper. In Proceedings of the WWW 2009, pages 1085–1086. ACM. find
  52. Tim Weninger, William H Hsu, and Jiawei Han. 2010. CETR: content extraction via tag ratios. In Proceedings of the 19th international conference on World Wide Web, pages 971–980. find
  53. Lan Yi, Bing Liu, and Xiaoli Li. 2003. Eliminating noisy information in web pages for data mining. In Proceedings of the 9th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 296–305. find
  54. Cai-Nicolas Ziegler and Michal Skubacz. 2007. Content Extraction from News Pages using Particle Swarm Optimization on Linguistic and Structural Features. In Proceedings of the IEEE/WIC/ACM International Conference on Web Intelligence, pages 242–249. IEEE. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/barbaresi-2021-trafilatura-web-scraping-library-command/