Elamur Research⌕
☰
home / papers / 2021 / What's in the Box? An Analysis of Undesirable Content in the Common Crawl Corpus

What's in the Box? An Analysis of Undesirable Content in the Common Crawl Corpus

Paper · 2021 · Alexandra Luccioni and Joseph Viviano

Summary

Large language models such as the GPT series, BERT and FastText rely heavily on web-scraped training text, yet those corpora are rarely examined. The authors study the Common Crawl, the largest such source. They randomly sampled 1% of the Nov/Dec 2020 snapshot, which after language filtering came to about 115 GB, or roughly 5.8 million pages. They screened this sample for two kinds of problematic material:

  • Hate speech, using three detectors: DELIMIT (BERT-based), HateSonar (logistic regression) and an n-gram list derived from Hate Base.
  • Sexually explicit content, using an n-gram list.

They also tested whether perplexity-based quality filtering, as proposed by Wenzek et al. (2020), removes this material. Each detector flagged a few percent of pages as hateful, but the detectors often caught different pages. Perplexity filtering removed much of the sexual content but little of the hate speech, and perplexity correlated weakly with every detector.

The study shows that common cleaning pipelines leave harmful text in the data that models may learn and reproduce. The authors call for scalable corpus-analysis tools, more careful data collection, and corpus transparency as a basis for model accountability.

Key points

  • Sample: 1% of a 2.6-billion-page Common Crawl snapshot, giving 5,835,339 pages (about 115 GB). The code is released at github.com/josephdviviano/whatsinthebox.
  • Hate speech rates: DELIMIT flagged 5.24% of pages, HateSonar 4.02%, and the Hate Base n-gram method 6.38%. The n-gram method counted a page only if it contained at least 3 matches.
  • Sexual content rates: 2.36% of pages had at least one listed n-gram, 1.36% had 3 or more, and 0.73% had 10 or more.
  • Perplexity filtering: pages were split into three equal quality buckets by perplexity.
    • Pages with 3+ sexual n-grams fell from 3.11% in the low-quality bucket to 0.42% in the high-quality bucket.
    • Hate n-gram hits barely changed, from 8.26% to 6.19%.
  • Future work: the authors propose studying how undesirable data affects downstream models, how well filtering it out works, and output regularization or controlled generation as further remedies.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Université de MontréalMontreal · Canada87 works · 91,597 citations🔬Mila - Quebec Artificial Intelligence InstituteMontreal · Canada65 works · 14,425 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 works

Cite

Cite as Alexandra Luccioni and Joseph Viviano. 2021. What's in the Box? An Analysis of Undesirable Content in the Common Crawl Corpus. Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2021.acl-short.24

@inproceedings{luccioni2021what,
  title = {{What's in the Box? An Analysis of Undesirable Content in the Common Crawl Corpus}},
  author = {Alexandra Luccioni and Joseph Viviano},
  booktitle = {Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers)},
  year = {2021},
  doi = {10.18653/v1/2021.acl-short.24},
  url = {https://doi.org/10.18653/v1/2021.acl-short.24}
}

References

71 references, 30 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Carlini, N., Tramer, F., Wallace, E., Jagielski, M., Herbert-Voss, A., Lee, K., Roberts, A., Brown, T., Song, D., Erlingsson, U., et al. (2020). Extracting training data from large language models. arXiv preprint arXiv:2012.07805.
  2. Abid, A., Farooqi, M., and Zou, J. (2021). Persistent Anti-Muslim Bias in Large Language Models. arXiv preprint arXiv:2101.05783.
  3. Aluru, S. S., Mathew, B., Saha, P., and Mukherjee, A. (2020). Deep learning models for multilingual hate speech detection. arXiv preprint arXiv:2004.06465. — [arXiv:2004.06465](https://arxiv.org/abs/2004.06465) find
  4. Badjatiya, P., Gupta, S., Gupta, M., and Varma, V. (2017). Deep learning for hate speech detection in tweets. In Proceedings of the 26th International Conference on World Wide Web Companion, pages 759–760. find
  5. Basile, V., Bosco, C., Fersini, E., Debora, N., Patti, V., Pardo, F. M. R., Rosso, P., Sanguinetti, M., et al. (2019). Semeval-2019 task 5: Multilingual detection of hate speech against immigrants and women in Twitter. In 13th International Workshop on Semantic Evaluation, pages 54–63. Association f…
  6. Bender, E., Gebru, T., McMillan-Major, A., et al. (2021). On the dangers of stochastic parrots: Can language models be too big. Proceedings of FAccT. find
  7. Bender, E. M. and Friedman, B. (2018). Data statements for natural language processing: Toward mitigating system bias and enabling better science. Transactions of the Association for Computational Linguistics, 6:587–604. find
  8. Bhardwaj, R., Majumder, N., and Poria, S. (2021). Investigating gender bias in BERT. Cognitive Computation, pages 1–11. find
  9. Bolukbasi, T., Chang, K.-W., Zou, J. Y., Saligrama, V., and Kalai, A. T. (2016). Man is to computer programmer as woman is to homemaker? debiasing word embeddings. In Advances in Neural Information Processing Systems, pages 4349–4357.
  10. Bordia, S. and Bowman, S. R. (2019). Identifying and reducing gender bias in word-level language models. arXiv preprint arXiv:1904.03035.
  11. Bowman, S. R., Angeli, G., Potts, C., and Manning, C. D. (2015). A large annotated corpus for learning natural language inference. arXiv preprint arXiv:1508.05326.
  12. Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020). Language models are few-shot learners. arXiv preprint arXiv:2005.14165.
  13. Caliskan, A., Bryson, J. J., and Narayanan, A. (2017). Semantics derived automatically from language corpora contain human-like biases. Science, 356(6334):183–186. find
  14. Caswell, I., Kreutzer, J., Wang, L., Wahab, A., van Esch, D., Ulzii-Orshikh, N., Tapo, A., Subramani, N., Sokolov, A., Sikasote, C., et al. (2021). Quality at a glance: An audit of web-crawled multilingual datasets. arXiv preprint arXiv:2103.12028.
  15. Davidson, T., Warmsley, D., Macy, M., and Weber, I. (2017). Automated hate speech detection and the problem of offensive language. In Proceedings of the International AAAI Conference on Web and Social Media, volume 11. de Lima, L. H. C., Reis, J., Melo, P., Murai, F., and Benevenuto, F. (2021). Char…
  16. Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. (2018). BERT: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805.
  17. Engel, J., Hoffman, M., and Roberts, A. (2017). Latent constraints: Learning to generate conditionally from unconditional generative models. arXiv preprint arXiv:1711.05772. — [arXiv:1711.05772](https://arxiv.org/abs/1711.05772) find
  18. Farrell, T., Fernandez, M., Novotny, J., and Alani, H. (2019). Exploring misogyny across the manosphere in reddit. In Proceedings of the 10th ACM Conference on Web Science, pages 87–96. find
  19. Florez, O. U. (2019). On the unintended social bias of training language generation models with data from local media. arXiv preprint arXiv:1911.00461. — [arXiv:1911.00461](https://arxiv.org/abs/1911.00461) find
  20. Fortuna, P. and Nunes, S. (2018). A survey on automatic detection of hate speech in text. ACM Computing Surveys (CSUR), 51(4):1–30. find
  21. Foubert, J. D., Blanchard, W., Houston, M., and Williams, R. R. (2019). Pornography and sexual violence. In Handbook of Sexual Assault and Sexual Assault Prevention, pages 109–127. Springer. find
  22. Founta, A., Djouvas, C., Chatzakou, D., Leontiadis, I., Blackburn, J., Stringhini, G., Vakali, A., Sirivianos, M., and Kourtellis, N. (2018). Large scale crowdsourcing and characterization of Twitter abusive behavior. In Proceedings of the International AAAI Conference on Web and Social Media, volum… find
  23. Fritz, N., Malic, V., Paul, B., and Zhou, Y. (2020). Worse than objects: The depiction of black women and men and their sexual relationship in pornography. Gender Issues, pages 1–21. find
  24. Gebru, T., Morgenstern, J., Vecchione, B., Vaughan, J. W., Wallach, H., Daumé III, H., and Crawford, K. (2018). Datasheets for datasets. arXiv preprint arXiv:1803.09010. find
  25. Gehman, S., Gururangan, S., Sap, M., Choi, Y., and Smith, N. A. (2020). Realtoxicityprompts: Evaluating neural toxic degeneration in language models. arXiv preprint arXiv:2009.11462.
  26. Mathew, B., Illendula, A., Saha, P., Sarkar, S., Goyal, P., and Mukherjee, A. (2019). Temporal effects of unmoderated hate speech in gab. arXiv preprint arXiv:1909.10966. — [arXiv:1909.10966](https://arxiv.org/abs/1909.10966) find
  27. Gonen, H. and Goldberg, Y. (2019). Lipstick on a pig: Debiasing methods cover up systematic gender biases in word embeddings but do not remove them. arXiv preprint arXiv:1903.03862.
  28. Matic, S., Iordanou, C., Smaragdakis, G., and Laoutaris, N. (2020). Identifying sensitive URLs at web-scale. In Proceedings of the ACM Internet Measurement Conference, pages 619–633. find
  29. Grave, E., Bojanowski, P., Gupta, P., Joulin, A., and Mikolov, T. (2018). Learning word vectors for 157 languages. arXiv preprint arXiv:1802.06893.
  30. May, C., Wang, A., Bordia, S., Bowman, S. R., and Rudinger, R. (2019). On measuring social biases in sentence encoders. arXiv preprint arXiv:1903.10561.
  31. Hammami, M., Chahir, Y., and Chen, L. (2003). Web-guard: Web based adult content detection and filtering system. In Proceedings IEEE/WIC International Conference on Web Intelligence (WI 2003), pages 574–578. IEEE. find
  32. Hashemi, M. (2020). Web page classification: a survey of perspectives, gaps, and future directions. Multimedia Tools and Applications, pages 1–25. find
  33. Ho, W. H. and Watters, P. A. (2004). Statistical and structural approaches to filtering internet pornography. In 2004 IEEE International Conference on Systems, Man and Cybernetics (IEEE Cat. No. 04CH37583), volume 5, pages 4792–4798. IEEE. find
  34. Hube, C. (2017). Bias in Wikipedia. In Proceedings of the 26th International Conference on World Wide Web Companion, pages 717–721. find
  35. Hutchinson, B., Prabhakaran, V., Denton, E., Webster, K., Zhong, Y., and Denuyl, S. (2020). Social biases in NLP models as barriers for persons with disabilities. arXiv preprint arXiv:2005.00813.
  36. Mishra, P., Yannakoudakis, H., and Shutova, E. (2019). Tackling online abuse: A survey of automated abuse detection methods. arXiv preprint arXiv:1908.06024. — [arXiv:1908.06024](https://arxiv.org/abs/1908.06024) find
  37. Nadeem, M., Bethke, A., and Reddy, S. (2020). Stereoset: Measuring stereotypical bias in pretrained language models. arXiv preprint arXiv:2004.09456.
  38. Ortiz Suarez, P. J., Sagot, B., and Romary, L. (2019). Asynchronous pipelines for processing huge corpora on medium to low resource infrastructures. Proceedings of the Workshop on Challenges in the Management of Large Corpora (CMLC-7) 2019. Cardiff, 22nd July 2019, pages 9 – 16, Mannheim. Leibniz-In… find
  39. Paullada, A., Raji, I. D., Bender, E. M., Denton, E., and Hanna, A. (2020). Data and its (dis) contents: A survey of dataset development and use in machine learning research. arXiv preprint arXiv:2012.05345. find
  40. Jo, E. S. and Gebru, T. (2020). Lessons from archives: strategies for collecting sociocultural data in machine learning. In Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency, pages 306–316. find
  41. Polpinij, J., Chotthanom, A., Sibunruang, C., Cham-chong, R., and Puangpronpitag, S. (2006). Content-based text classifiers for pornographic web filtering. In 2006 IEEE International Conference on Systems, Man and Cybernetics, volume 2, pages 1481–1485. IEEE. find
  42. Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., and Amodei, D. (2020). Scaling laws for neural language models. arXiv preprint arXiv:2001.08361.
  43. Radford, A., Narasimhan, K., Salimans, T., and Sutskever, I. (2018). Improving language understanding by generative pre-training. find
  44. Kiritchenko, S. and Mohammad, S. M. (2018). Examining gender and race bias in two hundred sentiment analysis systems. arXiv preprint arXiv:1805.04508.
  45. Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., and Sutskever, I. (2019). Language models are unsupervised multitask learners. OpenAI blog, 1(8):9. find
  46. Kolias, V., Anagnostopoulos, I., and Kayafas, E. (2014). Exploratory analysis of a terabyte scale web corpus. arXiv preprint arXiv:1409.5443. — [arXiv:1409.5443](https://arxiv.org/abs/1409.5443) find
  47. Rajpurkar, P., Jia, R., and Liang, P. (2018). Know what you don’t know: Unanswerable questions for SQuAD. arXiv preprint arXiv:1806.03822.
  48. Manzini, T., Lim, Y. C., Tsvetkov, Y., and Black, A. W. (2019). Black is to criminal as caucasian is to police: Detecting and removing multiclass bias in word embeddings. arXiv preprint arXiv:1904.04047.
  49. Rowley, H. A., Jing, Y., and Baluja, S. (2006). Large scale image-based adult-content filtering. Google Research Paper. find
  50. Massanari, A. (2017). \# gamergate and the fappen-ing: How reddit’s algorithm, governance, and culture support toxic technocultures. New media & society, 19(3):329–346. find
  51. Roziewski, S. and Stokowiec, W. (2016). Language-crawl: A generic tool for building language models upon common-crawl. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16), pages 2789–2793. find
  52. Schick, T., Udupa, S., and Schütze, H. (2021). Self-diagnosis and self-debiasing: A proposal for reducing corpus-based bias in nlp. arXiv preprint arXiv:2103.00453.
  53. Schmidt, A. and Wiegand, M. (2017). A survey on hate speech detection using natural language processing. In Proceedings of the Fifth International workshop on natural language processing for social media, pages 1–10.
  54. Sheng, E., Chang, K.-W., Natarajan, P., and Peng, N. (2019). The woman worked as a babysitter: On biases in language generation. arXiv preprint arXiv:1909.01326.
  55. Shim, J. W., Kwon, M., and Cheng, H.-I. (2015). Analysis of representation of sexuality on women’s and men’s pornographic websites. Social Behavior and Personality: an international journal, 43(1):53–62. find
  56. Solaiman, I., Brundage, M., Clark, J., Askell, A., Herbert-Voss, A., Wu, J., Radford, A., Krueger, G., Kim, J. W., Kreps, S., et al. (2019). Release strategies and the social impacts of language models. arXiv preprint arXiv:1908.09203.
  57. Sun, C., Shrivastava, A., Singh, S., and Gupta, A. (2017). Revisiting unreasonable effectiveness of data in deep learning era. In Proceedings of the IEEE international conference on computer vision, pages 843–852.
  58. Sweeney, C. and Najafian, M. (2019). A transparent framework for evaluating unintended demographic bias in word embeddings. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1662–1667. find
  59. Tadesse, M. M., Lin, H., Xu, B., and Yang, L. (2019). Detection of depression-related posts in reddit social media forum. IEEE Access, 7:44883–44893. find
  60. Vidgen, B. and Derczynski, L. (2020). Directions in abusive language training data: Garbage in, garbage out. arXiv preprint arXiv:2004.01670. find
  61. Wachs, S., Wright, M. F., and Vazsonyi, A. T. (2019). Understanding the overlap between cyberbullying and cyberhate perpetration: Moderating effects of toxic online disinhibition. Criminal Behaviour and Mental Health, 29(3):179–188. find
  62. Wang, Z. J., Choi, D., Xu, S., and Yang, D. (2021). Putting humans in the natural language processing loop: A survey. arXiv preprint arXiv:2103.04044. find
  63. Waseem, Z. and Hovy, D. (2016). Hateful symbols or hateful people? predictive features for hate speech detection on Twitter. In Proceedings of the NAACL student research workshop, pages 88–93.
  64. Wehrmann, J., Simões, G. S., Barros, R. C., and Cav-alcante, V. F. (2018). Adult content detection in videos with convolutional and recurrent neural networks. Neurocomputing, 272:432–438. find
  65. Wenzek, G., Lachaux, M.-A., Conneau, A., Chaudhary, V., Guzmán, F., Joulin, A., and Grave, É. (2020). CCNet: Extracting high quality monolingual datasets from web crawl data. In Proceedings of The 12th Language Resources and Evaluation Conference, pages 4003–4012.
  66. Wolf, M. J., Miller, K. W., and Grodzinsky, F. S. (2017). Why we should have seen that coming: comments on microsoft’s tay “experiment,” and wider implications. The ORBIT Journal, 1(2):1–12. find
  67. Indiviuals using the Internet. https://data.worldbank.org/indicator/ IT.NET.USER.ZS?end=2017&locations=US& start=2015 . Accessed: 2021-01-10. ↗
  68. Zhang, Y., Wang, G., Li, C., Gan, Z., Brockett, C., and Dolan, B. (2020). Pointer: Constrained text generation via insertion-based generative pre-training. arXiv preprint arXiv:2005.00558. — [arXiv:2005.00558](https://arxiv.org/abs/2005.00558) find
  69. Zhao, J., Wang, T., Yatskar, M., Cotterell, R., Ordonez, V., and Chang, K.-W. (2019). Gender bias in contextualized word embeddings. arXiv preprint arXiv:1904.03310.
  70. Zhao, J., Zhou, Y., Li, Z., Wang, W., and Chang, K.-W. (2018). Learning gender-neutral word embeddings. arXiv preprint arXiv:1809.01496.
  71. Zhu, Y., Kiros, R., Zemel, R., Salakhutdinov, R., Urtasun, R., Torralba, A., and Fidler, S. (2015). Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In Proceedings of the IEEE international conference on computer vision, pages 19–27.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/luccioni-2021-what-s-box-analysis-undesirable/