Elamur Research⌕
☰
home / papers / 2022 / A New Generation of Perspective API: Efficient Multilingual Character-level Transformers

A New Generation of Perspective API: Efficient Multilingual Character-level Transformers

Paper · 2022 · Alyssa Lees, Vinh Q. Tran, Yi Tay et al.

Summary

Perspective API, Jigsaw's toxicity-scoring service, needs a single classifier that works across many languages. That classifier must also cope with slang, misspellings, emojis and deliberately obfuscated text. Models built on fixed subword vocabularies handle these poorly and are difficult to extend. The authors introduce UTC (Unified Toxic Content Classification), a compact byte-level Transformer. It takes raw UTF-8 bytes as input and uses Charformer's gradient-based subword tokenization (GBST) to learn latent "subwords" inside the network, so it needs no vocabulary. UTC is pretrained with a seq2seq denoising objective on multilingual documents and on user-generated comments from online forums. For fine-tuning, the decoder is removed and regression heads are attached to the encoder. On multilingual benchmarks built from real API traffic, and on tests of code-switching, covert toxicity, emoji hate, obfuscation, distribution shift and bias, UTC matches or beats mT5 and a comment-pretrained multilingual BERT. It also uses far fewer parameters than the top Kaggle solutions. The work shows that a token-free model can run fast enough for a production service. It also simplifies maintenance, since one model replaces many language-specific ones.

Key points

  • Architecture: GBST scores candidate byte blocks of sizes 1–4 at every position and then downsamples the sequence. Together with dropping the decoder after pretraining, this raised fine-tuning throughput from 18.3 steps/s (byte-level T5 Base, 200M params) to 32.9 steps/s (UTC, 102M).
  • Efficiency: UTC holds its own against the top-placing solutions from Kaggle's 2020 Jigsaw Multilingual Toxic Comment Classification competition while using over 10x fewer parameters.
  • HatemojiCheck: UTC† (268M) reached 90.8 accuracy and 93.3 F1, ahead of the English-only DeBERTa from Kirk et al. (87.9 / 91.0). The smaller UTC (102M) also beat that baseline.
  • CovertToxicity: UTC† scored 0.607 AUC-ROC, compared with 0.590 for Covert-BERT and 0.520 for Toxic-BERT.
  • Deployment: UTC† went live in Perspective API on 9 December 2021 for 10 new languages, including Arabic, Chinese, Czech, Polish, Hindi and Hinglish. In production it runs at about 15 QPS with roughly 200 ms median latency. In load tests, the 102M model achieved 45 ms median latency at 1K QPS on one TPUv2 chip.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢GoogleMountain View · United States492 works · 440,462 citations

Also on the byline: Google (Singapore).

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 works

🧭 Topics 1Research subjects and languages the work is about.Multilingual & cross-lingual NLP · 882 works

Cite

Cite as Alyssa Lees, Vinh Q. Tran, Yi Tay, Jeffrey Sorensen, Jai Gupta, Donald Metzler and Lucy Vasserman. 2022. A New Generation of Perspective API: Efficient Multilingual Character-level Transformers. KDD 2022.

The paper itself, not this page. BibTeX from the library's record of the work. arXiv 2202.11176

@inproceedings{lees2022new,
  title = {{A New Generation of Perspective API: Efficient Multilingual Character-level Transformers}},
  author = {Alyssa Lees, Vinh Q. Tran, Yi Tay, Jeffrey Sorensen, Jai Gupta, Donald Metzler and Lucy Vasserman},
  booktitle = {KDD 2022},
  year = {2022},
  doi = {10.1145/3534678.3539147},
  eprint = {2202.11176},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.1145/3534678.3539147}
}

References

36 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Tin Acosta, Alyssa Lees, Daniel Borkan, Jeffrey Sorensen, Alyssa Chvasta, Roelle Thorpe, and Lucy Vasserman. 2021. 10 New Languages for Perspective API. https://medium.com/jigsaw/10-new-languages-for-perspective-api-8cb0ad599d7c ↗
  2. Azalden Alakrot, Liam Murray, and Nikola S Nikolov. 2018. Towards accurate detection of offensive language in online communication in arabic. Procedia computer science 142 (2018), 315–320. find
  3. Francesco Barbieri, Jose Camacho-Collados, Luis Espinosa Anke, and Leonardo Neves. 2020. TweetEval: Unified Benchmark and Comparative Evaluation for Tweet Classification. In Findings of the Association for Computational Linguistics: EMNLP 2020. Association for Computational Linguistics, Online, 1644…
  4. Valerio Basile, Cristina Bosco, Elisabetta Fersini, Debora Nozza, Viviana Patti, Francisco Manuel Rangel Pardo, Paolo Rosso, and Manuela Sanguinetti. 2019. SemEval-2019 Task 5: Multilingual Detection of Hate Speech Against Immigrants and Women in Twitter. In Proceedings of the 13th International Wor…
  5. Daniel Borkan, Lucas Dixon, Jeffrey Sorensen, Nithum Thain, and Lucy Vasserman. 2019. Nuanced Metrics for Measuring Unintended Bias with Real Data for Text Classification. In Companion Proceedings of The 2019 World Wide Web A New Generation of Perspective API: Efficient Multilingual Character-level… — [doi:10.1145/3308560.3317593](https://doi.org/10.1145/3308560.3317593) find
  6. Emmanuel Gbenga Dada, Joseph Stephen Bassi, Haruna Chiroma, Shafi’i Muhammad Abdulhamid, Adebayo Olusola Adetunmbi, and Opeyemi Emmanuel Ajibuwa. 2019. Machine learning for email spam filtering: review, approaches and open research problems. Heliyon 5, 6 (2019), e01802. https://doi.org/10.1016/ j.he… find
  7. Thomas Davidson, Debasmita Bhattacharya, and Ingmar Weber. 2019. Racial Bias in Hate Speech and Abusive Language Detection Datasets. In Proceedings of the Third Workshop on Abusive Language Online. Association for Computational Linguistics, Florence, Italy, 25–35. https://doi.org/10.18653/v1/W19-350… ↗
  8. Thomas Davidson, Dana Warmsley, Michael Macy, and Ingmar Weber. 2017. Automated Hate Speech Detection and the Problem of Offensive Language. In Proceedings of the 11th International AAAI Conference on Web and Social Media (ICWSM ’17). AAAI, Palo Alto, CA, 512–515.
  9. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,…
  10. Lucas Dixon, John Li, Jeffrey Sorensen, Nithum Thain, and Lucy Vasserman. 2018. Measuring and Mitigating Unintended Bias in Text Classification. In Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society (AIES ’18). Association for Computing Machinery, New York, NY, USA, 67–73. https:… find
  11. Tommi Gröndahl, Luca Pajola, Mika Juuti, Mauro Conti, and N. Asokan. 2018. All You Need is "Love": Evading Hate Speech Detection. In Proceedings of the 11th ACM Workshop on Artificial Intelligence and Security (AISec ’18). Association for Computing Machinery, New York, NY, USA, 2–12. https://doi.org… ↗
  12. Abigail Z. Jacobs, Su Lin Blodgett, Solon Barocas, Hal Daumé, and Hanna Wallach. 2020. The Meaning and Measurement of Bias: Lessons from Natural Language Processing. In Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency (FAT* ’20). Association for Computing Machinery, N… — [doi:10.1145/3351095.3375671](https://doi.org/10.1145/3351095.3375671) find
  13. Google Jigsaw. 2017. Perspective API. https://www.perspectiveapi.com/. Accessed: 2021-02-02. ↗
  14. Brendan Kennedy, Mohammad Atari, Aida M Davani, Leigh Yeh, Ali Omrani, Yehsong Kim, Kris Coombs, Shreya Havaldar, Gwenyth Portillo-Wightman, Elaine Gonzalez, and et al. 2018. The Gab Hate Corpus: A collection of 27k posts annotated for hate speech. https://doi.org/10.31234/osf.io/hqjxn — [doi:10.31234/osf.io/hqjxn](https://doi.org/10.31234/osf.io/hqjxn) ↗
  15. H Kirk, B Vidgen, P Röttger, and SA Hale. 2021. Hatemoji: A test suite and adversarially-generated dataset for benchmarking and detecting emoji-based hate. (2021). find
  16. Pang Wei Koh, Shiori Sagawa, Henrik Marklund, Sang Michael Xie, Marvin Zhang, Akshay Balsubramani, Weihua Hu, Michihiro Yasunaga, Richard Lanas Phillips, Irena Gao, Tony Lee, Etienne David, Ian Stavness, Wei Guo, Berton Earnshaw, Imran Haque, Sara M Beery, Jure Leskovec, Anshul Kundaje, Emma Pierson… find
  17. Taku Kudo and John Richardson. 2018. SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations. Association for Computational Linguist…
  18. Keita Kurita, Anna Belova, and Antonios Anastasopoulos. 2019. Towards robust toxic content classification. find
  19. Alyssa Lees, Daniel Borkan, Ian Kivlichan, Jorge Nario, and Tesh Goyal. 2021. Capturing Covertly Toxic Speech via Crowdsourcing. In Proceedings of the First Workshop on Bridging Human–Computer Interaction and Natural Language Processing. Association for Computational Linguistics, Online, 14–20. http… find
  20. Thomas Mandl, Sandip Modha, Prasenjit Majumder, Daksh Patel, Mohana Dave, Chintak Mandlia, and Aditya Patel. 2019. Overview of the HASOC Track at FIRE 2019: Hate Speech and Offensive Content Identification in Indo-European Languages. In Proceedings of the 11th Forum for Information Retrieval Evaluat… — [doi:10.1145/3368567.3368584](https://doi.org/10.1145/3368567.3368584) find
  21. Dat Quoc Nguyen, Thanh Vu, and Anh Tuan Nguyen. 2020. BERTweet: A pre-trained language model for English Tweets. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations. Association for Computational Linguistics, Online, 9–14. https://doi.org/…
  22. Chikashi Nobata, Joel Tetreault, Achint Thomas, Yashar Mehdad, and Yi Chang. 2016. Abusive Language Detection in Online User Content. In Proceedings of the 25th International Conference on World Wide Web (WWW ’16). International World Wide Web Conferences Steering Committee, Republic and Canton of G… — [doi:10.1145/2872427.2883062](https://doi.org/10.1145/2872427.2883062) find
  23. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer. Journal of Machine Learning Research 21, 140 (2020), 1–67. http://jmlr.org/papers/v21… ↗
  24. Tharindu Ranasinghe, Marcos Zampieri, and Hansi Hettiarachchi. 2019. BRUMS at HASOC 2019: Deep Learning Models for Multilingual Hate Speech and Offensive Language Identification.. In FIRE (Working Notes). CEUR, Aachen, Germany, 199–207. find
  25. Maarten Sap, Dallas Card, Saadia Gabriel, Yejin Choi, and Noah A. Smith. 2019. The Risk of Racial Bias in Hate Speech Detection. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics, Florence, Italy, 1668–1678. https://…
  26. Noam Shazeer, Youlong Cheng, Niki Parmar, Dustin Tran, Ashish Vaswani, Penporn Koanantakool, Peter Hawkins, HyoukJoong Lee, Mingsheng Hong, Cliff Young, Ryan Sepassi, and Blake Hechtman. 2018. Mesh-TensorFlow: Deep Learning for Supercomputers. In Proceedings of the 32nd International Conference on N…
  27. Noam Shazeer and Mitchell Stern. 2018. Adafactor: Adaptive Learning Rates with Sublinear Memory Cost. In Proceedings of the 35th International Conference on Machine Learning (Proceedings of Machine Learning Research), Jennifer Dy and Andreas Krause (Eds.), Vol. 80. PMLR, Cambridge, Massachusetts, 45… find
  28. Guizhe Song, Degen Huang, and Zhifeng Xiao. 2021. A Study of Multilingual Toxic Text Detection Approaches under Imbalanced Sample Distribution. Information 12, 5 (2021), 1–16. https://doi.org/10.3390/info12050205 — [doi:10.3390/info12050205](https://doi.org/10.3390/info12050205) ↗
  29. Yi Tay, Vinh Q Tran, Sebastian Ruder, Jai Gupta, Hyung Won Chung, Dara Bahri, Zhen Qin, Simon Baumgartner, Cong Yu, and Donald Metzler. 2021. Charformer: Fast character transformers via gradient-based subword tokenization.
  30. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. 2017. Attention is All you Need. In Advances in Neural Information Processing Systems, I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garn…
  31. Cindy Wang and Michele Banko. 2021. Practical Transformer-based Multilingual Text Classification. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies: Industry Papers. Association for Computational Linguist… find
  32. Zeerak Waseem and Dirk Hovy. 2016. Hateful Symbols or Hateful People? Predictive Features for Hate Speech Detection on Twitter. In Proceedings of the NAACL Student Research Workshop. Association for Computational Linguistics, San Diego, California, 88–93. https://doi.org/10.18653/v1/N16-2013 ↗
  33. Ellery Wulczyn, Nithum Thain, and Lucas Dixon. 2017. Ex Machina: Personal Attacks Seen at Scale. In Proceedings of the 26th International Conference on World Wide Web (WWW ’17). International World Wide Web Conferences Steering Committee, Republic and Canton of Geneva, CHE, 1391–1399. https://doi.or… ↗
  34. Linting Xue, Aditya Barua, Noah Constant, Rami Al-Rfou, Sharan Narang, Mihir Kale, Adam Roberts, and Colin Raffel. 2021. ByT5: Towards a token-free future with pre-trained byte-to-byte models. CoRR abs/2105.13626 (2021). arXiv:2105.13626 https://arxiv.org/abs/2105.13626 ↗
  35. Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2021. mT5: A Massively Multilingual Pre-trained Text-to-Text Transformer. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Ling…
  36. W Yin and A. Zubiaga. 2021. Towards generalisable hate speech detection: a review on obstacles and solutions. PeerJ Computer Science 7, 5986 (2021), 1–38. https://doi.org/10.7717/peerj-cs.598 ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/lees-2022-new-generation-perspective-api/