Elamur Research
☰
home / papers / 2019 / Build it Break it Fix it for Dialogue Safety: Robustness from Adversarial Human Attack

Build it Break it Fix it for Dialogue Safety: Robustness from Adversarial Human Attack

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/D19-1461.pdf

Summary

The paper tackles offensive-language detection in dialogue, where malicious users deliberately probe classifiers for weak spots. The authors automate the "Build it Break it Fix it" idea with crowdworkers in the loop. They start with a BERT-based binary classifier (A0) trained on Wikipedia Toxic Comments (WTC). Crowdworkers then write messages they consider offensive but that the current models label SAFE. Each new model is retrained on all adversarial rounds collected so far plus WTC, and the cycle repeats. Over successive rounds, the models became much harder to fool. The collected attacks also changed character: obvious profanity gave way to figurative language, negation, and statements that require world knowledge to interpret. A second, multi-turn task shows that offensiveness often exists only in relation to earlier turns. Models that encode the dialogue context explicitly perform best on that task. The work argues that safety classifiers for chatbots and forums need adversarially collected, context-aware training data. All tasks, models and code are released through ParlAI.

Key points

  • WTC baseline: the BERT-based classifier (BERT-base, 12 layers) reaches 83.4 OFFENSIVE-class F1, compared with 71.4 for fastText. Its weighted F1 is 96.7, versus 95.4 for the BiLSTM of Khatri et al. (2018).
  • Single-turn robustness: in the single-turn rounds, the average crowdworker success score fell from 4.56 in round 1 to 1.6 in round 3. This drop indicates that the models grew progressively more robust.
  • Multi-turn data: the multi-turn adversarial task contains 24,000 train, 3,000 validation and 3,000 test examples, with about 10% labelled OFFENSIVE. Even against the best single-turn model (A3), workers broke the system more easily here, with an average score of 2.89. Their messages were also shorter.
  • Effect of context: BERT with dialogue segment embeddings scores 66.4 F1 with context versus 59.0 without. fastText goes the other way, scoring 23.6 with context versus 37.1 without.
  • Released resources: the data, trained models, training code and crowdsourcing interface are available via ParlAI (parl.ai/projects/dialogue_safety).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

134 papers of our library cite it (of the 140673 whose bibliographies we read); 19 of them have a page here, 115 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/d19-1461 · arXiv 1908.06083

@inproceedings{dinan2019build,
  title = {{Build it Break it Fix it for Dialogue Safety: Robustness from Adversarial Human Attack}},
  author = {Dinan, Emily and Humeau, Samuel and Chintagunta, Bharath and Weston, Jason},
  booktitle = {Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)},
  year = {2019},
  doi = {10.18653/v1/d19-1461},
  eprint = {1908.06083},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/d19-1461}
}

References

34 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. 2018. 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings. OpenReview.net. find
  2. Yonatan Belinkov and Yonatan Bisk. 2018. Synthetic and natural noise both break neural machine translation. In (DBL, 2018). find
  3. Jonathan Bishop. 2014. Representations of trolls in mass media communication: a review of media-texts and moral panics relating to internet trolling. International Journal of Web Based Communities, 10(1):7–24. find
  4. Andrew Brock, Jeff Donahue, and Karen Simonyan. 2018. Large scale gan training for high fidelity natural image synthesis. arXiv preprint arXiv:1809.11096. — [arXiv:1809.11096](https://arxiv.org/abs/1809.11096) find
  5. Justin Cheng, Cristian Danescu-Niculescu-Mizil, Jure Leskovec, and Michael Bernstein. 2017. Anyone can become a troll: Causes of trolling behavior in online discussions. American Scientist, 105(3):152. find
  6. Thomas Davidson, Dana Warmsley, Michael Macy, and Ingmar Weber. 2017. Automated hate speech detection and the problem of offensive language. In Eleventh International AAAI Conference on Web and Social Media. find our card
  7. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. CoRR, abs/1810.04805. find
  8. Emily Dinan, Varvara Logacheva, Valentin Malykh, Alexander Miller, Kurt Shuster, Jack Urbanek, Douwe Kiela, Arthur Szlam, Iulian Serban, Ryan Lowe, et al. 2019. The second conversational intelligence challenge (convai2). arXiv preprint arXiv:1902.00098. find our card
  9. Javid Ebrahimi, Anyi Rao, Daniel Lowd, and Dejing Dou. 2018. Hotflip: White-box adversarial examples for text classification. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, ACL 2018, Melbourne, Australia, July 15-20, 2018, Volume 2: Short Papers, pages 31… find our card
  10. Allyson Ettinger, Sudha Rao, Hal Daumé III, and Emily M Bender. 2017. Towards linguistically generalizable nlp systems: A workshop and shared task. arXiv preprint arXiv:1711.01505. find
  11. Patxi Galán-Garcı́a, José Gaviria de la Puerta, Carlos Laorden Gómez, Igor Santos, and Pablo Garcı́a Bringas. 2016. Supervised machine learning for the detection of troll profiles in twitter social network: Application to a real case of cyberbullying. Logic Journal of the IGPL, 24(1):42–53. find
  12. Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. 2014. Generative adversarial nets. In Advances in neural information processing systems, pages 2672–2680. find
  13. Google. 2018. Toxic comment classification challenge. find
  14. Tommi Gröndahl, Luca Pajola, Mika Juuti, Mauro Conti, and N Asokan. 2018. All you need is” love”: Evading hate-speech detection. arXiv preprint arXiv:1808.09115. — [arXiv:1808.09115](https://arxiv.org/abs/1808.09115) find
  15. Peter Henderson, Koustuv Sinha, Nicolas Angelard-Gontier, Nan Rosemary Ke, Genevieve Fried, Ryan Lowe, and Joelle Pineau. 2018. Ethical challenges in data-driven dialogue systems. In Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society, AIES ’18, pages 123–129, New York, NY, USA. A… find
  16. Susan Herring, Kirk Job-Sluder, Rebecca Scheckler, and Sasha Barab. 2002. Searching for safety online: Managing” trolling” in a feminist forum. The information society, 18(5):371–384. find
  17. Hossein Hosseini, Sreeram Kannan, Baosen Zhang, and Radha Poovendran. 2017. Deceiving google’s perspective api built for detecting toxic comments. arXiv preprint arXiv:1702.08138. — [arXiv:1702.08138](https://arxiv.org/abs/1702.08138) find
  18. Robin Jia and Percy Liang. 2017. Adversarial examples for evaluating reading comprehension systems. In (Palmer et al., 2017), pages 2021–2031. find our card
  19. Armand Joulin, Edouard Grave, Piotr Bojanowski, and Tomas Mikolov. 2017. Bag of tricks for efficient text classification. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers, pages 427–431. Association for Computation… find
  20. Chandra Khatri, Behnam Hedayatnia, Rahul Goel, Anushree Venkatesh, Raefer Gabriel, and Arindam Mandal. 2018. Detecting offensive content in open-domain conversations using two stage semi-supervision. CoRR, abs/1811.12900. — [arXiv:1811.12900](https://arxiv.org/abs/1811.12900) find
  21. Ritesh Kumar, Atul Kr. Ojha, Shervin Malmasi, and Marcos Zampieri. 2018. Benchmarking aggression identification in social media. In Proceedings of the First Workshop on Trolling, Aggression and Cyberbullying (TRAC-2018), pages 1–11, Santa Fe, New Mexico, USA. Association for Computational Linguistic… find
  22. Jiwei Li, Will Monroe, Tianlin Shi, Sébastien Jean, Alan Ritter, and Dan Jurafsky. 2017. Adversarial learning for neural dialogue generation. In (Palmer et al., 2017), pages 2157–2169. find
  23. Aishan Liu, Xianglong Liu, Jiaxin Fan, Yuqing Ma, Anlan Zhang, Huiyuan Xie, and Dacheng Tao. 2019. Perceptual-sensitive gan for generating adversarial patches. find
  24. David Noever. 2018. Machine learning suites for online toxicity detection. arXiv preprint arXiv:1810.01869. — [arXiv:1810.01869](https://arxiv.org/abs/1810.01869) find
  25. Marcos Zampieri, Shervin Malmasi, Preslav Nakov, Sara Rosenthal, Noura Farra, and Ritesh Kumar. 2019. Semeval-2019 task 6: Identifying and categorizing offensive language in social media (offenseval). arXiv preprint arXiv:1903.08983. find our card
  26. Saizheng Zhang, Emily Dinan, Jack Urbanek, Arthur Szlam, Douwe Kiela, and Jason Weston. 2018. Personalizing dialogue agents: I have a dog, do you have pets too? In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, ACL 2018, Melbourne, Australia, July 15-20, 201… find our card
  27. Martha Palmer, Rebecca Hwa, and Sebastian Riedel, editors. 2017. Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, EMNLP 2017, Copenhagen, Denmark, September 9-11, 2017. Association for Computational Linguistics. find
  28. Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. Glove: Global vectors for word representation. In Empirical Methods in Natural Language Processing (EMNLP), pages 1532– 1543. find
  29. Andrew Ruef, Michael Hicks, James Parker, Dave Levin, Michelle L Mazurek, and Piotr Mardziel. 2016. Build it, break it, fix it: Contesting secure development. In Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security, pages 690–703. ACM. find
  30. Pnina Shachaf and Noriko Hara. 2010. Beyond vandalism: Wikipedia trolls. Journal of Information Science, 36(3):357–370. find
  31. Leandro Silva, Mainack Mondal, Denzil Correa, Fabrı́cio Benevenuto, and Ingmar Weber. 2016. Analyzing the targets of hate in online social media. In Tenth International AAAI Conference on Web and Social Media. find
  32. Marty J Wolf, K Miller, and Frances S Grodzinsky. 2017. Why we should have seen that coming: comments on microsoft’s tay experiment, and wider implications. ACM SIGCAS Computers and Society, 47(3):54–64. find
  33. Ellery Wulczyn, Nithum Thain, and Lucas Dixon. 2017. Ex machina: Personal attacks seen at scale. In Proceedings of the 26th International Conference on World Wide Web, WWW 2017, Perth, Australia, April 3-7, 2017, pages 1391–1399. ACM. find
  34. Zhilin Yang, Saizheng Zhang, Jack Urbanek, Will Feng, Alexander H. Miller, Arthur Szlam, Douwe Kiela, and Jason Weston. 2018. Mastering the dungeon: Grounded language learning by mechanical turker descent. In (DBL, 2018). find