Elamur Research⌕
☰
home / papers / 2023 / NL-Augmenter: A Framework for Task-Sensitive Natural Language Augmentation

NL-Augmenter: A Framework for Task-Sensitive Natural Language Augmentation

Summary

Data augmentation is used both to stress-test NLP models and to make their training data more diverse. Most existing perturbation tools offer a narrow, largely English set of edits that are rarely matched to a specific task. NL-Augmenter addresses this with an open, community-built Python library. It grew out of the GEM benchmark and follows the BIG-bench model: researchers submitted code via GitHub, went through peer review, and became co-authors. The library contains 117 transformations that rewrite inputs, such as typos, socio-linguistic variation, style and syntax changes, and odd but human-readable distortions. It also has 23 filters that return a yes/no decision for building data subsets. Each contribution is tagged with its supported tasks, data formats and languages, and is documented in a data card. The authors applied randomly sampled perturbations to four popular pre-trained models on four tasks and saw near-consistent score drops, with vulnerability varying by model and task. The library gives researchers a shared, extensible toolkit for robustness analysis, paraphrase generation and low-resource augmentation.

Key points

  • The initial release has 117 transformations and 23 filters, annotated with descriptive tags that the authors themselves admit are noisy. The code, data cards and evaluation results are at github.com/GEM-benchmark/NL-Augmenter.
  • The library is compared with TextAttack, OpenAttack, NLPAug, Checklist, Robustness Gym and TextFlint. It has the most transformations (TextFlint is next with 80), and only it and TextFlint offer both task-specific transformations and filters.
  • In the robustness tests, transformer models failed even under non-adversarial, randomly chosen perturbations. The tool currently measures average-case performance, and worst-case evaluation is planned.
  • The authors list several limitations:
    • Tag assignment is inconsistent, which may explain surprising findings such as meaning-preserving edits hurting models more than meaning-altering ones.
    • It is not yet shown that the perturbations help fix the weaknesses they expose.
    • Individual contributions risk being diluted in such a large collection.
  • The abstract is published in several languages, including Spanish, Hindi, Chinese, Quechua and Indonesian.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cited by

67 papers of our library cite it (of the 140673 whose bibliographies we read); 7 of them have a page here, 60 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.3384/nejlt.2000-1533.2023.4725

@article{dhole2023nl,
  title = {{NL-Augmenter: A Framework for Task-Sensitive Natural Language Augmentation}},
  author = {Dhole, Kaustubh and Gangal, Varun and Gehrmann, Sebastian and Gupta, Aadesh and Li, Zhenhao and Mahamood, Saad and Mahadiran, Abinaya and Mille, Simon and Shrivastava, Ashish and Tan, Samson and Wu, Tongshang and Sohl-Dickstein, Jascha and Choi, Jinho and Hovy, Eduard and Dušek, Ondřej and Ruder, Sebastian and Anand, Sajant and Aneja, Nagender and Banjade, Rabin and Barthe, Lisa and Behnke, Hanna and Berlot-Attwell, Ian and Boyle, Connor and Brun, Caroline and Cabezudo, Marco Antonio Sobrevilla and Cahyawijaya, Samuel and Chapuis, Emile and Che, Wanxiang and Choudhary, Mukund and Clauss, Christian and Colombo, Pierre and Cornell, Filip and Dagan, Gautier and Das, Mayukh and Dixit, Tanay and Dopierre, Thomas and Dray, Paul-Alexis and Dubey, Suchitra and Ekeinhor, Tatiana and Giovanni, Marco Di and Goyal, Tanya and Gupta, Rishabh and Hamla, Louanes and Han, Sang and Harel-Canada, Fabrice and Honoré, Antoine and Jindal, Ishan and Joniak, Przemysław and Kleyko, Denis and Kovatchev, Venelin and Krishna, Kalpesh and Kumar, Ashutosh and Langer, Stefan and Lee, Seungjae Ryan and Levinson, Corey James and Liang, Hualou and Liang, Kaizhao and Liu, Zhexiong and Lukyanenko, Andrey and Marivate, Vukosi and Melo, Gerard de and Meoni, Simon and Meyer, Maxine and Mir, Afnan and Moosavi, Nafise Sadat and Meunnighoff, Niklas and Mun, Timothy Sum Hon and Murray, Kenton and Namysl, Marcin and Obedkova, Maria and Oli, Priti and Pasricha, Nivranshu and Pfister, Jan and Plant, Richard and Prabhu, Vinay and Pais, Vasile and Qin, Libo and Raji, Shahab and Rajpoot, Pawan Kumar and Raunak, Vikas and Rinberg, Roy and Roberts, Nicholas and Rodriguez, Juan Diego and Roux, Claude and Samus, Vasconcellos and Sai, Ananya and Schmidt, Robin and Scialom, Thomas and Sefara, Tshephisho and Shamsi, Saqib and Shen, Xudong and Shi, Yiwen and Shi, Haoyue and Shvets, Anna and Siegel, Nick and Sileo, Damien and Simon, Jamie and Singh, Chandan and Sitelew, Roman and Soni, Priyank and Sorensen, Taylor and Soto, William and Srivastava, Aman and Srivatsa, Aditya and Sun, Tony and Varma, Mukund and Tabassum, A and Tan, Fiona and Teehan, Ryan and Tiwari, Mo and Tolkiehn, Marie and Wang, Athena and Wang, Zijian and Wang, Zijie and Wang, Gloria and Wei, Fuxuan and Wilie, Bryan and Winata, Genta Indra and Wu, Xinyu and Wydmanski, Witold and Xie, Tianbao and Yaseen, Usama and Yee, Michael and Zhang, Jing and Zhang, Yue},
  journal = {Northern European Journal of Language Technology},
  year = {2023},
  doi = {10.3384/nejlt.2000-1533.2023.4725},
  url = {https://doi.org/10.3384/nejlt.2000-1533.2023.4725}
}

References

140 references, 34 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. 2006. Respectful Disability Language: Heres Whats Up! https://www.aucd.org/docs/add/sa_ summits/Language%20Doc.pdf. ↗
  2. Bamman, David. 2017. Natural language processing for the long tail. In DH. find
  3. Berard, Alexandre, Ioan Calapodescu, and Claude Roux. 2019. Naver labs europe’s systems for the wmt19 machine translation robustness task. arXiv preprint arXiv:1907.06488. find
  4. Bhagat, Rahul and Eduard Hovy. 2013. What is a paraphrase? Computational Linguistics, 39(3):463–472. find
  5. Bhatt, Abhinav and Kaustubh D. Dhole. 2020. Benchmarking biorelex for entity tagging and relation extraction. find
  6. Bird, Steven. 2006. Nltk: the natural language toolkit. In Proceedings of the COLING/ACL 2006 Interactive Presentation Sessions, pages 69–72. find
  7. Board, Smorga’s. 2021. Frequently misspelled word list for dyslexia. find
  8. Bonial, Claire, Jena Hwang, Julia Bonn, Kathryn Conger, Olga Babko-Malaya, and Martha Palmer. 2012. English propbank annotation guidelines. Center for Computational Language and Education Research Institute of Cognitive Science University of Colorado at Boulder, 48. find
  9. Cahyawijaya, Samuel, Holy Lovenia, Alham Fikri Aji, Genta Indra Winata, Bryan Wilie, Rahmad Mahendra, Christian Wibisono, Ade Romadhony, Karissa Vincentio, Fajri Koto, Jennifer Santoso, David Moeljadi, Cahya Wirawan, Frederikus Hudi, Ivan Halim Parmonangan, Ika Alfina, Muhammad Satrio Wicaksono, Ilh… find
  10. Chen, Jiaao, Derek Tam, Colin Raffel, Mohit Bansal, and Diyi Yang. 2021. An empirical survey of data augmentation for limited data learning in nlp. find
  11. Dai, Xiang and Heike Adel. 2020. An analysis of simple data augmentation for named entity recognition. In Proceedings of the 28th International Conference on Computational Linguistics, pages 3861–3867, Barcelona, Spain (Online). International Committee on Computational Linguistics. find
  12. Deorowicz, Sebastian and Marcin G Ciura. 2005. Correcting spelling errors by modelling their causes. International journal of applied mathematics and computer science, 15:275–285. find
  13. Dhole, Kaustubh D. 2020. Resolving intent ambiguities by retrieving discriminative clarifying questions. find
  14. Dinan, Emily, Angela Fan, Adina Williams, Jack Urbanek, Douwe Kiela, and Jason Weston. 2020. Queens are powerful too: Mitigating gender bias in dialogue generation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 8173–8188, Online. Association… find our card
  15. Dolan, William B and Chris Brockett. 2005. Automatically constructing a corpus of sentential paraphrases. In Proceedings of the Third International Workshop on Paraphrasing (IWP2005). find our card
  16. Dopierre, Thomas, Christophe Gravier, and Wilfried Logerais. 2021. Protaugment: Unsupervised diverse short-texts paraphrasing for intent detection meta-learning. CoRR, abs/2105.12995. find
  17. Eger, Steffen and Yannik Benz. 2020. From hero to zéroe: A benchmark of low-level adversarial attacks. find
  18. Eger, Steffen, Gözde Gül Şahin, Andreas Rücklé, Ji-Ung Lee, Claudia Schulz, Mohsen Mesgar, Krishnkant Swarnkar, Edwin Simpson, and Iryna Gurevych. 2019a. Text processing like humans do: Visually attacking and shielding NLP systems. In Proceedings of the 2019 Conference of the North American Chapter… find
  19. Eger, Steffen, Gözde Gül Sahin, Andreas Rücklé, Ji-Ung Lee, Claudia Schulz, Mohsen Mesgar, Krishnkant Swarnkar, Edwin Simpson, and Iryna Gurevych. 2019b. Text processing like humans do: Visually attacking and shielding NLP systems. CoRR, abs/1903.11508. find
  20. Eisner, Ben, Tim Rocktäschel, Isabelle Augenstein, Matko Bošnjak, and Sebastian Riedel. 2016. emoji2vec: Learning emoji representations from their description. In Proceedings of The Fourth International Workshop on Natural Language Processing for Social Media, pages 48–54, Austin, TX, USA. Associati… find
  21. Fadaee, Marzieh, Arianna Bisazza, and Christof Monz. 2017. Data augmentation for low-resource neural machine translation. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 567–573, Vancouver, Canada. Association for Computation… find our card
  22. Fan, Angela, Shruti Bhosale, Holger Schwenk, Zhiyi Ma, Ahmed El-Kishky, Siddharth Goyal, Mandeep Baines, Onur Celebi, Guillaume Wenzek, Vishrav Chaudhary, et al. 2021. Beyond english-centric multilingual machine translation. Journal of Machine Learning Research, 22(107):1–48. find
  23. Fan, Angela, Shruti Bhosale, Holger Schwenk, Zhiyi Ma, Ahmed El-Kishky, Siddharth Goyal, Mandeep Baines, Onur Çelebi, Guillaume Wenzek, Vishrav Chaudhary, Naman Goyal, Tom Birch, Vitaliy Liptchinsky, Sergey Edunov, Edouard Grave, Michael Auli, and Armand Joulin. 2020. Beyond english-centric multilin… find
  24. Feng, Steven Y, Varun Gangal, Jason Wei, Sarath Chandar, Soroush Vosoughi, Teruko Mitamura, and Eduard Hovy. 2021. A survey of data augmentation approaches for nlp. arXiv preprint arXiv:2105.03075. find our card
  25. Fort, Karën, Gilles Adda, and K. Bretonnel Cohen. 2011. Amazon Mechanical Turk: Gold Mine or Coal Mine? Computational Linguistics, 37(2):413–420. find
  26. Galton, Francis. 1907. Vox populi (the wisdom of crowds). Nature, 75(7):450–451. find
  27. Gangal, Varun, Steven Y Feng, Eduard Hovy, and Teruko Mitamura. 2021. Nareor: The narrative reordering problem. arXiv preprint arXiv:2104.06669. — [arXiv:2104.06669](https://arxiv.org/abs/2104.06669) find
  28. Gardner, Matt, Yoav Artzi, Victoria Basmov, Jonathan Berant, Ben Bogin, Sihao Chen, Pradeep Dasigi, Dheeru Dua, Yanai Elazar, Ananth Gottumukkala, et al. 2020. Evaluating models local decision boundaries via contrast sets. In Proceedings of the 2020 Conference on Empirical Methods in Natural Languag… find our card
  29. Gauthier, Jon, Jennifer Hu, Ethan Wilcox, Peng Qian, and Roger Levy. 2020. Syntaxgym: An online platform for targeted evaluation of language models. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 70–76. find
  30. Gehrmann, Sebastian, Tosin Adewumi, Karmanya Aggarwal, Pawan Sasanka Ammanamanchi, Anuoluwapo Aremu, Antoine Bosselut, Khyathi Raghavi Chandu, Miruna-Adriana Clinciu, Dipanjan Das, Kaustubh Dhole, Wanyu Du, Esin Durmus, Ondřej Dušek, Chris Chinenye Emezue, Varun Gangal, Cristina Garbacea, Tatsunori… find
  31. Gehrmann, Sebastian, Abhik Bhattacharjee, Abinaya Mahendiran, Alex Wang, Alexandros Papangelis, Aman Madaan, Angelina McMillan-Major, Anna Shvets, Ashish Upadhyay, Bingsheng Yao, Bryan Wilie, Chandra Bhagavatula, Chaobin You, Craig Thomson, Cristina Garbacea, Dakuo Wang, Daniel Deutsch, Deyi Xiong,… find
  32. Gildea, Daniel and Martha Stone Palmer. 2002. The necessity of parsing for predicate argument recognition. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, July 6-12, 2002, Philadelphia, PA, USA, pages 239–246. ACL. find
  33. Goel, Karan, Nazneen Rajani, Jesse Vig, Samson Tan, Jason Wu, Stephan Zheng, Caiming Xiong annd Mohit Bansal, and Christopher Ré. 2021. Robustness Gym: Unifying the NLP evaluation landscape. arXiv preprint arXiv:2101.04840. find
  34. Goldberg, Yoav. 2017. Neural network methods for natural language processing. Synthesis lectures on human language technologies, 10(1):1–309. find
  35. Goyal, Tanya and Greg Durrett. 2020. Neural syntactic preordering for controlled paraphrase generation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 238–252, Online. Association for Computational Linguistics. find our card
  36. Guntuku, Sharath Chandra, Mingyang Li, Louis Tay, and Lyle H Ungar. 2019. Studying cultural differences in emoji usage across the east and the west. In Proceedings of the International AAAI Conference on Web and Social Media, volume 13, pages 226–235. find
  37. Gupta, Aadesh, Kaustubh D. Dhole, Rahul Tarway, Swetha Prabhakar, and Ashish Shrivastava. 2021. Candle: Decomposing conditional and conjunctive queries for task-oriented dialogue systems. find
  38. Harel-Canada, Fabrice. 2021. Sibyl. find
  39. Hendrickx, Iris, Zornitsa Kozareva, Preslav Nakov, Diarmuid Ó Séaghdha, Stan Szpakowicz, and Tony Veale. 2013. Semeval-2013 task 4: Free paraphrases of noun compounds. In Second Joint Conference on Lexical and Computational Semantics (* SEM), Volume 2: Proceedings of the Seventh International Worksh… find
  40. Jalalzai, Hamid, Pierre Colombo, Chloé Clavel, Eric Gaussier, Giovanna Varni, Emmanuel Vignon, and Anne Sabourin. 2020. Heavy-tailed representations, text polarity classification & data augmentation. In Advances in Neural Information Processing Systems, volume 33, pages 4295–4307. Curran Associates,… find
  41. Jia, Robin and Percy Liang. 2017a. Adversarial examples for evaluating reading comprehension systems. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, EMNLP 2017, Copenhagen, Denmark, September 9-11, 2017, pages 2021–2031. Association for Computational Lingu… find our card
  42. Jia, Robin and Percy Liang. 2017b. Adversarial examples for evaluating reading comprehension systems. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2021–2031, Copenhagen, Denmark. Association for Computational Linguistics. find our card
  43. Jindal, Ishan, Ranit Aharonov, Siddhartha Brahma, Huaiyu Zhu, and Yunyao Li. 2020. Improved semantic role labeling using parameterized neighborhood memory adaptation. arXiv preprint arXiv:2011.14459. — [arXiv:2011.14459](https://arxiv.org/abs/2011.14459) find
  44. Kaushik, Divyansh, Eduard Hovy, and Zachary C Lipton. 2019. Learning the difference that makes a difference with counterfactually-augmented data. arXiv preprint arXiv:1909.12434. find
  45. Khachatrian, Hrant, Lilit Nersisyan, Karen Hambardzumyan, Tigran Galstyan, Anna Hakobyan, Arsen Arakelyan, A. Rzhetsky, and A. G. Galstyan. 2019. Biorelex 1.0: Biological relation extraction benchmark. In BioNLP@ACL. find
  46. Kiela, Douwe, Max Bartolo, Yixin Nie, Divyansh Kaushik, Atticus Geiger, Zhengxuan Wu, Bertie Vidgen, Grusha Prasad, Amanpreet Singh, Pratik Ringshia, Zhiyi Ma, Tristan Thrush, Sebastian Riedel, Zeerak Waseem, Pontus Stenetorp, Robin Jia, Mohit Bansal, Christopher Potts, and Adina Williams. 2021. Dyn… find
  47. Kingsbury, Paul R. and Martha Palmer. 2002. From treebank to propbank. In Proceedings of the Third International Conference on Language Resources and Evaluation, LREC 2002, May 29-31, 2002, Las Palmas, Canary Islands, Spain. European Language Resources Association. and the 11th International Joint C… find our card
  48. Krishna, Kalpesh, John Wieting, and Mohit Iyyer. 2020. Reformulating unsupervised style transfer as paraphrase generation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 737–762, Online. Association for Computational Linguistics. find our card
  49. Kumar, Ashutosh, Satwik Bhattamishra, Manik Bhandari, and Partha Talukdar. 2019. Submodular optimization-based diverse paraphrasing and its effectiveness in data augmentation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human… find our card
  50. Lample, Guillaume, Alexis Conneau, Marc’Aurelio Ranzato, Ludovic Denoyer, and Hervé Jégou. 2018. Word translation without parallel data. In International Conference on Learning Representations. find
  51. Laserna, Charlyn M, Yi-Tai Seih, and James W Pennebaker. 2014. Um... who like says you know: Filler word use as a function of age, gender, and personality. Journal of Language and Social Psychology, 33(3):328–338. find
  52. Lauer, Mark. 1995. Designing Statistical Language Learners: Experiments on Noun Compounds. Ph.D. thesis. find
  53. Lee, Kenton, Luheng He, and Luke Zettlemoyer. 2018. Higher-order coreference resolution with coarse-to-fine inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 687… find our card
  54. Kočiskỳ, Tomáš, Jonathan Schwarz, Phil Blunsom, Chris Dyer, Karl Moritz Hermann, Gábor Melis, and Edward Grefenstette. 2018. The narrativeqa reading comprehension challenge. Transactions of the Association for Computational Linguistics, 6:317–328. find our card
  55. Lewis, Mike, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of th… find our card
  56. Kovatchev, Venelin, Phillip Smith, Mark Lee, and Rory Devine. 2021. Can vectors read minds better than experts? comparing data augmentation strategies for the automated scoring of children’s mindreading ability. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguisti… find
  57. Lhoest, Quentin, Albert Villanova del Moral, Patrick von Platen, Thomas Wolf, Mario ako, Yacine Jernite, Abhishek Thakur, Lewis Tunstall, Suraj Patil, Mariama Drame, Julien Chaumond, Julien Plu, Joe Davison, Simon Brandeis, Victor Sanh, Teven Le Scao, Kevin Canwen Xu, Nicolas Patry, Steven Liu, Ange… find
  58. Li, Dianqi, Yizhe Zhang, Hao Peng, Liqun Chen, Chris Brockett, Ming-Ting Sun, and Bill Dolan. 2020a. Contextualized perturbation for textual adversarial attack. arXiv preprint arXiv:2009.07502. find
  59. Li, Dianqi, Yizhe Zhang, Hao Peng, Liqun Chen, Chris Brockett, Ming-Ting Sun, and Bill Dolan. 2020b. Contextualized perturbation for textual adversarial attack. CoRR, abs/2009.07502. find
  60. Li, Zhenhao and Lucia Specia. 2019. Improving neural machine translation robustness via data augmentation: Beyond back-translation. Proceedings of the 5th Workshop on Noisy User-generated Text (W-NUT 2019). find
  61. Lin, Bill Yuchen, Wangchunshu Zhou, Ming Shen, Pei Zhou, Chandra Bhagavatula, Yejin Choi, and Xiang Ren. 2019. Commongen: A constrained text generation challenge for generative commonsense reasoning. arXiv preprint arXiv:1911.03705. find
  62. Liu, Zihan, Genta Indra Winata, and Pascale Fung. 2021. Continual mixed-language pre-training for extremely low-resource neural machine translation. arXiv preprint arXiv:2105.03953. find
  63. Liu, Zihan, Genta Indra Winata, Zhaojiang Lin, Peng Xu, and Pascale Fung. 2020. Attention-informed mixed-language training for zero-shot cross-lingual task-oriented dialogue systems. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 8433–8440. find
  64. Logeswaran, Lajanugen, Honglak Lee, and Samy Bengio. 2018. Content preserving text generation with attribute controls. In Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, December 3-8, 2018, Montréal, Canada, pages 5… find
  65. Lu, Kaiji, Piotr Mardziel, Fangjing Wu, Preetam Amancharla, and Anupam Datta. 2019. Gender bias in neural natural language processing. find our card
  66. Ma, Edward. 2019. Nlp augmentation. https://github.com/makcedward/nlpaug. ↗
  67. Maas, Andrew, Raymond E Daly, Peter T Pham, Dan Huang, Andrew Y Ng, and Christopher Potts. 2011. Learning word vectors for sentiment analysis. In Proceedings of the 49th annual meeting of the association for computational linguistics: Human language technologies, pages 142–150. find our card
  68. Marcus, Mitchell P., Beatrice Santorini, and Mary Ann Marcinkiewicz. 1993. Building a large annotated corpus of English: The Penn Treebank. Computational Linguistics, 19(2):313–330. find our card
  69. Marivate, Vukosi and Tshephisho Sefara. 2020. Improving short text classification through global augmentation methods. In International Cross-Domain Conference for Machine Learning and Knowledge Extraction, pages 385–399. Springer. find
  70. McCoy, Tom, Ellie Pavlick, and Tal Linzen. 2019. Right for the wrong reasons: Diagnosing syntactic heuristics in natural language inference. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3428–3448, Florence, Italy. Association for Computational Lin… find our card
  71. Merriam-Webster. What is a diacritic, anyway? find
  72. Mille, Simon, Kaustubh Dhole, Saad Mahamood, Laura Perez-Beltrachini, Varun Prashant Gangal, Mihir Kale, Emiel van Miltenburg, and Sebastian Gehrmann. 2021. Automatic construction of evaluation suites for natural language generation datasets. In Proceedings of the Neural Information Processing Syste… find
  73. Miller, George A. 1998. WordNet: An electronic lexical database. MIT press. find
  74. Mishra, Shubhanshu, Sijun He, and Luca Belli. 2020. Assessing demographic bias in named entity recognition. CoRR, abs/2008.03415. — [arXiv:2008.03415](https://arxiv.org/abs/2008.03415) find
  75. Mitchell, Margaret, Simone Wu, Andrew Zaldivar, Parker Barnes, Lucy Vasserman, Ben Hutchinson, Elena Spitzer, Inioluwa Deborah Raji, and Timnit Gebru. 2019. Model cards for model reporting. In Proceedings of the Conference on Fairness, Accountability, and Transparency, FAT* 2019, Atlanta, GA, USA, J… find
  76. Morris, John, Eli Lifland, Jin Yong Yoo, Jake Grigsby, Di Jin, and Yanjun Qi. 2020. Textattack: A framework for adversarial attacks, data augmentation, and adversarial training in nlp. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, p… find our card
  77. Namysl, Marcin, Sven Behnke, and Joachim Köhler. 2020. NAT: Noise-aware training for robust neural sequence labeling. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1501–1517, Online. Association for Computational Linguistics. find
  78. Namysl, Marcin, Sven Behnke, and Joachim Köhler. 2021. Empirical error modeling improves robustness of noisy neural sequence labeling. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 314–329, Online. Association for Computational Linguistics. find
  79. Nekoto, Wilhelmina, Vukosi Marivate, Tshinondiwa Matsila, Timi Fasubaa, Taiwo Fagbohungbe, Solomon Oluwole Akinola, Shamsuddeen Muhammad, Salomon Kabongo Kabenamualu, Salomey Osei, Freshia Sackey, Rubungo Andre Niyongabo, Ricky Macharm, Perez Ogayo, Orevaoghene Ahia, Musie Meressa Berhe, Mofetoluwa… find our card
  80. Palmer, Martha, Paul R. Kingsbury, and Daniel Gildea. 2005. The proposition bank: An annotated corpus of semantic roles. Comput. Linguistics, 31(1):71–106. find our card
  81. Parikh, Soham, Ananya B. Sai, Preksha Nema, and Mitesh M. Khapra. 2019. Eliminet: A model for eliminating options for reading comprehension with multiple choice questions. CoRR, abs/1904.02651. — [arXiv:1904.02651](https://arxiv.org/abs/1904.02651) find
  82. Park, Kyubyong and Seanie Lee. 2020. g2pm: A neural grapheme-to-phoneme conversion package for mandarin chinese based on a new open benchmark dataset. CoRR, abs/2004.03136. — [arXiv:2004.03136](https://arxiv.org/abs/2004.03136) find
  83. Pierse, Charles. 2021. Transformers Interpret. find
  84. Piktus, Aleksandra, Necati Bora Edizel, Piotr Bojanowski, Edouard Grave, Rui Ferreira, and Fabrizio Silvestri. 2019. Misspelling oblivious word embeddings. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologie… find
  85. Pitler, Emily, Mridhula Raghupathy, Hena Mehta, Ani Nenkova, Alan Lee, and Aravind Joshi. 2008. Easily identifiable discourse relations. In Coling 2008: Companion volume: Posters, pages 87–90, Manchester, UK. find our card
  86. Coling 2008 Organizing Committee. find
  87. Ponkiya, Girishkumar, Rudra Murthy, Pushpak Bhattacharyya, and Girish Palshikar. 2020. Looking inside noun compounds: Unsupervised prepositional and free paraphrasing using language models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: Findings, pages 431… find
  88. Ponkiya, Girishkumar, Kevin Patel, Pushpak Bhattacharyya, and Girish Palshikar. 2018. Treat us like the sequences we are: Prepositional paraphrasing of noun compounds using lstm. In Proceedings of the 27th International Conference on Computational Linguistics, pages 1827–1836. find
  89. Nguyen, Toan Q, Kenton Murray, and David Chiang. 2021. Data augmentation by concatenation for low-resource translation: A mystery and a solution. In Proceedings of the International Workshop on Spoken Language Translation, Online. Association for Computational Linguistics. find
  90. Prasad, Rashmi, Nikhil Dinesh, Alan Lee, Eleni Miltsakaki, Livio Robaldo, Aravind Joshi, and Bonnie Webber. 2008. The Penn Discourse TreeBank 2.0. In Proceedings of the Sixth International Conference on Language Resources and Evaluation (LREC’08), Marrakech, Morocco. European Language Resources Asso… find our card
  91. Pais, Vasile Florian. 2019. Contributions to semantic processing of texts; Identification of entities and relations between textual units; Case study on Romanian language. Ph.D. thesis. find
  92. Pruksachatkun, Yada, Satyapriya Krishna, Jwala Dhamala, Rahul Gupta, and Kai-Wei Chang. 2021. Does robustness improve fairness? approaching fairness with word substitution robustness methods for text classification. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages… find
  93. Qin, Libo, Minheng Ni, Yue Zhang, and Wanxiang Che. 2020. Cosda-ml: Multi-lingual code-switching data augmentation for zero-shot cross-lingual nlp. In Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAI-20, pages 3853–3860. International Joint Conferences… find
  94. Radford, Alec, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. find
  95. Raffel, Colin, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2019. Exploring the limits of transfer learning with a unified text-to-text transformer. find
  96. Raunak, Vikas, Arul Menezes, and Marcin Junczys-Dowmunt. 2021. The curious case of hallucinations in neural machine translation. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1172–1183, Online.… find our card
  97. Ravichander, Abhilasha, Siddharth Dalmia, Maria Ryskina, Florian Metze, Eduard Hovy, and Alan W Black. 2021. NoiseQA: Challenge Set Evaluation for User-Centric Question Answering. In Conference of the European Chapter of the Association for Computational Linguistics (EACL), Online. find
  98. Regina, Mehdi, Maxime Meyer, and Sébastien Goutal. 2020. Text data augmentation: Towards better detection of spear-phishing emails. CoRR, abs/2007.02033. — [arXiv:2007.02033](https://arxiv.org/abs/2007.02033) find
  99. Ribeiro, Marco Tulio, Tongshuang Wu, Carlos Guestrin, and Sameer Singh. 2020. Beyond accuracy: Behavioral testing of NLP models with CheckList. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4902–4912, Online. Association for Computational Linguisti… find
  100. Shi, Haoyue, Karen Livescu, and Kevin Gimpel. 2021. Substructure substitution: Structured data augmentation for NLP. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 3494–3508, Online. Association for Computational Linguistics. find
  101. Shi, Peng and Jimmy Lin. 2019a. Simple bert models for relation extraction and semantic role labeling. arXiv preprint arXiv:1904.05255. find our card
  102. Shi, Peng and Jimmy Lin. 2019b. Simple BERT models for relation extraction and semantic role labeling. CoRR, abs/1904.05255. find our card
  103. Shrivastava, Ashish, Kaustubh Dhole, Abhinav Bhatt, and Sharvani Raghunath. 2021. Saying No is An Art: Contextualized Fallback Responses for Unanswerable Dialogue Queries. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conf… find
  104. Shwartz, Vered and Ido Dagan. 2018. Paraphrase to explicate: Revealing implicit noun-compound relations. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1200–1211. find
  105. Si, Chenglei, Zhengyan Zhang, Fanchao Qi, Zhiyuan Liu, Yasheng Wang, Qun Liu, and Maosong Sun. 2021. Better robustness by more coverage: Adversarial and mixup data augmentation for robust finetuning. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 1569–1576, Onli… find
  106. Smith, R. 2007. An overview of the tesseract OCR engine. In 9th International Conference on Document Analysis and Recognition (ICDAR 2007), 23-26 September, Curitiba, Paraná, Brazil, pages 629–633. IEEE Computer Society. find
  107. Socher, Richard, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 1631… find
  108. Srivastava, Aarohi, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, et al. 2022. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. arXiv preprint arXiv:2206.04615. find our card
  109. Sugiyama, Amane and Naoki Yoshinaga. 2019. Data augmentation using back-translation for context-aware neural machine translation. In Proceedings of the Fourth Workshop on Discourse in Machine Translation (DiscoMT 2019), pages 35–44, Hong Kong, China. Association for Computational Linguistics. find
  110. Sun, Tony, Kellie Webster, Apurva Shah, William Yang Wang, and Melvin Johnson. 2021. They, them, theirs: Rewriting with gender-neutral english. CoRR, abs/2102.06788. find
  111. Tan, Fiona Anting, Devamanyu Hazarika, See-Kiong Ng, Soujanya Poria, and Roger Zimmermann. 2021a. Causal augmentation for causal sentence classification. In Proceedings of the First Workshop on Causal Inference and NLP, pages 1–20, Punta Cana, Dominican Republic. Association for Computational Lingui… find
  112. Tan, Samson and Shafiq Joty. 2021. Code-mixing on sesame street: Dawn of the adversarial polyglots. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3596–3616, Online. Association for Computationa… find
  113. Tan, Samson, Shafiq Joty, Kathy Baxter, Araz Taeihagh, Gregory A. Bennett, and Min-Yen Kan. 2021b. Reliability testing for natural language processing systems. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Na… find
  114. Tan, Samson, Shafiq Joty, Min-Yen Kan, and Richard Socher. 2020. It’s morphin’ time! Combating linguistic discrimination with inflectional perturbations. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2920–2935, Online. Association for Computational… find
  115. Tiedemann, Jörg and Santhosh Thottingal. 2020. OPUS-MT Building open translation services for the World. In Proceedings of the 22nd Annual Conferenec of the European Association for Machine Translation (EAMT), Lisbon, Portugal. find
  116. Vijayakumar, Ashwin, Michael Cogswell, Ramprasaath Selvaraju, Qing Sun, Stefan Lee, David Crandall, and Dhruv Batra. 2018. Diverse beam search for improved description of complex scenes. find
  117. Vijayakumar, Ashwin K., Michael Cogswell, Ramprasaath R. Selvaraju, Qing Sun, Stefan Lee, David J. Crandall, and Dhruv Batra. 2016. Diverse beam search: Decoding diverse solutions from neural sequence models. CoRR, abs/1610.02424. find our card
  118. Wang, Alex, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019a. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenRev… find our card
  119. Wang, Xiao, Qin Liu, Tao Gui, Qi Zhang, Yicheng Zou, Xin Zhou, Jiacheng Ye, Yongxin Zhang, Rui Zheng, Zexiong Pang, Qinzhuo Wu, Zhengyan Li, Chong Zhang, Ruotian Ma, Zichu Fei, Ruijian Cai, Jun Zhao, Xingwu Hu, Zhiheng Yan, Yiding Tan, Yuan Hu, Qiyuan Bian, Zhihua Liu, Shan Qin, Bolin Zhu, Xiaoyu Xi… find
  120. Wang, Yuxuan, Wanxiang Che, Jiang Guo, Yijia Liu, and Ting Liu. 2019b. Cross-lingual bert transformation for zero-shot dependency parsing. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processi… find
  121. Wang, Yuxuan, Wanxiang Che, Ivan Titov, Shay B. Cohen, Zhilin Lei, and Ting Liu. 2021b. A closer look into the robustness of neural dependency parsers using better adversarial examples. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 2344–2354, Online. Associatio… find our card
  122. Wei, Jason W. and Kai Zou. 2019. EDA: easy data augmentation techniques for boosting performance on text classification tasks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJ… find our card
  123. Wieting, John and Kevin Gimpel. 2017. Pushing the limits of paraphrastic sentence embeddings with millions of machine translations. In arXiv preprint arXiv:1711.05732. — [arXiv:1711.05732](https://arxiv.org/abs/1711.05732) find
  124. Wieting, John, Jonathan Mallinson, and Kevin Gimpel. 2017. Learning paraphrastic sentence embeddings from back-translated bitext. In Proceedings of Empirical Methods in Natural Language Processing. find
  125. Yorke, Alex. butter-fingers. https://github.com/ alexyorke/butter-fingers. ↗
  126. Williams, Adina, Nikita Nangia, and Samuel R Bowman. 2017. A broad-coverage challenge corpus for sentence understanding through inference. arXiv preprint arXiv:1704.05426. find
  127. Yunfei. Chinese-Names-Corpus . https://github.com/wainshine/Chinese-Names-Corpus. ↗
  128. Wilson, Steven, Walid Magdy, Barbara McGillivray, Kiran Garimella, and Gareth Tyson. 2020. Urban dictionary embeddings for slang NLP applications. In Proceedings of the 12th Language Resources and Evaluation Conference, pages 4764–4773, Marseille, France. European Language Resources Association. find
  129. Wiseman, Sam and Alexander M. Rush. 2016. Sequence-to-sequence learning as beam-search optimization. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1296–1306, Austin, Texas. Association for Computational Linguistics. find
  130. Wolf, Thomas, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lh… find our card
  131. Wu, Tongshuang, Marco Tulio Ribeiro, Jeffrey Heer, and Daniel S Weld. 2021. Polyjuice: Generating counterfactuals for explaining, evaluating, and improving models. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics. find
  132. Xie, Qizhe, Zihang Dai, Eduard Hovy, Thang Luong, and Quoc Le. 2020. Unsupervised data augmentation for consistency training. Advances in Neural Information Processing Systems, 33. find our card
  133. Xu, Liang, Qianqian Dong, Cong Yu, Yin Tian, Weitang Liu, Lu Li, and Xuanwei Zhang. 2020. Cluener2020: Fine-grained name entity recognition for chinese. arXiv preprint arXiv:2001.04351. find
  134. Yaseen, Usama and Stefan Langer. 2021. Data augmentation for low-resource named entity recognition using backtranslation. CoRR, abs/2108.11703. find
  135. Yi, Sheng Kung, Mark Steyvers, Michael Lee, and Matthew Dry. 2010. Wisdom of the crowds in minimum spanning tree problems. In Proceedings of the Annual Meeting of the Cognitive Science Society, volume 32. find
  136. Zhang, Jing, Bonggun Shin, Jinho D Choi, and Joyce C Ho. 2021. Smat: An attention-based deep learning solution to the automation of schema matching. In European Conference on Advances in Databases and Information Systems, pages 260–274. Springer. find
  137. Zhang, Wei Emma, Quan Z. Sheng, and Ahoud Abdul-rahmn F. Alhazmi. 2019a. Generating textual adversarial examples for deep learning models: A survey. CoRR, abs/1901.06796. — [arXiv:1901.06796](https://arxiv.org/abs/1901.06796) find
  138. Zhang, Yuan, Jason Baldridge, and Luheng He. 2019b. PAWS: paraphrase adversaries from word scrambling. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 20… find our card
  139. Zhao, Zhe, Hui Chen, Jinbin Zhang, Xin Zhao, Tao Liu, Wei Lu, Xi Chen, Haotang Deng, Qi Ju, and Xiaoyong Du. 2019. Uer: An open-source toolkit for pre-training models. find
  140. EMNLP-IJCNLP 2019, page 241. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/dhole-2023-nl-augmenter-framework-task-sensitive/