Elamur Research⌕
☰
home / papers / 2020 / DAGA: Data Augmentation with a Generation Approach for Low-resource Tagging Tasks

DAGA: Data Augmentation with a Generation Approach for Low-resource Tagging Tasks

Paper · 2020 · Bosheng Ding, Linlin Liu, Lidong Bing et al.

Summary

Sequence tagging tasks label every token, so they are easily hurt by noisy augmented data. Common augmentation strategies also depend on resources that low-resource languages often lack, such as WordNet, bilingual corpora or in-domain weak taggers. DAGA addresses this by generating new labeled sentences from scratch.

Each gold sentence is flattened into one token stream by placing tags next to their words. Frequent O tags are dropped, and [BOS]/[EOS] markers are added. A small one-layer LSTM language model is trained on these sequences and then sampled to produce new sentences that already carry tags. In semi-supervised settings, a conditional generation scheme also lets the model learn from unlabeled text and from sentences auto-annotated with a knowledge base.

Experiments cover NER, POS tagging and E2E-TBSA. DAGA consistently beats the baselines, and the gains are largest when gold data is scarce. Analysis shows that the synthetic data adds many new entity–context combinations, which reduces overfitting. The approach is cheap, needs no large pretrained model, and suits low-resource tagging.

Key points

  • Labeled sentences are linearized with tags inserted before (or after) their words, and O tags are removed for NER and E2E-TBSA. A one-layer LSTM RNNLM in the style of Sundermeyer et al. (2012) is trained on this data and sampled autoregressively, starting from [BOS].
  • The supervised evaluation covers NER, POS tagging and E2E-TBSA, where tags such as B-POS/E-NEG jointly mark targets and their sentiment. The semi-supervised evaluation is on NER, using unlabeled Wikipedia sentences, optionally combined with a knowledge base.
  • The knowledge base contains countries and cities (world-cities data from datahub.io) and labels unlabeled text by longest left-to-right matching (kb). The kb baseline is strong with very little gold data, but DAGA's gen kb variant overtakes it once the gold set exceeds about 2k sentences.
  • In the knowledge-base setup, 20k sampled synthetic sentences are added to the gold data, which is oversampled during training.
  • An analysis of contextualized entities (an entity plus its neighbouring words) shows that DAGA produces many novel combinations, such as new surnames paired with "Sandrine". A higher ratio of new combinations corresponds to larger F1 gains. The code is released at ntunlpsg.github.io/project/daga/.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢Alibaba GroupHangzhou · China64 works · 14,113 citations🎓Nanyang Technological UniversitySingapore · Singapore73 works · 24,390 citations🏢Alibaba DAMO AcademyHangzhou · China28 works · 6,115 citations🔬National Institute of InformaticsTokyo · Japan21 works · 3,045 citations🏢Alibaba Group (Cayman Islands)George Town · Cayman Islands15 works · 3,209 citations

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 9,086 worksArtificial Intelligence · 8,889 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 works

🧭 Topics 1Research subjects and languages the work is about.Low-resource languages · 166 works

Cite

Cite as Bosheng Ding, Linlin Liu, Lidong Bing, Canasai Kruengkrai, Thien Hai Nguyen, Shafiq Joty, Luo Si and Chunyan Miao. 2020. DAGA: Data Augmentation with a Generation Approach for Low-resource Tagging Tasks. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2020.emnlp-main.488

@inproceedings{ding2020daga,
  title = {{DAGA: Data Augmentation with a Generation Approach for Low-resource Tagging Tasks}},
  author = {Bosheng Ding, Linlin Liu, Lidong Bing, Canasai Kruengkrai, Thien Hai Nguyen, Shafiq Joty, Luo Si and Chunyan Miao},
  booktitle = {Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)},
  year = {2020},
  doi = {10.18653/v1/2020.emnlp-main.488},
  url = {https://doi.org/10.18653/v1/2020.emnlp-main.488}
}

References

44 references, 23 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Alan Akbik, Tanja Bergmann, Duncan Blythe, Kashif Rasul, Stefan Schweter, and Roland Vollgraf. 2019. Flair: An easy-to-use framework for state-of-the-art nlp. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (Demonstrations), pages…
  2. Alan Akbik, Duncan Blythe, and Roland Vollgraf. 2018. Contextual string embeddings for sequence labeling. In Proceedings of the 27th International Conference on Computational Linguistics, pages 1638–1649.
  3. Ateret Anaby-Tavor, Boaz Carmeli, Esther Goldbraich, Amir Kantor, George Kour, Segev Shlomov, Naama Tepper, and Naama Zwerdling. 2020. Do not have enough data? deep learning to the rescue! In Thirty-Third AAAI Conference on Artificial Intelligence. find
  4. M Saiful Bari, Tasnim Mohiuddin, and Shafiq Joty. 2020. Multimix: A robust data augmentation framework for cross-lingual nlp. find
  5. Bernd Bohnet, Ryan McDonald, Gonçalo Simões, Daniel Andor, Emily Pitler, and Joshua Maynez. 2018. Morphosyntactic tagging with a meta-bilstm model over context sensitive token encodings. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pape… find
  6. Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2017. Enriching word vectors with subword information. Transactions of the Association for Computational Linguistics, 5:135–146.
  7. Peng Chen, Zhongqian Sun, Lidong Bing, and Wei Yang. 2017. Recurrent attention network on memory for aspect sentiment analysis. In Proceedings of EMNLP, pages 463–472.
  8. Li Dong, Jonathan Mallinson, Siva Reddy, and Mirella Lapata. 2017. Learning to paraphrase for question answering. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 875–886, Copenhagen, Denmark. Association for Computational Linguistics.
  9. Antonio D’Innocente, Fabio Maria Carlucci, Mirco Colosi, and Barbara Caputo. 2017. Bridging between computer and robot vision through data augmentation: a case study on object recognition. In International Conference on Computer Vision Systems, pages 384–393. Springer. find
  10. Marzieh Fadaee, Arianna Bisazza, and Christof Monz. 2017. Data augmentation for low-resource neural machine translation. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 567– 573.
  11. Alhussein Fawzi, Horst Samulowitz, Deepak Turaga, and Pascal Frossard. 2016. Adaptive data augmentation for image classification. In 2016 IEEE International Conference on Image Processing (ICIP), pages 3688–3692. Ieee. find
  12. Benjamin Heinzerling and Michael Strube. 2019. Sequence tagging with contextual and non-contextual subword representations: A multilingual evaluation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 273– 291. find
  13. Nitish Shirish Keskar, Bryan McCann, Lav R Varshney, Caiming Xiong, and Richard Socher. 2019. Ctrl: A conditional transformer language model for controllable generation. arXiv preprint arXiv:1909.05858.
  14. Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980. find
  15. Tom Ko, Vijayaditya Peddinti, Daniel Povey, Michael L Seltzer, and Sanjeev Khudanpur. 2017. A study on data augmentation of reverberant speech for robust speech recognition. In 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 5220–5224. IEEE. find
  16. Sosuke Kobayashi. 2018. Contextual augmentation: Data augmentation by words with paradigmatic relations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 452–457.
  17. Canasai Kruengkrai. 2019. Better exploiting latent variables in text modeling. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5527–5532. find
  18. Varun Kumar, Ashutosh Choudhary, and Eunah Cho. 2020. Data augmentation using pre-trained transformer models. arXiv preprint arXiv:2003.02245.
  19. Guillaume Lample, Miguel Ballesteros, Sandeep Subramanian, Kazuya Kawakami, and Chris Dyer. 2016. Neural architectures for named entity recognition. arXiv preprint arXiv:1603.01360.
  20. Jing Li, Aixin Sun, Jianglei Han, and Chenliang Li. 2020. A survey on deep learning for named entity recognition. IEEE Transactions on Knowledge and Data Engineering. find
  21. Xin Li, Lidong Bing, Piji Li, and Wai Lam. 2019a. A unified model for opinion target extraction and target sentiment prediction. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 6714–6721. find
  22. Xin Li, Lidong Bing, Piji Li, Wai Lam, and Zhimou Yang. 2018. Aspect term extraction with history attention and selective transformation. In Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence (IJCAI), pages 4194– 4200. find
  23. Xin Li, Lidong Bing, Wenxuan Zhang, and Wai Lam. 2019b. Exploiting BERT for end-to-end aspect-based sentiment analysis. In Proceedings of the 5th Workshop on Noisy User-generated Text, W-NUT@EMNLP, pages 34–41.
  24. Pengfei Liu, Shafiq Joty, and Helen Meng. 2015. Fine-grained opinion mining with recurrent neural networks and word embeddings. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1433–1443, Lisbon, Portugal. Association for Computational Linguistics.
  25. Christopher D. Manning, Mihai Surdeanu, John Bauer, Jenny Finkel, Steven J. Bethard, and David Mc-Closky. 2014. The Stanford CoreNLP natural language processing toolkit. In Association for Computational Linguistics (ACL) System Demonstrations, pages 55–60.
  26. Joel Mathew, Shobeir Fakhraei, and José Luis Ambite. 2019. Biomedical named entity recognition via reference-set augmented bootstrapping. arXiv preprint arXiv:1906.00282. — [arXiv:1906.00282](https://arxiv.org/abs/1906.00282) find
  27. Andrei Mikheev, Marc Moens, and Claire Grover. 1999. Named entity recognition without gazetteers. In Proceedings of the ninth conference on European chapter of the Association for Computational Linguistics, pages 1–8. Association for Computational Linguistics.
  28. George A Miller. 1995. Wordnet: a lexical database for english. Communications of the ACM, 38(11):39– 41.
  29. Barbara Plank, Anders Søgaard, and Yoav Goldberg. 2016. Multilingual part-of-speech tagging with bidirectional long short-term memory models and auxiliary loss. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 412–418, Berlin,…
  30. Maria Pontiki, Dimitris Galanis, Haris Papageorgiou, Ion Androutsopoulos, Suresh Manandhar, Mohammad AL-Smadi, Mahmoud Al-Ayyoub, Yanyan Zhao, Bing Qin, Orphée De Clercq, Véronique Hoste, Marianna Apidianaki, Xavier Tannier, Natalia Loukachevitch, Evgeniy Kotelnikov, Nuria Bel, Salud Marı́a Jimén…
  31. Maria Pontiki, Dimitris Galanis, Haris Papageorgiou, Suresh Manandhar, and Ion Androutsopoulos. 2015. SemEval-2015 task 12: Aspect based sentiment analysis. In Proceedings of the 9th International Workshop on Semantic Evaluation (SemEval 2015), pages 486–495, Denver, Colorado. Association for Comput…
  32. Maria Pontiki, Dimitris Galanis, John Pavlopoulos, Harris Papageorgiou, Ion Androutsopoulos, and Suresh Manandhar. 2014. SemEval-2014 task 4: Aspect based sentiment analysis. In Proceedings of the 8th International Workshop on Semantic Evaluation (SemEval 2014), pages 27–35, Dublin, Ireland. Associa…
  33. Guillaume Raille, Sandra Djambazovska, and Claudiu Musat. 2020. Fast cross-domain data augmentation through neural sentence editing. arXiv preprint arXiv:2003.10254. — [arXiv:2003.10254](https://arxiv.org/abs/2003.10254) find
  34. Erik F Sang and Fien De Meulder. 2003. Introduction to the conll-2003 shared task: Language-independent named entity recognition. arXiv preprint cs/0306050.
  35. Jan Schlüter and Thomas Grill. 2015. Exploring data augmentation for improved singing voice detection with neural networks. In ISMIR, pages 121–126. find
  36. Hinrich Schütze. 1993. Part-of-speech induction from scratch. In Proceedings of the 31st annual meeting on Association for Computational Linguistics, pages 251–258. Association for Computational Linguistics. find
  37. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Improving neural machine translation models with monolingual data. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 86–96, Berlin, Germany. Association for Computational L… find
  38. Iulian Vlad Serban, Alberto Garcia-Duran, Caglar Gulcehre, Sungjin Ahn, Sarath Chandar, Aaron Courville, and Yoshua Bengio. 2016. Generating factoid questions with recurrent neural networks: The 30m factoid question-answer corpus. In Proceedings of the 54th Annual Meeting of the Association for Comp… find
  39. Jingbo Shang, Jialu Liu, Meng Jiang, Xiang Ren, Clare R Voss, and Jiawei Han. 2018. Automated phrase mining from massive text corpora. IEEE Transactions on Knowledge and Data Engineering, 30(10):1825–1837. find
  40. Patrice Y. Simard, Yann A. LeCun, John S. Denker, and Bernard Victorri. 1998. Transformation Invariance in Pattern Recognition — Tangent Distance and Tangent Propagation, pages 239–274. Springer Berlin Heidelberg, Berlin, Heidelberg. find
  41. Martin Sundermeyer, Ralf Schlüter, and Hermann Ney. 2012. Lstm neural networks for language modeling. In Thirteenth annual conference of the international speech communication association. find
  42. Erik F. Tjong Kim Sang. 2002. Introduction to the CoNLL-2002 shared task: Language-independent named entity recognition. In COLING-02: The 6th Conference on Natural Language Learning 2002 (CoNLL-2002).
  43. Erik F. Tjong Kim Sang and Fien De Meulder. 2003. Introduction to the CoNLL-2003 shared task: Language-independent named entity recognition. In Proceedings of the Seventh Conference on Natural Language Learning at HLT-NAACL 2003, pages 142–147.
  44. Dingquan Wang and Jason Eisner. 2016. The galactic dependencies treebanks: Getting more data by synthesizing new languages. Transactions of the Association for Computational Linguistics, 4:491–505.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/ding-2020-daga-data-augmentation-generation-approach/