Elamur Research⌕
☰
home / papers / 2021 / Towards Robustness of Text-to-SQL Models against Synonym Substitution

Towards Robustness of Text-to-SQL Models against Synonym Substitution

Paper · 2021 · Yujian Gan, Xinyun Chen, Qiuping Huang et al.

Summary

Cross-domain text-to-SQL systems score well on benchmarks such as Spider. Much of that success, however, comes from schema linking, which matches question words literally against table and column names. Gan et al. examine what happens when users refer to schema items or cell values with different but equivalent wording. To measure this, they built Spider-Syn, a manually curated variant of Spider. In it, schema-related words in the questions are replaced with common, domain-appropriate synonyms or multi-word phrases, while the gold SQL stays unchanged. Models trained only on the original Spider lose a large share of their accuracy on these questions, even though the substitutions were not designed to attack any particular model. The authors test two defenses. The first, multi-annotation selection (MAS), adds synonym annotations to the schema at input time without retraining. The second is adversarial training on synonym-perturbed questions. Both improve robustness, and MAS is the more effective and cheaper of the two. The work matters because real users rarely know exact schema names, so benchmarks built on lexical overlap overstate how robust these models are in practice.

Key points

  • Dataset size: Spider-Syn mirrors Spider's splits, with 7000 training and 1034 development examples. It has no test set because the Spider test set is not public. Code and data are released at github.com/ygan/Spider-Syn.
  • Annotation process: Four computer-science graduate students annotated the data domain by domain, followed by two rounds of consensus review. The words id, age, name and year were treated as reserved and left unchanged.
  • Accuracy drop: In Table 7, GNN falls from 48.5% on Spider to 23.6% on Spider-Syn, and IRNet falls from 53.2% to 28.4%.
  • MAS variants: ManualMAS uses human synonym annotations and lifts Spider-Syn accuracy to 38.2% for GNN and 39.3% for IRNet. AutoMAS uses automatically generated synonyms and mainly helps against the ADV GLOVE and ADV BERT worst-case attacks. For example, IRNet improves from 26.4% to 44.3% on ADV GLOVE.
  • Trade-offs: MAS needs no extra training and can be combined with adversarially trained models. It costs some accuracy on the original Spider questions, for example GNN drops from 48.5% to 44.0%. Similar trends were reported for RAT-SQL.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of California, BerkeleyBerkeley · United States127 works · 212,523 citations🎓Queen Mary University of LondonLondon · United Kingdom39 works · 3,307 citations

Also on the byline: People's Bank of China · Guangxi University of Finance and Economics.

📚 Shelves 4 Computer Science · 9,086 works Artificial Intelligence · 8,889 works Arts and Humanities · 9,296 works Language and Linguistics · 9,206 works

🧭 Topics 2 Semantic parsing & meaning representation · 135 works Code & programming languages · 55 works

Cite

Cite as Yujian Gan, Xinyun Chen, Qiuping Huang, Matthew Purver, John R. Woodward, Jinxia Xie and Pengsheng Huang. 2021. Towards Robustness of Text-to-SQL Models against Synonym Substitution. Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2021.acl-long.195

@inproceedings{gan2021robustness,
  title = {{Towards Robustness of Text-to-SQL Models against Synonym Substitution}},
  author = {Yujian Gan, Xinyun Chen, Qiuping Huang, Matthew Purver, John R. Woodward, Jinxia Xie and Pengsheng Huang},
  booktitle = {Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)},
  year = {2021},
  doi = {10.18653/v1/2021.acl-long.195},
  url = {https://doi.org/10.18653/v1/2021.acl-long.195}
}

References

36 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Basemah Alshemali and Jugal Kalita. 2020. Generalization to Mitigate Synonym Substitution Attacks. find
  2. In Proceedings of Deep Learning Inside Out (Dee-LIO): The First Workshop on Knowledge Extraction and Integration for Deep Learning Architectures, pages 20–28, Online. Association for Computational Linguistics. find
  3. Moustafa Alzantot, Yash Sharma, Ahmed Elgohary, Bo-Jhang Ho, Mani Srivastava, and Kai-Wei Chang. 2018. Generating natural language adversarial examples. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2890–2896. our card
  4. Ana-Maria Popescu, Oren Etzioni, and Henry Kautz. 2003. Towards a Theory of Natural Language Interfaces to Databases. In Proceedings of the 8th International Conference on Intelligent User Interfaces, pages 149–157. find
  5. Ben Bogin, Jonathan Berant, and Matt Gardner. 2019a. Representing schema structure with graph neural networks for text-to-SQL parsing. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4560–4565, Florence, Italy. Association for Computational Linguisti… our card
  6. Ben Bogin, Matt Gardner, and Jonathan Berant. 2019b. Global Reasoning over Database Structures for Text-to-SQL Parsing. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP),… find
  7. Javid Ebrahimi, Anyi Rao, Daniel Lowd, and Dejing Dou. 2018. HotFlip: White-box adversarial examples for text classification. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 31–36, Melbourne, Australia. Association for Comput… our card
  8. Alessandra Giordani and Alessandro Moschitti. 2012. Automatic Generation and Reranking of SQL-derived Answers to NL Questions. In Proceedings of the Second International Conference on Trustworthy Eternal Systems via Evolving Software, Data and Knowledge, pages 59–76. find
  9. Di Jin, Zhijing Jin, Joey Tianyi Zhou, and Peter Szolovits. 2020. Is bert really robust? a strong baseline for natural language attack on text classification and entailment. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 8018–8025. our card
  10. Winfred Philipp Lehmann and RA Stachowitz. 1972. Normalization of natural language for information retrieval. Technical report, TEXAS UNIV AUSTIN LINGUISTICS RESEARCH CENTER. find
  11. Fei Li and H. V. Jagadish. 2014. Constructing an interactive natural language interface for relational databases. Proceedings of the VLDB Endowment, 8(1):73–84. find
  12. Jingjing Li, Wenlu Wang, Wei Shinn Ku, Yingtao Tian, and Haixun Wang. 2019. SpatialNLI: A spatial domain natural language interface to databases using spatial comprehension. In GIS: Proceedings of the ACM International Symposium on Advances in Geographic Information Systems, pages 339–348, New York,… find
  13. Linyang Li, Ruotian Ma, Qipeng Guo, Xiangyang Xue, and Xipeng Qiu. 2020. BERT-ATTACK: Adversarial Attack Against BERT Using BERT. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6193–6202, Stroudsburg, PA, USA. Association for Computational Li… our card
  14. Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. 2018. Towards deep learning models resistant to adversarial attacks. In International Conference on Learning Representations. our card
  15. John Morris, Eli Lifland, Jin Yong Yoo, Jake Grigsby, Di Jin, and Yanjun Qi. 2020. TextAttack: A Framework for Adversarial Attacks, Data Augmentation, and Adversarial Training in NLP. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pa… our card
  16. Jiaqi Guo, Zecheng Zhan, Yan Gao, Yan Xiao, Jian-Guang Lou, Ting Liu, and Dongmei Zhang. 2019. Towards Complex Text-to-SQL in Cross-Domain Database with Intermediate Representation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4524– 4535, Florence… find
  17. Nikola Mrkšić, Diarmuid Ó Séaghdha, Blaise Thomson, Milica Gašić, Lina M. Rojas-Barahona, Pei-Hao Su, David Vandyke, Tsung-Hsien Wen, and Steve Young. 2016. Counter-fitting word vectors to linguistic constraints. In Proceedings of the 2016 Conference of the North American Chapter of the Associ… find
  18. Srinivasan Iyer, Ioannis Konstas, Alvin Cheung, Jayant Krishnamurthy, and Luke Zettlemoyer. 2017. Learning a Neural Semantic Parser from User Feedback. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 963–973. find
  19. Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. Glove: Global vectors for word representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1532–1543, Doha, Qatar. Association for Computational Linguistics. complex and cr… our card
  20. Karthik Radhakrishnan, Arvind Srikantan, and Xi Victoria Lin. 2020. ColloQL: Robust Cross-Domain Text-to-SQL Over Search Queries. find
  21. Shuhuai Ren, Yihe Deng, Kun He, and Wanxiang Che. 2019. Generating Natural Language Adversarial Examples through Probability Weighted Word Saliency. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1085–1097, Florence, Italy. Association for Computati… our card
  22. Georgios Rizos, Konstantin Hemker, and Björn Schuller. 2019. Augment to prevent: Short-text data augmentation in deep learning for hate-speech classification. In Proceedings of the 28th ACM International Conference on Information and Knowledge Management, CIKM ’19, page 991–1000, New York, NY, USA.… find
  23. Lappoon R Tang and Raymond J Mooney. 2000. Automated Construction of Database Interfaces: Intergrating Statistical and Relational Learning for Semantic Parsing. In 2000 Joint SIGDAT Conference on Empirical Methods in Natural Language Processing and Very Large Corpora, pages 133–141. find
  24. David L Waltz. 1978. An english language question answering system for a large relational database. Communications of the ACM, 21(7):526–539. find
  25. Bailin Wang, Richard Shin, Xiaodong Liu, Oleksandr Polozov, and Matthew Richardson. 2020. RAT-SQL: Relation-Aware Schema Encoding and Linking for Text-to-SQL Parsers. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7567–7578, Online. Association for… our card
  26. Jason Wei and Kai Zou. 2019. EDA: Easy data augmentation techniques for boosting performance on text classification tasks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP… our card
  27. Hongvu Xiong and Ruixiao Sun. 2019. Transferable Natural Language Interface to Structured Queries Aided by Adversarial Generation. In 2019 IEEE 13th International Conference on Semantic Computing (ICSC), pages 255–262. IEEE. find
  28. Tao Yu, Rui Zhang, Kai Yang, Michihiro Yasunaga, Dongxu Wang, Zifan Li, James Ma, Irene Li, Qingning Yao, Shanelle Roman, Zilin Zhang, and Dragomir Radev. 2018b. Spider: A large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to-SQL task. In Proceedings of the 2018… our card
  29. John M Zelle and Raymond J Mooney. 1996. Learning to Parse Database Queries Using Inductive Logic Programming. In Proceedings of the Thirteenth National Conference on Artificial Intelligence - Volume 2, pages 1050–1055. our card
  30. Jichuan Zeng, Xi Victoria Lin, Steven C.H. Hoi, Richard Socher, Caiming Xiong, Michael Lyu, and Irwin King. 2020. Photon: A Robust Cross-Domain Text-to-SQL System. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 204–214, Stroud… find
  31. Huangzhao Zhang, Hao Zhou, Ning Miao, and Lei Li. 2019a. Generating fluent adversarial examples for natural languages. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5564–5569, Florence, Italy. Association for Computational Linguistics. find
  32. Rui Zhang, Tao Yu, He Yang Er, Sungrok Shim, Eric Xue, Xi Victoria Lin, Tianze Shi, Caiming Xiong, Richard Socher, and Dragomir Radev. 2019b. Editing-Based SQL Query Generation for Cross-Domain Context-Dependent Questions. our card
  33. Victor Zhong, Caiming Xiong, and Richard Socher. 2017. Seq2SQL: Generating Structured Queries from Natural Language using Reinforcement Learning. CoRR, abs/1709.0. find
  34. Yi Zhu, Yiwei Zhou, and Menglin Xia. 2020. Generating Semantically Valid Adversarial Questions for TableQA. find
  35. Navid Yaghmazadeh, Yuepeng Wang, Isil Dillig, and Thomas Dillig. 2017. SQLizer: Query Synthesis from Natural Language. In International Conference on Object-Oriented Programming, Systems, Languages, and Applications, ACM, pages 63:1—-63:26. find
  36. Tao Yu, Michihiro Yasunaga, Kai Yang, Rui Zhang, Dongxu Wang, Zifan Li, and Dragomir Radev. 2018a. SyntaxSQLNet: Syntax tree networks for find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/gan-2021-robustness-text-sql-models-against/