Improving Text-to-SQL Evaluation Methodology

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/P18-1033.pdf

Summary

The paper argues that common practices for evaluating text-to-SQL systems overstate how well models generalize. The authors cleaned and unified seven existing corpora, giving them a consistent SQL style, removing duplicate queries, fixing annotation mistakes and marking query variables by hand. They also contribute Advising, a new dataset of student questions about University of Michigan courses, which require especially complex SQL. Comparing human-written with automatically generated questions, they find the human-written ones demand richer queries. They also identify a flaw in conventional data splits. When splitting by question, most test queries also appear in training with only different wording and entities. A simple template-filling baseline exploits this and still rivals published systems. Under their proposed query-based split, where each SQL query appears on only one side, strong models perform much worse. Finally, they show that anonymizing variables hides a real source of ambiguity. Together, the released data and guidelines support more honest benchmarking of NLIDB systems.

Key points

• Resources are released at github.com/jkkummerfeld/text2sql-data. They include standardized versions of datasets such as ATIS, GeoQuery, Restaurants, Scholar, Academic, Yelp, IMDB and WikiSQL, plus the new Advising corpus.
• Advising has 4,570 questions mapped to only 211 unique queries. By contrast, WikiSQL has 80,654 questions and 77,840 unique queries, yet those queries are much simpler (one table, no nesting).
• Manual deduplication grouped equivalent questions that had been given different SQL:
  • ATIS: 380 affected queries in 141 sets
  • GeoQuery: 39 affected queries in 17 sets
  • Scholar: 152 affected queries in 60 sets
• Canonicalization sorted fields, tables and constraints alphabetically and standardized table aliases (e.g. RIVERalias0) and formatting.
• The authors recommend reporting results on both question-based and query-based splits, and on non-anonymized data. When applied to logical forms for ATIS and GeoQuery, the query-based split had only a limited effect.