Elamur Research⌕
☰
home / papers / 2026 / Learning from Near-Misses: Error-Aware Contrastive Few-Shot Learning for NL2Formula

Learning from Near-Misses: Error-Aware Contrastive Few-Shot Learning for NL2Formula

Paper · 2026 · Zhihao Shuai, Yiyun Chen, Maolin Ma et al.

Summary

The paper addresses NL2Formula, the task of turning natural-language requests into Excel formulas. Its focus is "near-miss" outputs: formulas that parse correctly but give wrong results because one function, operator, reference or value is wrong. An error analysis showed these mistakes cluster around a few structural decision points in the formula's AST.

The authors therefore define a deterministic 9-type error taxonomy. It uses a skeleton-first comparison: function and operator nodes are checked before leaf-level references and values. On top of this they build ECFL (Error-Aware Contrastive Few-Shot Learning):

  • Error bank: Monte Carlo sampling collects the model's wrong predictions, each typed and paired with its correct formula.
  • Training: LoRA fine-tuning uses contrastive prompts with demonstrations matched by error type.
  • Inference: a lightweight predictor guesses which error types are likely and retrieves one demonstration per type before a single decoding pass.

The authors also clean the NL2Formula benchmark and release a new expert-built benchmark, TFBench. ECFL gains 6.4 EM points over SFT and reaches SC@5-level accuracy at much lower cost. This suggests typed error supervision can replace expensive multi-sample decoding.

Key points

  • The taxonomy has 6 skeleton-level types (Func/Op × Mismatch/Missing/Redundant), 2 fill-level types (RefMismatch, ValMismatch) and a Miscellaneous class. Of roughly 10,000 errors, only 19 involved missing or redundant references or values.
  • TFBench is expert-curated and covers 215 functions and 12 operators, including nested compositions.
  • In the ablation on NL2Formula, full ECFL reaches 75.32% EM. Using demonstrations only in training gives 68.52%, and only at inference gives 62.71%, which is below the SFT baseline.
  • Retrieval by predicted error type beats both similarity-only and fixed-type retrieval. ECFL also improves Llama-3.1-8B-Instruct, and error banks transferred to Qwen3-8B from another model family still beat SFT.
  • Code and data are available at github.com/VICA-Lab-HKUST-GZ/ECFL.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University College LondonLondon · United Kingdom88 works · 19,432 citations🎓The Hong Kong University of Science and Technology (Guangzhou)Guangzhou · China53 works · 1,367 citations🎓Southwest UniversityChongqing · China2 works · 19 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputational Theory and Mathematics · 252 works

Cite

Cite as Zhihao Shuai, Yiyun Chen, Maolin Ma, Yutong Chen, Hanjia Qiu, Jing Xu, Ziye Chen and Weikai Yang. 2026. Learning from Near-Misses: Error-Aware Contrastive Few-Shot Learning for NL2Formula. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.1368

@inproceedings{shuai2026learning,
  title = {{Learning from Near-Misses: Error-Aware Contrastive Few-Shot Learning for NL2Formula}},
  author = {Zhihao Shuai, Yiyun Chen, Maolin Ma, Yutong Chen, Hanjia Qiu, Jing Xu, Ziye Chen and Weikai Yang},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.1368},
  url = {https://doi.org/10.18653/v1/2026.acl-long.1368}
}

References

25 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Efthimia Aivaloglou, David Hoepelman, and Felienne Hermans. 2017. Parsing excel formulas: A grammar and its application on 4 large datasets. Journal of Software: Evolution and Process, 29(12):e1895. find
  2. Qing Chang and Zhiming Hu. 2026. Gazeinterpreter: Parsing eye gaze to generate eye-body-coordinated narrations. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 40, pages 17410– 17418. find
  3. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, and 39 others. 202… — Evaluating Large Language Models Trained on Code
  4. Sibei Chen, Yeye He, Weiwei Cui, Ju Fan, Song Ge, Haidong Zhang, Dongmei Zhang, and Surajit Chaudhuri. 2024. Auto-formula: Recommend formulas in spreadsheets using contrastive learning for table representations. Proceedings of the ACM on Management of Data, 2(3):1–27. find
  5. Xinyun Chen, Maxwell Lin, Nathanael Schärli, and Denny Zhou. 2023. Teaching large language models to self-debug. Preprint, arXiv:2304.05128. find
  6. Xinyun Chen, Petros Maniatis, Rishabh Singh, Charles Sutton, Hanjun Dai, Max Lin, and Denny Zhou. 2021b. Spreadsheetcoder: Formula prediction from semi-structured context. In International Conference on Machine Learning, pages 1661–1672. PMLR. find
  7. Tianyu Gao, Xingcheng Yao, and Danqi Chen. 2021. Simcse: Simple contrastive learning of sentence embeddings. In Proceedings of the 2021 conference on empirical methods in natural language processing, pages 6894–6910.
  8. Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021. Lora: Low-rank adaptation of large language models. Preprint, arXiv:2106.09685.
  9. Harshit Joshi, Abishai Ebenezer, José Cambronero Sanchez, Sumit Gulwani, Aditya Kanade, Vu Le, Ivan Radiček, and Gust Verbruggen. 2024. Flame: a small language model for spreadsheet formulas. In Proceedings of the Thirty-Eighth AAAI Conference on Artificial Intelligence and Thirty-Sixth Conference… find
  10. Boyan Li, Jiayi Zhang, Ju Fan, Yanwei Xu, Chong Chen, Nan Tang, and Yuyu Luo. 2025. Alpha-SQL: Zero-shot text-to-SQL using monte carlo tree search. In Forty-second International Conference on Machine Learning. find
  11. Tianqi Luo, Chuhan Huang, Leixian Shen, Boyan Li, Shuyu Shen, Wei Zeng, Nan Tang, and Yuyu Luo. 2025. nvbench 2.0: Resolving ambiguity in text-to-visualization through stepwise reasoning. Preprint, arXiv:2503.12880. — [arXiv:2503.12880](https://arxiv.org/abs/2503.12880) find
  12. Ansong Ni, Srini Iyer, Dragomir Radev, Veselin Stoyanov, Wen-tau Yih, Sida Wang, and Xi Victoria Lin. 2023. Lever: Learning to verify language-to-code generation with execution. In International Conference on Machine Learning, pages 26106–26128. PMLR.
  13. Stephen G Powell, Kenneth R Baker, and Barry Lawson. 2008. A critical review of the literature on spreadsheet errors. Decision Support Systems, 46(1):128– 138. find
  14. Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D Manning, and Chelsea Finn. 2023. Direct preference optimization: your language model is secretly a reward model. In Proceedings of the 37th International Conference on Neural Information Processing Systems, pages 53728–53741…
  15. Baptiste Rozière, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Romain Sauvestre, Tal Remez, Jérémy Rapin, Artyom Kozhevnikov, Ivan Evtimov, Joanna Bitton, Manish Bhatt, Cristian Canton Ferrer, Aaron Grattafiori, Wenhan Xiong, Alexandre Défossez, a… — Code Llama: Open Foundation Models for Code
  16. Freda Shi, Daniel Fried, Marjan Ghazvininejad, Luke Zettlemoyer, and Sida I Wang. 2022. Natural language to code translation with execution. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 3533–3546. find
  17. Zhihao Shuai, Boyan Li, Siyu Yan, Yuyu Luo, and Weikai Yang. 2026. Deepvis: Bridging natural language and data visualization through step-wise reasoning. IEEE Transactions on Visualization and Computer Graphics, 32(1):868–878. find
  18. Sruti Srinivasa Ragavan, Zhitao Hou, Yun Wang, Andrew D Gordon, Haidong Zhang, and Dongmei Zhang. 2022. Gridbook: Natural language formulas for the spreadsheet grid. In Proceedings of the 27th International Conference on Intelligent User Interfaces, pages 345–368. find
  19. Wenhui Wang, Furu Wei, Li Dong, Hangbo Bao, Nan Yang, and Ming Zhou. 2020. Minilm: deep self-attention distillation for task-agnostic compression of pre-trained transformers. In Proceedings of the 34th International Conference on Neural Information Processing Systems, pages 5776–5788. find
  20. Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2023. Self-consistency improves chain of thought reasoning in language models. Preprint, arXiv:2203.11171.
  21. Yue Wang, Weishi Wang, Shafiq Joty, and Steven CH Hoi. 2021. Codet5: Identifier-aware unified pre-trained encoder-decoder models for code understanding and generation. In Proceedings of the 2021 conference on empirical methods in natural language processing, pages 8696–8708. find
  22. Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L Griffiths, Yuan Cao, and Karthik Narasimhan. 2023. Tree of thoughts: deliberate problem solving with large language models. In Proceedings of the 37th International Conference on Neural Information Processing Systems, pages 11809–11822. find
  23. Haocheng Yuan, Jing Xu, Hao Pan, Adrien Bousseau, Niloy J Mitra, and Changjian Li. 2024. CADTalk: An algorithm and benchmark for semantic commenting of cad programs. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3753–3762. find
  24. Kechi Zhang, Zhuo Li, Jia Li, Ge Li, and Zhi Jin. 2023. Self-edit: Fault-aware code editor for code generation. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 769–787. find
  25. Wei Zhao, Zhitao Hou, Siyuan Wu, Yan Gao, Haoyu Dong, Yao Wan, Hongyu Zhang, Yulei Sui, and Haidong Zhang. 2024. Nl2formula: Generating spreadsheet formulas from natural language queries. In Findings of the Association for Computational Linguistics: EACL 2024, pages 2377–2388. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/shuai-2026-learning-near-misses-error-aware/