Elamur Research⌕
☰
home / papers / 2026 / Asking language models how to represent data for fine-tuning

Asking language models how to represent data for fine-tuning

Paper · 2026 · Usneek Singh, Ananya Singha, Abhijeet Awasthi et al.

Summary

When small language models handle tables, database schemas or graphs, the structure must first be turned into text. Developers usually pick that serialization by intuition or through costly repeated fine-tuning runs. Singh et al. ask whether the choice still matters once a model is fine-tuned, and whether it can be made cheaply.

They test six tasks with Phi-3, Mistral and CodeLlama, each adapted with LoRA:

  • NL-to-pandas code (Arcade)
  • NL-to-Excel formulas
  • text-to-SQL (Spider)
  • three graph problems: cycle detection, flow estimation and shortest path

To find candidate formats, they truncate a prompt just before the data structure, let the base model fill it in, and classify the completions with regular expressions.

Fine-tuning does not make models indifferent to format, since some serializations stay clearly stronger. More importantly, how formats rank under few-shot inference on the base model largely carries over after fine-tuning. A format can therefore be selected through inference alone, which saves substantial compute for anyone fine-tuning on structured inputs.

Key points

  • Models and training: Phi-3-mini-4k-instruct (3.8B), Mistral-7B-Instruct-v0.2 and CodeLlama-7b-hf were trained with LoRA (rank 64) for 10 epochs on one A100 GPU. Results are scored with pass@5.
  • Format gaps after fine-tuning: The spread between best and worst format averages 0.03 pass@5 on text-to-code tasks and 0.33 on graph tasks. It reaches up to 0.50 in the largest case.
  • Base model predicts the winner: In 16 of 18 model–task settings, the best format on the base model stays among the top candidates after fine-tuning (mean Kendall's τ = 0.67). In 17 of 18 settings it lands in the top-2.
  • Model-suggested formats: Models most often proposed the column-dictionary format for dataframes and the closed-bracket style for SQL schemas. For graphs, edge lists and adjacency dicts dominated. However, how often a model suggests a format is not a reliable indicator of how well that format performs.
  • Limitations: Fine-tuning gave little gain for Phi-3 on text-to-Python and for CodeLlama on text-to-SQL. The analysis also covers only one structured element per prompt.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢MicrosoftRedmond · United States325 works · 142,931 citations🎓Georgia Institute of TechnologyAtlanta · United States133 works · 24,497 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Vision and Pattern Recognition · 2,869 worksInformation Systems · 1,437 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,320 works

Cite

Cite as Usneek Singh, Ananya Singha, Abhijeet Awasthi, Sumit Gulwani, Aditya Kanade, Vu Le, Mukul Singh and Gust Verbruggen. 2026. Asking language models how to represent data for fine-tuning. Proceedings of the First Workshop on Structured Understanding, Retrieval, and Generation in the LLM Era (SURGeLLM 2026).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.surgellm-1.13

@inproceedings{singh2026asking,
  title = {{Asking language models how to represent data for fine-tuning}},
  author = {Usneek Singh, Ananya Singha, Abhijeet Awasthi, Sumit Gulwani, Aditya Kanade, Vu Le, Mukul Singh and Gust Verbruggen},
  booktitle = {Proceedings of the First Workshop on Structured Understanding, Retrieval, and Generation in the LLM Era (SURGeLLM 2026)},
  year = {2026},
  doi = {10.18653/v1/2026.surgellm-1.13},
  url = {https://doi.org/10.18653/v1/2026.surgellm-1.13}
}

References

17 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ziquan Liu, Yi Xu, Yuanhong Xu, Qi Qian, Hao Li, Xiangyang Ji, Antoni Chan, and Rong Jin. 2022. Improved fine-tuning by better leveraging pre-training data. Advances in Neural Information Processing Systems, 35:32568–32581. find
  2. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901.
  3. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374.
  4. Xi Fang, Weijie Xu, Fiona Anting Tan, Jiani Zhang, Ziqing Hu, Yanjun Jane Qi, Scott Nickleach, Diego Socolinsky, Srinivasan Sengamedu, Christos Faloutsos, et al. 2024. Large language models (llms) on tabular data: Prediction, generation, and understanding-a survey. find
  5. Bahare Fatemi, Jonathan Halcrow, and Bryan Perozzi. 2023. Talk like a graph: Encoding graphs for large language models. arXiv preprint arXiv:2310.04560. find
  6. Dawei Gao, Haibin Wang, Yaliang Li, Xiuyu Sun, Yichen Qian, Bolin Ding, and Jingren Zhou. 2023. Text-to-sql empowered by large language models: A benchmark evaluation. arXiv preprint arXiv:2308.15363.
  7. Usneek Singh, José Cambronero, Sumit Gulwani, Aditya Kanade, Anirudh Khatry, Vu Le, Mukul Singh, and Gust Verbruggen. 2024. An empirical study of validating synthetic data for formula generation. arXiv preprint arXiv:2407.10657. find
  8. Ananya Singha, José Cambronero, Sumit Gulwani, Vu Le, and Chris Parnin. 2023. Tabular representation, noisy operators, and impacts on table structure understanding tasks in llms. arXiv preprint arXiv:2310.10358. — [arXiv:2310.10358](https://arxiv.org/abs/2310.10358) find
  9. Ananya Singha, Bhavya Chopra, Anirudh Khatry, Sumit Gulwani, Austin Henley, Vu Le, Chris Parnin, Mukul Singh, and Gust Verbruggen. 2024. Semantically aligned question and code generation for automated insight generation. In Proceedings of the 1st International Workshop on Large Language Models for C… find
  10. Yuan Sui, Mengyu Zhou, Mingjie Zhou, Shi Han, and Dongmei Zhang. 2024. Table meets llm: Can large language models understand structured table data? a benchmark and empirical study. In Proceedings of the 17th ACM International Conference on Web Search and Data Mining, pages 645–654. find
  11. Heng Wang, Shangbin Feng, Tianxing He, Zhaoxuan Tan, Xiaochuang Han, and Yulia Tsvetkov. 2024a. Can language models solve graph problems in natural language? Advances in Neural Information Processing Systems, 36. find
  12. Zilong Wang, Hao Zhang, Chun-Liang Li, Julian Martin Eisenschlos, Vincent Perot, Zifeng Wang, Lesly Miculicich, Yasuhisa Fujii, Jingbo Shang, Chen-Yu Lee, et al. 2024b. Chain-of-table: Evolving tables in the reasoning chain for table understanding. arXiv preprint arXiv:2401.04398. find
  13. Ruosong Ye, Caiqi Zhang, Runhui Wang, Shuyuan Xu, Yongfeng Zhang, et al. 2023a. Natural language is all a graph needs. arXiv preprint arXiv:2308.07134, 4(5):7. find
  14. Yunhu Ye, Binyuan Hui, Min Yang, Binhua Li, Fei Huang, and Yongbin Li. 2023b. Large language models are versatile decomposers: Decomposing evidence and questions for table-based reasoning. In Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrie… find
  15. Pengcheng Yin, Wen-Ding Li, Kefan Xiao, Abhishek Rao, Yeming Wen, Kensen Shi, Joshua Howland, Paige Bailey, Michele Catasta, Henryk Michalewski, et al. 2022. Natural language to code generation in interactive data science notebooks. arXiv preprint arXiv:2212.09248. find
  16. Tao Yu, Rui Zhang, Kai Yang, Michihiro Yasunaga, Dongxu Wang, Zifan Li, James Ma, Irene Li, Qingning Yao, Shanelle Roman, et al. 2018. Spider: A large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to-sql task. arXiv preprint arXiv:1809.08887.
  17. Alex Zhuang, Ge Zhang, Tianyu Zheng, Xinrun Du, Junjie Wang, Weiming Ren, Stephen W Huang, Jie Fu, Xiang Yue, and Wenhu Chen. 2024. Structlm: Towards building generalist models for structured knowledge grounding. arXiv preprint arXiv:2402.16671. — [arXiv:2402.16671](https://arxiv.org/abs/2402.16671) find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/singh-2026-asking-language-models-how-represent/