Elamur Research⌕
☰
home / papers / 2026 / PromptFE: Automated Feature Engineering by Prompting

PromptFE: Automated Feature Engineering by Prompting

Paper · 2026 · Yufeng Zou, Jean Utke, Diego Klabjan et al.

Summary

Automated feature engineering (AutoFE) for tabular data usually relies on brute-force search over candidate transformations. It ignores what the columns actually mean, has to start from scratch on every new dataset, and gives no rationale for the features it creates. PromptFE addresses this by putting an LLM (GPT-3.5 and GPT-4) inside an iterative search loop.

The prompt gives the model:

  • a description of the dataset and its attributes,
  • a set of transformation operators,
  • a ranked list of previously scored example features.

Features are written as canonical Reverse Polish Notation (cRPN) strings. cRPN fixes the ordering of commutative operators so that each feature has exactly one string form, which shrinks the search space. The LLM proposes new features with natural-language explanations. These are evaluated, and the best ones are fed back into the prompt, so the model improves through in-context learning.

On real-world tabular datasets, PromptFE beat state-of-the-art AutoFE baselines. Ablations confirmed that both the semantic context and the canonicalization help. The work matters because it shows LLM domain knowledge can make automated feature construction more effective, cheaper to search, and easier to interpret.

Key points

  • Across three downstream models (including RF, LGBM and linear models), PromptFE yields a mean performance gain of more than 5%.
  • Each prompt has six parts:
    • a dataset meta description,
    • indexed attributes with types, ranges and descriptions,
    • operators grouped by arity,
    • ranked example features with scores,
    • feedback from the previous iteration,
    • an output template for features plus explanations.
  • With GPT-3.5, a sampling temperature of 1 gave both the best mean validation score (0.8197) and the fewest LLM generations (about 369). Temperature 0.5 scored 0.8169 with about 863 generations, and 1.5 scored 0.8194 with about 643. The gap versus 0.5 is significant at p < 0.01.
  • Using 10 example features in the prompt worked best (mean 0.6840). This was significantly better than using 1 example (p < 0.05), and adding more examples reduced performance.
  • The authors note two limitations: the prompt template was designed by hand, and only OpenAI GPT models were tested.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Northwestern UniversityEvanston · United States46 works · 2,850 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksInformation Systems · 1,730 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,894 works

Cite

Cite as Yufeng Zou, Jean Utke, Diego Klabjan and Han Liu. 2026. PromptFE: Automated Feature Engineering by Prompting. Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.eacl-long.28

@inproceedings{zou2026promptfe,
  title = {{PromptFE: Automated Feature Engineering by Prompting}},
  author = {Yufeng Zou, Jean Utke, Diego Klabjan and Han Liu},
  booktitle = {Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.eacl-long.28},
  url = {https://doi.org/10.18653/v1/2026.eacl-long.28}
}

References

38 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Leo Breiman. 2001. Random forests. Machine learning, 45:5–32. find
  2. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. In Proceedings of Advances in Neural Information Processing Systems, volume 33, pages 1877– 19…
  3. Milton Friedman. 1937. The use of ranks to avoid the assumption of normality implicit in the analysis of variance. Journal of the American Statistical Association, 32(200):675–701. find
  4. Sainyam Galhotra, Udayan Khurana, Oktie Hassanzadeh, Kavitha Srinivas, and Horst Samulowitz. 2019. Kafe: Automated feature enhancement for predictive modeling using external knowledge. In Proceedings of NeurIPS 2019 Workshop: Knowledge Representation & Reasoning Meets Machine Learning. find
  5. Leo Grinsztajn, Edouard Oyallon, and Gael Varoquaux. 2022. Why do tree-based models still outperform deep learning on typical tabular data? In Proceedings of the thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track. find
  6. Charles L Hamblin. 1962. Translation to and from polish notation. The Computer Journal, 5(3):210– 213. find
  7. Sungwon Han, Jinsung Yoon, Sercan O Arik, and Tomas Pfister. 2024. Large language models can automatically engineer features for few-shot tabular learning. In International Conference on Machine Learning, pages 17454–17479. PMLR. find
  8. Noah Hollmann, Samuel Müller, and Frank Hutter. 2023. Large language models for automated data science: Introducing CAAFE for context-aware automated feature engineering. In Proceedings of the thirty-seventh Conference on Neural Information Processing Systems. find
  9. Jaehyun Nam, Kyuyoung Kim, Seunghyuk Oh, Jihoon Tack, Jaehyung Kim, and Jinwoo Shin. 2024. Optimized feature generation for tabular data via llms with decision tree reasoning. In Advances in Neural Information Processing Systems, volume 37, pages 92352–92380. find
  10. Fatemeh Nargesian, Horst Samulowitz, Udayan Khurana, Elias B Khalil, and Deepak S Turaga. 2017. Learning feature engineering for classification. In Proceedings of International Joint Conference on Artificial Intelligence, volume 17, pages 2529–2535. find
  11. Franziska Horn, Robert Pack, and Michael Rieger. 2020. The autofeat python library for automated feature engineering and selection. In Proceedings of Machine Learning and Knowledge Discovery in Databases, pages 111–120. Springer. find
  12. Peter Bjorn Nemenyi. 1963. Distribution-free multiple comparisons. Princeton University. find
  13. Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. 2023. Survey of hallucination in natural language generation. ACM Computing Surveys, 55(12):1–38.
  14. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instructions with human feedback. In Proceedings of Advances in Neural Information Processing Systems, volume…
  15. James Max Kanter and Kalyan Veeramachaneni. 2015. Deep feature synthesis: Towards automating data science endeavors. In Proceedings of 2015 IEEE International Conference on Data Science and Advanced Analytics (DSAA), pages 1–10. IEEE. find
  16. Guolin Ke, Qi Meng, Thomas Finley, Taifeng Wang, Wei Chen, Weidong Ma, Qiwei Ye, and Tie-Yan Liu. 2017. Lightgbm: A highly efficient gradient boosting decision tree. In Proceedings of Advances in Neural Information Processing Systems, volume 30. find
  17. Udayan Khurana, Horst Samulowitz, and Deepak Turaga. 2018. Feature engineering for predictive modeling using reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32. find
  18. Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022. Large language models are zero-shot reasoners. In Advances in Neural Information Processing Systems.
  19. Liyao Li, Haobo Wang, Liangyu Zha, Qingyi Huang, Sai Wu, Gang Chen, and Junbo Zhao. 2023. Learning a data-driven policy network for pre-training automated feature engineering. In Proceedings of the eleventh International Conference on Learning Representations. find
  20. OpenAI. 2023. Gpt-4 technical report. arXiv:2303.08774.
  21. Preprint, Tom Overman, Diego Klabjan, and Jean Utke. 2024. IIFE: Interaction information based automated feature engineering. In 2024 IEEE International Conference on Data Mining (ICDM), pages 803–808. IEEE. find
  22. Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. find
  23. Matthew G Smith and Larry Bull. 2005. Genetic programming with a genetic algorithm for feature construction and selection. Genetic Programming and Evolvable Machines, 6:265–281. find
  24. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023a. Llama: Open and efficient foundation language models. Preprin…
  25. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023b. Llama 2: Open foundation and fine-tuned chat models. Preprint, arXiv:2307.09288.
  26. Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. 2023. Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing. ACM Comput. Surv., 55(9).
  27. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Proceedings of Advances in Neural Information Processing Systems, volume 30.
  28. Clint Morris, Michael Jurado, and Jason Zutty. 2024. Llm guided evolution - the automation of models advancing models. In Proceedings of the Genetic and Evolutionary Computation Conference, GECCO ’24, page 377–384. find
  29. Kafeng Wang, Pengyang Wang, and Chengzhong Xu. 2023. Toward efficient automated feature engineering. In Proceedings of 2023 IEEE 39th International Conference on Data Engineering (ICDE), pages 1625–1637. find
  30. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed H. Chi, Quoc V Le, and Denny Zhou. 2022. Chain of thought prompting elicits reasoning in large language models. In Advances in Neural Information Processing Systems.
  31. Chengrun Yang, Xuezhi Wang, Yifeng Lu, Hanxiao Liu, Quoc V Le, Denny Zhou, and Xinyun Chen. 2024. Large language models as optimizers. In Proceedings of the twelfth International Conference on Learning Representations.
  32. Kaizhong Zhang and Dennis Shasha. 1989. Simple fast algorithms for the editing distance between trees and related problems. SIAM J. Comput., 18:1245–1262. find
  33. Tianping Zhang, Zheyu Zhang, Zhiyuan Fan, Haoyan Luo, Fengyuan Liu, Qian Liu, Wei Cao, and Jian Li. 2023. Openfe: automated feature generation with expert-level performance. In Proceedings of the 40th International Conference on Machine Learning, ICML’23. find
  34. Xinhao Zhang, Jinghan Zhang, Banafsheh Rekabdar, Yuanchun Zhou, Pengfei Wang, and Kunpeng Liu. 2024a. Dynamic and adaptive feature generation with llm. arXiv preprint arXiv:2406.03505. — [arXiv:2406.03505](https://arxiv.org/abs/2406.03505) find
  35. Yanlin Zhang, Ning Li, Quan Gan, Weinan Zhang, David Wipf, and Minjie Wang. 2024b. Elf-gym: Evaluating large language models generated features for tabular prediction. In Proceedings of the 33rd ACM International Conference on Information and Knowledge Management, pages 5420–5424. find
  36. Guanghui Zhu, Shen Jiang, Xu Guo, Chunfeng Yuan, and Yihua Huang. 2022a. Evolutionary automated feature engineering. In Proceedings of Pacific Rim International Conference on Artificial Intelligence, pages 574–586. Springer. find
  37. Guanghui Zhu, Zhuoer Xu, Chunfeng Yuan, and Yihua Huang. 2022b. Difer: Differentiable automated feature engineering. In Proceedings of the first International Conference on Automated Machine Learning, volume 188 of Proceedings of Machine Learning Research, pages 17/1–17. PMLR. find
  38. Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. 2019. Fine-tuning language models from human preferences. Preprint, arXiv:1909.08593. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zou-2026-promptfe-automated-feature-engineering-prompting/