Elamur Research⌕
☰
home / papers / 2026 / Building a Custom Taxonomy of AI Skills and Tasks from the Ground Up with Job Postings

Building a Custom Taxonomy of AI Skills and Tasks from the Ground Up with Job Postings

Paper · 2026 · Stephen Meisenbacher and Peter Norlander

Summary

The paper asks how to best use very large, fast-growing text collections when building a domain taxonomy automatically with LLMs. The test case is AI-related skills, tasks, experience and qualifications in US job ads, where no established taxonomy exists. The authors propose TaxonomyBuilder, a bottom-up pipeline:

  • Candidate sentences are mined with keywords and scored against class vectors.
  • These sentences are clustered into leaf nodes and labeled by an LLM.
  • Higher levels of the hierarchy are then built iteratively.

They test 12 pipeline configurations on two corpora, varying three factors: data augmentation, soft vs. hard clustering, and percentile-based filtering of candidates. Evaluation uses automatic metrics, LLM-as-a-Judge scores, and coverage on held-out postings, with factorial ANOVA to estimate the effect of each factor.

The core finding is that using less data works better. Filtering inputs strictly and skipping augmentation gives more focused, domain-specific taxonomies than feeding in everything. Lenient inclusion mainly helps under very strict coverage thresholds. This offers practical guidance for taxonomy induction at the scale of millions of documents, a setting earlier work, built on far smaller samples, did not address.

Key points

  • Corpora:
    • NLx: 32.12M US job postings from 2024–2025, proprietary.
    • USAJOBS: 1.54M federal postings from 2017–2023, released by Resh et al. (2025).
    • The final month of each corpus (Dec 2025 and Nov 2023) is held out for testing.
  • Augmentation hurts: it won just one of eight LLM-as-a-Judge categories. It also sharply lowered strict coverage at τ = 0.8 (F = 40.81, p < 0.001, η² = 0.836) and reduced label utilization.
  • Best recipe: soft clustering without augmentation achieved the top score in 20 of 28 cases.
  • Coverage trade-off: taxonomies built with the looser 25th-percentile filter won at τ = 0.9 but won no LLM-judge categories. Those built with the stricter 75th-percentile filter won at τ = 0.6.
  • Released resources: TaxonomyBuilder is open-sourced on GitHub (sjmeis/TaxonomyBuilder) and as a Python package. The AI-skills taxonomies are public, with company names redacted.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Technical University of MunichMunich · Germany55 works · 48,474 citations🎓Loyola University ChicagoChicago · United States2 works · 1 citation

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksBusiness, Management and Accounting · 32 worksOrganizational Behavior and Human Resource Management · 12 works

💡 Concepts 1Key ideas and methods explained on our concept pages.Inverted index · 26 works

Cite

Cite as Stephen Meisenbacher and Peter Norlander. 2026. Building a Custom Taxonomy of AI Skills and Tasks from the Ground Up with Job Postings. Proceedings of the Second Workshop on Customizable NLP: Progress and Challenges in Customizing NLP for a Domain, Application, Group, or Individual (CustomNLP4U).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.customnlp4u-1.11

@inproceedings{meisenbacher2026building,
  title = {{Building a Custom Taxonomy of AI Skills and Tasks from the Ground Up with Job Postings}},
  author = {Stephen Meisenbacher and Peter Norlander},
  booktitle = {Proceedings of the Second Workshop on Customizable NLP: Progress and Challenges in Customizing NLP for a Domain, Application, Group, or Individual (CustomNLP4U)},
  year = {2026},
  doi = {10.18653/v1/2026.customnlp4u-1.11},
  url = {https://doi.org/10.18653/v1/2026.customnlp4u-1.11}
}

References

55 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Daron Acemoglu, David Autor, and Simon Johnson. 2026. Building Pro-Worker Artificial Intelligence. find
  2. Liudmila Alekseeva, José Azar, Mireia Giné, Sampsa Samila, and Bledi Taska. 2021. The demand for AI skills in the labor market. Labour Economics, 71:102002. find
  3. Stefano Baruffaldi, Brigitte van Beuzekom, Hélène Der-nis, Dietmar Harhoff, Nandan Rao, David Rosenfeld, and Mariagrazia Squicciarini. 2020. Identifying and measuring developments in artificial intelligence: Making the impossible possible. OECD Science, Technology and Industry Working Papers, 2020(0… find
  4. Ricardo JGB Campello, Davoud Moulavi, and Jörg Sander. 2013. Density-based clustering based on hierarchical density estimates. In Pacific-Asia conference on knowledge discovery and data mining, pages 160–172. Springer. find
  5. Boqi Chen, Fandi Yi, and Dániel Varró. 2023. Prompting or fine-tuning? a comparative study of large language models for taxonomy construction. In 2023 ACM/IEEE International Conference on Model Driven Engineering Languages and Systems Companion (MODELS-C), pages 588–596. find
  6. Philipp Cimiano, Andreas Hotho, and Steffen Staab. 2004. Comparing conceptual, divisive and agglomerative clustering for learning taxonomies from text. In Proceedings of the 16th European Conference on Artificial Intelligence, ECAI’04, page 435–439, NLD. IOS Press. find
  7. Luis Espinosa-Anke, Jose Camacho-Collados, Claudio Delli Bovi, and Horacio Saggion. 2016. Supervised distributional hypernym discovery via domain adaptation. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 424–435, Austin, Texas. Association for Compu… find
  8. Trevor Fountain and Mirella Lapata. 2012. Taxonomy induction using hierarchical random graphs. In Proceedings of the 2012 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 466–476, Montréal, Canada. Association for Computati… find
  9. Morgan R. Frank, David Autor, James E. Bessen, Erik Brynjolfsson, Manuel Cebrian, David J. Deming, Maryann Feldman, Matthew Groh, José Lobo, Esteban Moro, Dashun Wang, Hyejin Youn, and Iyad Rahwan. 2019. Toward understanding the impact of artificial intelligence on labor. Proceedings of the National… find
  10. Ruiji Fu, Jiang Guo, Bing Qin, Wanxiang Che, Haifeng Wang, and Ting Liu. 2014. Learning semantic hierarchies via word embeddings. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1199–1209, Baltimore, Maryland. Association for…
  11. Avi Goldfarb, Bledi Taska, and Florenta Teodoridis. 2023. Could machine learning be a general purpose technology? A comparison of emerging technologies using data from online job postings. Research Policy, 52(1):104653. find
  12. Allan D Gordon. 1996. Hierarchical classification. Clustering and classification, page 65. find
  13. Marti A. Hearst. 1992. Automatic acquisition of hyponyms from large text corpora. In COLING 1992 Volume 2: The 14th International Conference on Computational Linguistics.
  14. Chen Huang and Guoxiu He. 2025. Text clustering as classification with llms. In Proceedings of the 2025 Annual International ACM SIGIR Conference on Research and Development in Information Retrieval in the Asia Pacific Region, SIGIR-AP 2025, page 374–384, New York, NY, USA. Association for Computing… find
  15. Dino Ienco and Rosa Meo. 2008. Towards the automatic construction of conceptual taxonomies. In International Conference on Data Warehousing and Knowledge Discovery, pages 327–336. Springer. find
  16. Angelika Kaplan, Thomas Kühn, Sebastian Hahner, Niko Benkler, Jan Keim, Dominik Fuchß, Sophie Corallo, and Robert Heinrich. 2022. Introducing an evaluation method for taxonomies. In Proceedings of the 26th International Conference on Evaluation and Assessment in Software Engineering, EASE ’22, page… find
  17. Yuning Mao, Xiang Ren, Jiaming Shen, Xiaotao Gu, and Jiawei Han. 2018. End-to-end reinforcement learning for automatic taxonomy induction. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2462–2472, Melbourne, Australia. Associ… find
  18. Oleksandr Marchenko and Danylo Dvoichenkov. 2024. Taxorankconstruct: A novel rank-based iterative approach to taxonomy construction with large language models. In ISS@ IT&I, pages 11–27. find
  19. Priyanka Kargupta, Nan Zhang, Yunyi Zhang, Rui Zhang, Prasenjit Mitra, and Jiawei Han. 2025. TaxoAdapt: Aligning LLM-based multidimensional taxonomy construction to evolving research corpora. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long P… find
  20. Zornitsa Kozareva and Eduard Hovy. 2010. A semi-supervised method to learn and construct taxonomies using the web. In Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing, pages 1110–1118, Cambridge, MA. Association for Computational Linguistics. find
  21. Avishek Lahiri, Yufang Hou, and Debarshi Kumar Sanyal. 2025. TaxoAlign: Scholarly taxonomy generation using language models. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 30203–30223, Suzhou, China. Association for Computational Linguistics. find
  22. Josh Lerner and Jean Tirole. 2002. Some Simple Economics of Open Source. The Journal of Industrial Economics, 50(2):197–234. find
  23. Nan Li, Bo Kang, and Tijl De Bie. 2025. Building data-driven occupation taxonomies: A bottom-up multi-stage approach via semantic clustering and multi-agent collaboration. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing: Industry Track, pages 1596–1614, Suzh… find
  24. Zehan Li, Xin Zhang, Yanzhao Zhang, Dingkun Long, Pengjun Xie, and Meishan Zhang. 2023. Towards general text embeddings with multi-stage contrastive learning. Preprint, arXiv:2308.03281.
  25. Anoush Margaryan. 2023. Artificial intelligence and skills in the workplace: An integrative research agenda. Big Data & Society, 10(2):20539517231206804. find
  26. Nestor Maslej, Loredana Fattorini, Raymond Perrault, Yolanda Gil, Vanessa Parli, Njenga Kariuki, Emily Capstick, Anka Reuel, Erik Brynjolfsson, John Etchemendy, Katrina Ligett, Terah Lyons, James Manyika, Juan Carlos Niebles, Yoav Shoham, Russell Wald, Toby Walsh, Armin Hamrah, Lapo Santarlasci, and… find
  27. Artificial Intelligence Index Report 2025. arXiv preprint. ArXiv:2504.07139 [cs]. — [arXiv:2504.07139](https://arxiv.org/abs/2504.07139) find
  28. Stephen Meisenbacher, Tim Schopf, Weixin Yan, Patrick Holl, and Florian Matthes. 2024. An improved method for class-specific keyword extraction: A case study in the German business registry. In Proceedings of the 20th Conference on Natural Language Processing (KONVENS 2024), pages 159–165, Vienna, A… find
  29. Ryszard S Michalski and Robert E Stepp. 1983. Learning from observation: Conceptual clustering. In Machine learning, pages 331–363. Elsevier. find
  30. Viktor Moskvoretskii, Irina Nikishina, Ekaterina Neminova, Alina Lobanova, Alexander Panchenko, and Chris Biemann. 2026. Large language models for creation, enrichment and evaluation of taxonomic graphs. Semantic Web, 17(1):22104968251404186. find
  31. Elham Motamedi, Inna Novalija, and Luis Rei. 2026. Semi-automatic hierarchical taxonomy creation from existing taxonomies with large language models. Business & Information Systems Engineering, pages 1–23. find
  32. Abhishek Nagaraj. 2022. The Private Impact of Public Data: Landsat Satellite Maps Increased Gold Discoveries and Encouraged Entry. Management Science, 68(1):564–582. find
  33. Lightcast. 2023. The Lightcast open skills taxonomy. Technical report, Lightcast. White paper. Formerly Burning Glass Technologies/Emsi Burning Glass. find
  34. Bowen Lou and Lynn Wu. 2021. AI on Drugs: Can Artificial Intelligence Accelerate Drug Development? Evidence from a Large-Scale Examination of Bio-Pharma Firms. Management Information Systems Quarterly, 45(3):1451–1482. find
  35. Abhishek Nagaraj, Esther Shears, and Mathijs de Vaan. 2020. Improving data access democratizes and diversifies science. Proceedings of the National Academy of Sciences, 117(38):23490–23498. find
  36. Engineering National Academies of Sciences, and Medicine. 2025. Artificial Intelligence and the Future of Work. The National Academies Press, Washington, DC. find
  37. Raúl Navarro-Almanza, Reyes Juárez-Ramírez, Guillermo Licea, and Juan R Castro. 2020. Automated ontology extraction from unstructured texts using deep learning. In Intuitionistic and Type-2 fuzzy logic enhancements in neural and optimization algorithms: Theory and applications, pages 727–755. Spring… find
  38. Rion Snow, Daniel Jurafsky, and Andrew Y. Ng. 2006. Semantic taxonomy induction from heterogenous evidence. In Proceedings of the 21st International Conference on Computational Linguistics and 44th Annual Meeting of the Association for Computational Linguistics, pages 801–808, Sydney, Australia. Ass… find
  39. Robert C Nickerson, Upkar Varshney, and Jan Muntermann. 2013. A method for taxonomy development and its application in information systems. European journal of information systems, 22(3):336–359. find
  40. Prasanna B. Tambe. 2026. Reskilling the Workforce for AI: Domain Expertise and Algorithmic Literacy. Management Science, 72(1):515–537. find
  41. OpenAI, :, Aaron Hurst, Adam Lerer, Adam P. Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, Aleksander M adry, Alex Baker-Whitcomb, Alex Beutel, Alex Borzunov, Alex Carney, Alex Chow, Alex Kirillov, and 401 others. 2024. Gpt-4o system card.…
  42. Sebastian Padó and Mirella Lapata. 2007. Dependency-based construction of semantic space models. Computational Linguistics, 33(2):161–199.
  43. Marius Pasca. 2004. Acquisition of categorized named entities for web search. In Proceedings of the Thirteenth ACM International Conference on Information and Knowledge Management, CIKM ’04, page 137–145, New York, NY, USA. Association for Computing Machinery. find
  44. Hoifung Poon and Pedro Domingos. 2010. Unsupervised ontology induction from text. In Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics, pages 296–305, Uppsala, Sweden. Association for Computational Linguistics. find
  45. Nils Reimers and Iryna Gurevych. 2019. Sentence-BERT: Sentence embeddings using Siamese BERT-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3982–3992,…
  46. William Resh, Keunyoung Lee, and Yi Ming. 2025. U.S. Federal Civil Position Job Postings (2018-2023). find
  47. Tim Schopf, Daniel Braun, and Florian Matthes. 2020. Semantic label representations with lbl2vec: A similarity-based approach for unsupervised text classification. In International conference on web information systems and technologies, pages 59–73. Springer. find
  48. Aaditya Singh, Adam Fry, Adam Perelman, Adam Tart, Adi Ganesh, Ahmed El-Kishky, Aidan McLaughlin, Aiden Low, AJ Ostrow, Akhila Ananthram, Akshay Nathan, Alan Luo, Alec Helyar, Aleksander Madry, Aleksandr Efremov, Aleksandra Spyra, Alex Baker-Whitcomb, Alex Beutel, Alex Karpenko, and 465 others. 2025… — OpenAI GPT-5 System Card
  49. Paola Velardi, Stefano Faralli, and Roberto Navigli. 2013. OntoLearn reloaded: A graph-based algorithm for taxonomy induction. Computational Linguistics, 39(3):665–707. find
  50. Henrique Schechter Vera, Sahil Dua, Biao Zhang, Daniel Salz, Ryan Mullins, Sindhu Raghuram Panyam, Sara Smoot, Iftekhar Naim, Joe Zou, Feiyang Chen, Daniel Cer, Alice Lisak, Min Choi, Lucas Gonzalez, Omar Sanseviero, Glenn Cameron, Ian Ballantyne, Kat Black, Kaifeng Chen, and 70 others. 2025. Embedd… — [arXiv:2509.20354](https://arxiv.org/abs/2509.20354) find
  51. Binh Vu, Rashmi Govindraju Naik, Bao Khanh Nguyen, Sina Mehraeen, and Matthias Hemmje. 2025. Automated taxonomy construction using large language models: A comparative study of fine-tuning and prompt engineering. Eng, 6(11):283. find
  52. Mengting Wan, Tara Safavi, Sujay Kumar Jauhar, Yujin Kim, Scott Counts, Jennifer Neville, Siddharth Suri, Chirag Shah, Ryen W. White, Longqi Yang, Reid Andersen, Georg Buscher, Dhruv Joshi, and Nagu Rangan. 2024. Tnt-llm: Text mining at scale with large language models. In Proceedings of the 30th AC… find
  53. Qingkai Zeng, Yuyang Bai, Zhaoxuan Tan, Shangbin Feng, Zhenwen Liang, Zhihan Zhang, and Meng Jiang. 2024. Chain-of-layer: Iteratively prompting large language models for taxonomy induction from limited examples. In Proceedings of the 33rd ACM International Conference on Information and Knowledge Man… find
  54. Chao Zhang, Fangbo Tao, Xiusi Chen, Jiaming Shen, Meng Jiang, Brian Sadler, Michelle Vanni, and Jiawei Han. 2018. Taxogen: Unsupervised topic taxonomy construction by adaptive term embedding and clustering. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data… find
  55. Haiqi Zhang, Zhengyuan Zhu, Zeyu Zhang, and Chengkai Li. 2025. LLMTaxo: Leveraging large language models for constructing taxonomy of factual claims from social media. In Findings of the Association for Computational Linguistics: ACL 2025, pages 19627–19641, Vienna, Austria. Association for Computat… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/meisenbacher-2026-building-custom-taxonomy-ai-skills/