Elamur Research⌕
☰
home / papers / 2026 / A Scalable Pipeline for Novelty Detection in Skill Extraction Using Large Language Models

A Scalable Pipeline for Novelty Detection in Skill Extraction Using Large Language Models

Summary

Skill ontologies need constant updating because new competences keep appearing in job postings faster than curators can track them by hand. Seifert and Clematide extend an existing skill extraction system (BERT-based zoning, LLM extraction of skill mentions, embedding-based mapping to the ontology) with a three-stage module for spotting skills the ontology lacks. First, an LLM reviews the full advertisement together with the extracted mentions and matched skills. It proposes missing skills in the ontology's topic-predicate format as JSON. Next, a filter called "Turbo" re-matches each proposal against the whole ontology. It keeps only proposals whose similarity distance exceeds a configurable threshold, and it separates LLM false positives from upstream matching errors. Finally, the surviving novelties are merged using string-based and embedding-based clustering. The authors tested the pipeline on Swiss job ads with GPT-4o, Gemini-2.0-flash and DeepSeek-V3. It cut false positives by 82% and reduced the number of items needing review by 97%. Gemini-2.0-flash was the most precise model. The approach makes semi-automatic ontology maintenance considerably more practical.

Key points

  • Data: Three samples were taken from a proprietary collection, gathered since 2012, that holds over 20 million job advertisements from Switzerland:
    • 10_isco: 10 German ads, one for each major ISCO-08 group.
    • 30_industry_after_2024: 30 multilingual ads from innovative sectors, published after 1 January 2024.
    • 1000_industry_after_2024: 1000 ads, used for tuning and testing aggregation.
    • A data subset, the prompts and the code are to be released on GitHub (gseife/Novelty-Detection-in-Skill-Extraction).
  • Candidate generation: On the 30-ad set, DeepSeek-V3 produced 233 candidates, Gemini-2.0-flash 131 and GPT-4o 77. Many of these proposals were noise.
  • Turbo thresholds: The tested distance thresholds were 1.7 (strict), 1.4 and 1.3. At 1.7, Turbo discarded 97% of GPT-4o, 93% of Gemini and 84% of DeepSeek candidates. Gemini had the lowest detection-fail rate (33% at 1.4), while DeepSeek had the highest (60%).
  • Aggregation: The first step groups word-order variants with the FuzzyWuzzy token sort algorithm. The second step clusters semantically similar skills with DBSCAN over OpenAI text-embedding-3-small vectors. Together they cut curation items by 97%.
  • Manual evaluation:
    • Gemini-2.0-flash reached true-novelty ratios of 73%, 72% and 62% at thresholds 1.4, 1.3 and 1.2.
    • GPT-4o had the best format compliance (up to 67%).
    • DeepSeek-V3 scored only 27% novelty and 4% correct format.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.63317/3twvjs5vmuwt

@inproceedings{seifert2026scalable,
  title = {{A Scalable Pipeline for Novelty Detection in Skill Extraction Using Large Language Models}},
  author = {Seifert, Gian and Clematide, Simon},
  booktitle = {Proceedings of the Fifteenth Language Resources and Evaluation Conference},
  year = {2026},
  doi = {10.63317/3twvjs5vmuwt},
  url = {https://doi.org/10.63317/3twvjs5vmuwt}
}

References

13 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Adam Cohen. 2011. Fuzzy String Matching in Python. Accessed: 2025-05-21. find
  2. European Commission, Social Affairs Directorate-General for Employment, and Inclusion. 2017. ESCO handbook – European skills, competences, qualifications and occupations. Publications Office. find
  3. DeepSeek-AI, Aixin Liu, Bei Feng, et al. 2025. DeepSeek-V3 Technical Report. find
  4. Martin Ester, Hans-Peter Kriegel, Jörg Sander, and Xiaowei Xu. 1996. A Density-Based Algorithm for Discovering Clusters in Large Spatial Databases with Noise. In Proceedings of the Second International Conference on Knowledge Discovery and Data Mining, KDD’96, page 226–231. AAAI Press. find
  5. Tirthankar Ghosal, Tanik Saikh, Tameesh Biswas, Asif Ekbal, and Pushpak Bhattacharyya. 2022. Novelty Detection: A Perspective from Natural Language Processing. Computational Linguistics, 48(1):77–117. _eprint: https://direct.mit.edu/coli/article-pdf/48/1/77/2006641/coli_a_00429.pdf. ↗
  6. Google. 2025. Gemini 2.0 Flash. Accessed: 2025-06-19. find
  7. Christina Gregory, Phil Lewis, Pamela Frugoli, and Alexander Nallin. 2019. Updating the O*NET®-SOC Taxonomy: Incorporating the 2018 SOC Structure. Technical report, National Center for O*NET Development, Raleigh, NC. Accessed: 2025-06-06. find
  8. OpenAI. 2024. text-embedding-3-small. Accessed: 2025-05-21. find
  9. OpenAI, Aaron Hurst, Adam Lerer, , et al. 2024. GPT-4o System Card. find our card
  10. Marco A. F. Pimentel, David A. Clifton, Lei Clifton, and Lionel Tarassenko. 2014. A review of novelty detection. Signal Processing, 99:215–249. find
  11. Maria Taboada, Diego Martinez, Mohammed Arideh, and Rosa Mosquera. 2025. Ontology matching with Large Language Models and prioritized depth-first search. Information Fusion, 123:103254. find
  12. Zhi Rui Tam, Cheng-Kuang Wu, Yi-Lin Tsai, Chieh-Yen Lin, Hung yi Lee, and Yun-Nung Chen. 2024. Let Me Speak Freely? A Study on the Impact of Format Restrictions on Performance of Large Language Models. find
  13. Mike Zhang, Kristian Nørgaard Jensen, Rob van der Goot, and Barbara Plank. 2022. Skill Extraction from Job Postings using Weak Supervision. ArXiv:2209.08071 [cs]. — [arXiv:2209.08071](https://arxiv.org/abs/2209.08071) find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/seifert-2026-scalable-pipeline-novelty-detection-skill/