⚖️ TF-IDF
Term frequency–inverse document frequency (TF-IDF) is a weighting scheme that scores how important a word is to one document in a collection. The score rises with how often the word occurs in that document and falls with how many documents contain it.
What it is
Imagine searching a pile of articles for "photosynthesis". An article that uses the word ten times is probably more about it than one that uses it once. A word like "the", though, appears everywhere and says nothing about any particular article. TF-IDF combines these two observations into one number per word per document. Frequent-here words get credit, and common-everywhere words get discounted.
Each document becomes a long, mostly empty vector with one slot per vocabulary word, filled with TF-IDF weights. These vectors are the classic input for information retrieval, text classification and text mining. They are cheap to compute, an inverted index makes them fast to search, and anyone can read them, because every dimension is a word.
The key idea
The formula
tf-idf(t, d) = tf(t, d) × log(N / df(t))
Here t is a term, usually a word after tokenization and often stemming. d is a document. tf(t, d) is the number of times t occurs in d. N is the size of the collection, counted in documents. df(t) is the number of documents that contain t at least once. The logarithm compresses the ratio. A term found in every document gets a weight of zero, and a term found in a single document gets the largest boost. Because word frequencies follow Zipf's law, a few very common words dominate raw counts, and the idf factor is what suppresses them.
A worked example
Take three documents, N = 3, and use the natural logarithm:
- d1: "the cat sat on the mat"
- d2: "the dog sat on the log"
- d3: "the cat chased the dog"
"the" occurs in all three documents, so its idf is log(3/3) = 0. Although it appears twice in d1, it contributes nothing. "cat", "sat", "on" and "dog" each occur in two documents, so their idf is log(3/2) ≈ 0.405. "mat", "log" and "chased" occur once, so their idf is log(3) ≈ 1.099. In d1, "mat" (1 × 1.099) therefore outweighs "cat" (1 × 0.405). In d3, "chased" is the most distinctive word. The weighting recovers what a human reader would call each document's characteristic words.
Variants
| Component | Common forms |
|---|---|
| tf | raw count; log-scaled 1 + log tf (0 if absent); normalised by document length or by the largest count in d; binary presence |
| idf | plain log(N/df); smoothed log((1 + N)/(1 + df)) + 1, which avoids division by zero; probabilistic log((N − df)/df), which turns negative once a term occurs in more than half the documents |
Log-scaled ("sublinear") tf stops one repeated word from dominating a document. Smoothing keeps unseen or universal terms well defined.
Comparing documents
Two TF-IDF vectors are usually compared by cosine similarity:
cos(a, b) = Σᵢ aᵢ bᵢ / ( √Σᵢ aᵢ² × √Σᵢ bᵢ² )
a and b are the weight vectors of two documents (or a query and a document), and i runs over vocabulary terms. Dividing by the vector lengths removes the advantage long documents would otherwise have. The result measures shared distinctive vocabulary, a lexical proxy for semantic similarity.
Try it: TF-IDF and cosine similarity
Two texts become vectors of word weights, term frequency times inverse document frequency, and their similarity is the cosine of the angle between the vectors. The document frequencies are those of this library’s paper texts: a word that nearly every paper uses («the», «of», «model») weighs little, a word few use weighs a lot. Edit either text and compare the TF-IDF cosine with the cosine of plain counts.
A: «The parser assigns a syntactic head to every word of the sentence.» B: «In the sentence, every word gets a head from a dependency parser.» Cosine 0.583 with TF-IDF weights, 0.643 with plain counts.
| word | texts with it | idf |
|---|---|---|
| parser | 4,059 | 2.75 |
| head | 7,377 | 2.15 |
| every | 12,443 | 1.63 |
| sentence | 15,529 | 1.41 |
| word | 17,036 | 1.31 |
| a | 23,318 | 1.00 |
| the | 23,318 | 1.00 |
idf = ln((1 + N) / (1 + df)) + 1 over N = 23,318 paper texts of this library, as of 2026-10-24; a word outside the 5,000 most widespread counts as rare (df = 0).
History and founding papers
Hans Peter Luhn (1957) argued that how often a word occurs in a document signals what the document is about. Karen Spärck Jones (1972) added the complementary idea of term specificity: a term used in fewer documents discriminates better and deserves more weight. This is the origin of idf. Gerard Salton and colleagues (Salton, Wong and Yang, 1975) placed such weights in the vector space model, where documents and queries are points in term space and ranking is by cosine. Latent semantic indexing (LSI) later factorised these matrices.
The probabilistic line grew from the same roots. The Okapi system was built on the Robertson–Spärck Jones probabilistic model. At the third Text REtrieval Conference (TREC-3), the City University, London team introduced BM25 by merging the earlier BM11 and BM15 schemes (Robertson et al. 1994):
BM25(q, d) = Σ_{t ∈ q} idf(t) × tf(t, d) × (k₁ + 1) / ( tf(t, d) + k₁ × (1 − b + b × |d| / avgdl) )
q is the query, |d| is the document's length and avgdl is the average length in the collection. k₁ controls how quickly repeated occurrences saturate, and b controls length normalisation. The idf is typically log((N − df + 0.5)/(df + 0.5)). BM25 keeps TF-IDF's two ingredients but caps the effect of tf and normalises for length. The team typically set b to about 0.75 and paired it with a larger k₁ (Robertson et al. 1994).
In 1998 Ponte and Croft proposed ranking documents by query likelihood under per-document language models. On TREC ad hoc collections this beat the TF-IDF weighting of the INQUERY retrieval system, without hand-tuned heuristics (Ponte & Croft 1998). In the same year Joachims showed that support vector machines on TF-IDF bag-of-words vectors outperformed naive Bayes, Rocchio, k-nearest neighbours (k-NN) and C4.5 decision trees on Reuters-21578 and Ohsumed (Joachims 1998). Bagga and Baldwin applied tf × log(N/df) vectors with cosine normalisation to coreference resolution across documents. On B-CUBED, a per-mention score they introduced, their best F-measure was 84.6% on 197 New York Times "John Smith" articles (Bagga & Baldwin 1998). Yamamoto and Church used suffix arrays to obtain tf and df for every substring of a corpus. They found that residual IDF surfaces names, technical terms and useful keywords. They define it as observed idf minus the idf a Poisson model predicts (Yamamoto & Church 2001). Latent Dirichlet allocation (LDA) was explicitly positioned as a successor to TF-IDF, LSI and probabilistic LSI (pLSI) for compact document description (Blei et al. 2003).
How it is used today
Search and first-stage retrieval
BM25 remains the default sparse retriever. Neural re-rankers commonly take the BM25 top 1,000 passages and reorder them. A fine-tuned BERT cross-encoder, which reads query and passage together, topped MS MARCO this way (Nogueira & Cho 2019). Dense retrievers use BM25 both as the baseline and as a source of hard negatives, meaning non-relevant passages that look relevant (Karpukhin et al. 2020). In conversational retrieval-augmented generation, an untuned BM25 with k₁ = 1.5 and b = 0.75 serves as a transparent, training-free reference point (Sivanaiah et al. 2026). TF-IDF retrieval has also been used to build datasets. Automatically retrieved paragraphs made easy unanswerable questions, which motivated crowdsourced ones in version 2.0 of the Stanford Question Answering Dataset (SQuAD) (Rajpurkar et al. 2018).
Classification and text mining
Shared-task systems still lean on TF-IDF over word and character n-grams. Examples include:
- Arabic dialect identification (Lichouri et al. 2021; AAlAbdulsalam 2022; Lichouri et al. 2024);
- offensive-language detection in Arabic tweets (Nayel 2020);
- Arabic authorship attribution, where word and character views proved complementary (Sabaa & Sabaa 2025);
- detection of machine-generated code (Ozaylar 2026).
Preprocessing choices interact with the weighting. In Arabic topic classification, raw term frequency won only when stop words were removed (Alhenawi et al. 2024). Because the models are transparent, they are also useful for audits. Logistic regression on TF-IDF n-grams exposed racial bias in five hate-speech datasets. Every model assigned African-American English tweets to negative classes, such as hate speech or harassment, at significantly higher rates than Standard American English tweets (Davidson et al. 2019).
Keyword extraction and topic description
Ranking a document's terms by TF-IDF is the simplest keyword extractor. In topic modeling, BERTopic clusters sentence embeddings and then labels each cluster with class-based TF-IDF. The weight is tf(t, c) × log(1 + A / tf(t)), where tf(t, c) is counted over the concatenated text of cluster c. A is the mean word count per class, and tf(t) is the term's frequency summed over all classes (Grootendorst 2022). One Arabic dialect study likewise merges all tweets of one dialect into a single class document before weighting its character n-gram patterns (Shammary et al. 2022).
TF-IDF against embeddings
Word embeddings and dense encoders capture synonymy, which exact term matching cannot. The trade-off depends on the setting:
| Setting | Finding |
|---|---|
| Supervised open-domain QA | Dense Passage Retrieval (DPR) beat Lucene-BM25 by 9–19 points of top-20 accuracy (Karpukhin et al. 2020) |
| Questions written by people who already knew the answer | BM25 stayed stronger than the learned Open-Retrieval Question Answering (ORQA) retriever on SQuAD and TriviaQA (Lee et al. 2019) |
| Zero-shot, unlabelled | Contriever had higher Recall@100 than BM25 on 11 of the 15 datasets of the BEIR retrieval benchmark; Recall@100 is the share of relevant documents found in the top 100 (Izacard et al. 2021). E5 was the first model trained without labels to beat BM25 on the BEIR benchmark as a whole (Wang et al. 2022) |
Mining hard negatives from the whole corpus let the ANCE dense retriever beat BM25 on 29 of 43 document-ranking queries from the TREC 2019 Deep Learning Track (Xiong et al. 2020). One comparison of unsupervised sentence representations included a TF-IDF bag-of-words baseline. When the vectors were queried directly by cosine, bag-of-words models averaged 0.55 across the similarity benchmarks, against 0.42 for order-sensitive recurrent neural networks (RNNs) (Hill et al. 2016).
Under distribution shift, TF-IDF with tree models held up in code detection. Embedding pipelines lost up to 77% of their validation macro-F1 (F1 averaged equally over classes) on the test set. XGBoost on GraphCodeBERT embeddings, for instance, fell from 89.47 to 23.09 (Sahlen et al. 2026). Still, AraBERT, an Arabic BERT model, beat a character TF-IDF model on NADI 2022 dialect identification, scoring 30.01 against 22.89 macro-F1 on TEST-A (AAlAbdulsalam 2022).
Open questions
- Robustness versus recall. Sparse features generalise surprisingly well under domain shift, yet they miss paraphrases and fail across scripts. Contriever, for example, can match Arabic queries to English documents, something exact term overlap cannot achieve (Izacard et al. 2021). The ORQA authors suggest hybrid sparse–dense retrieval as future work to combine both strengths (Lee et al. 2019).
- Below the word level. Character-level TF-IDF may capture dialect cues that the subword tokenization of a transformer such as MARBERT hides (Shammary et al. 2022). How best to choose n-gram units remains open.
- Spurious signal. The most discriminative n-grams may be country names (AAlAbdulsalam 2022) or whitespace habits (Sahlen et al. 2026), so a high weight is not always meaningful evidence.
- Principled weights. BM25 and query-likelihood models give TF-IDF-like behaviour a probabilistic footing (Ponte & Croft 1998; Robertson et al. 1994). How much dense or hybrid retrievers really gain over an untuned lexical baseline in conversational search is still being measured (Sivanaiah et al. 2026).
- Precision without coverage. In multi-label dialect identification, a tuned TF-IDF system had the highest precision of any NADI 2024 team (63.22%). Its recall was only 12.87%, so it missed most valid labels (Lichouri et al. 2024).
Key papers
Related concepts
- Inverted index — An inverted index is a data structure that maps each term of a vocabulary to a postings list of the documents containing it, usually with…
- Latent semantic indexing — Latent semantic indexing (LSI) is a retrieval and text-analysis method that factorizes a weighted term–document matrix with a truncated…
- Text classification — Text classification is the task of assigning one or more labels from a fixed, predefined set to a piece of text, such as a sentence…
More papers
Topics: 🔍 Information retrieval & search · ✍️ Text generation
Shelves: 🤖 Artificial Intelligence
Updated
This page in Markdown → https://md.research.elamur.ai/free/en/concepts/tf-idf/