🗃️ Text classification

About 18 minutes to read · every paper named here has its own page in this library

What it is

A spam filter decides whether each incoming email is spam or not. A news site files every article under a section such as sports or politics. A company sorts product reviews into positive and negative. Each is a case of text classification: a program reads a text and returns a label from a list fixed in advance. It is one of the oldest and most widely deployed tasks in natural language processing and text mining. Other tasks are often handled in the same form: natural language inference, for instance, can be treated as choosing a label for a pair of sentences (Schick & Schütze 2021).

A standard survey frames the task as approximating an unknown function. That function assigns true or false to every pair of a document and a category (Sebastiani 2002). In this framing a category is a bare symbol, and the classifier relies only on the text itself, not on outside metadata (Sebastiani 2002). Three settings are usually distinguished:

• Binary classification has two classes, such as spam versus legitimate mail, or positive versus negative opinion (Pang et al. 2002).
• Multi-class classification has more than two mutually exclusive classes, for example star ratings from 1 to 5, or one of ten topics for a question (Yang et al. 2016).
• Multi-label classification lets one text carry several labels at once. One example is a newswire story coded with several topics and regions (Lewis et al. 2004). Another is a clinical note given several diagnosis codes, a setting tested in one experiment of a general label-embedding method (Wang et al. 2018).

A multi-label problem over a set of categories can be split into one independent binary decision per category, provided the categories can be treated as independent. For that reason much of the classical literature studies the binary case (Sebastiani 2002). Thorsten Joachims, for example, trained one binary classifier for each category (Joachims 1998).

The labels can describe almost any property of a text:

• its topic;
• the sentiment it expresses, a large area covered under sentiment and emotion analysis;
• the attitude toward a single aspect or target inside a sentence (Wang et al. 2016);
• whether it is abusive or hateful (Waseem & Hovy 2016);
• its genre or author (Stamatatos et al. 2000);
• the dialogue act an utterance performs (Lee & Dernoncourt 2016).

The task is related to topic modeling and information retrieval, but it differs from both. The categories are given beforehand, rather than discovered from the data or ranked against a query.

How it works

Sebastiani organized the field around three problems (Sebastiani 2002):

1. Turning raw text into a representation a learner can use.
2. Building a classifier from labelled examples.
3. Measuring how well that classifier works.

Older systems were sets of hand-written rules. During the 1990s these gave way to a generic learning procedure that induces the classifier from a labelled collection (Sebastiani 2002).

Representing text

The classic representation is the bag of words. A document becomes a vector with one dimension per vocabulary term, recording whether the term occurs or how often. Word order is discarded. Such vectors are very high-dimensional and sparse. Joachims observed that text tasks easily involve more than ten thousand word stems, and that few of these features are useless on their own (Joachims 1998). Raw counts are commonly reweighted by term frequency–inverse document frequency (TF-IDF). This textbook scheme from information retrieval goes back to Spärck Jones (1972):

tfidf(t, d) = tf(t, d) · log( N / df(t) )

Here tf(t, d) is how often term t occurs in document d, and N is the number of documents in the collection. df(t) counts the documents in which t appears. The logarithm shrinks the weight of terms that appear almost everywhere. Vectors are often scaled to unit length afterwards, as in Joachims's experiments (Joachims 1999). Word n-grams (pairs or triples of adjacent words) can be added as extra features. Character n-grams are an alternative that copes better with misspellings and deliberate obfuscation (Schmidt & Wiegand 2017).

Neural models replace sparse counts with dense vectors. Each word, or each piece produced by subword tokenization, is mapped to a learned word embedding. A network then composes these vectors into a representation of the whole text (Kim 2014).

Learning a decision rule

Given vectors and labels, a learner fits a function that maps a new vector to a class. Most classical learners are linear. They compute a weighted sum of features and compare it with a threshold. Logistic regression, in the binary case, turns that sum into a probability:

P(y = 1 | x) = 1 / (1 + exp(−(w · x + b)))

Here x is the feature vector of a text, w is a vector of learned weights with one weight per feature, b is a bias term, and y is the label. Training adjusts w and b so that the labelled examples receive high probability. A neural classifier follows the same pattern. Its last layer is usually a softmax, the multi-class extension of this formula, placed on top of a learned text representation.

History

From rules to learning

Before the 1990s, text categorization systems were largely built by knowledge engineering: experts wrote the rules. Sebastiani's survey records how learned classifiers took over during that decade (Sebastiani 2002). Their effectiveness became comparable to that of trained human indexers, at far lower labour cost. The gains came largely from boosting and support vector machines (SVMs) (Sebastiani 2002). Rocchio's relevance-feedback method (Rocchio 1971) and nearest-neighbour classifiers served as common baselines.

Several papers from 1998 shaped the following decade. Joachims argued that SVMs suit text, because their capacity is controlled by the margin rather than by the number of features. In his experiments they beat naive Bayes, Rocchio, k-nearest neighbours (k-NN) and C4.5 decision trees on two collections (Joachims 1998). In the same year, Andrew McCallum and Kamal Nigam separated two incompatible models that had both been called naive Bayes (McCallum & Nigam 1998).

Labels were costly, so attention soon turned to unlabelled text. Nigam and colleagues combined naive Bayes with expectation-maximization (Nigam et al. 2000). Building on Vapnik's transductive framework, Joachims developed transductive SVMs with a training algorithm that scales to large collections. These SVMs fit a separator to the labelled data and the waiting test set together (Joachims 1999). In 2002, Bo Pang, Lillian Lee and Shivakumar Vaithyanathan asked whether opinion could be classified like topic. They found it noticeably harder (Pang et al. 2002).

Neural models

Around 2014 the field moved to neural networks built on pretrained word vectors. Yoon Kim showed that a single convolutional layer over word2vec embeddings could match far more elaborate systems (Kim 2014). In 2015 three alternatives followed:

• networks that read raw characters (Zhang et al. 2015);
• convolutions applied directly to one-hot word vectors (Johnson & Zhang 2015);
• deep averaging networks that ignore word order altogether (Iyyer et al. 2015).

Hierarchical models then encoded sentences first and documents second, with recurrent units (Tang et al. 2015) and with attention (Yang et al. 2016). In 2017 a very deep character-level convolutional network (ConvNet) showed that depth can pay off for text as it had for images (Conneau et al. 2017). In parallel, fastText showed that a well-engineered linear model remains a hard baseline to beat (Joulin et al. 2017).

Pretraining and prompting

The next shift was to pretrain a language model on raw text and then fine-tune it on the classification task. Universal Language Model Fine-tuning (ULMFiT) applied this recipe to text classification in 2018 (Howard & Ruder 2018). BERT, built on the Transformer encoder of Vaswani and colleagues (2017), made it the default. It added a single output layer and set new results on eleven benchmark tasks (Devlin et al. 2019). Later work recast classification as filling a blank in a natural-language prompt (Schick & Schütze 2021) or as checking whether a text entails a label description (Yin et al. 2019). Both moves led toward few-shot and zero-shot use of large language models.

Methods

Naive Bayes

Naive Bayes applies Bayes' rule and assumes that words are independent of one another once the class is known. In the multinomial event model, a document is a sequence of word draws from a class-specific distribution:

ĉ = argmax over c of  P(c) · ∏ᵢ P(wᵢ | c)

P(w | c) = (count(w, c) + 1) / (Σ_w′ count(w′, c) + |V|)

Here ĉ is the predicted class, P(c) is the prior probability of class c, w₁ … wₙ are the word tokens of the document, count(w, c) is how often word w occurs in the training documents of class c, and |V| is the vocabulary size. The added one is Laplace smoothing, which keeps unseen words from zeroing out the product. In practice the product is computed as a sum of logarithms.

The multi-variate Bernoulli model instead records only whether each vocabulary word is present or absent. It also includes explicit factors for absent words. McCallum and Nigam derived both models from one mixture framework (McCallum & Nigam 1998). Bernoulli sometimes won with very small vocabularies. The multinomial model generally won as the vocabulary grew, and across five corpora it cut error by 27% on average (McCallum & Nigam 1998).

Support vector machines and logistic regression

SVMs find the hyperplane that separates two classes with the widest margin. Joachims argued that sparse, high-dimensional and often linearly separable text vectors are exactly the case margins handle well. On his data, no overfitting appeared even with all features, so separate feature selection became unnecessary (Joachims 1998). The table gives his microaveraged precision/recall-breakeven scores on the Reuters ModApte split.

Source: (Joachims 1998).

On the RCV1 benchmark, SVMs again came out ahead of k-NN and Rocchio-style classifiers at every category frequency (Lewis et al. 2004). The same study found that the effectiveness gap between two SVM variants came mainly from how decision thresholds were chosen (Lewis et al. 2004).

Sida Wang and Christopher Manning showed that the ranking of naive Bayes and SVMs depends on the data. Multinomial naive Bayes did better on short snippets, while SVMs did better on full-length reviews (Wang & Manning 2012). Their hybrid, NBSVM (a naive Bayes–SVM combination), trains an SVM on features scaled by naive Bayes log-count ratios. It reached 91.22% accuracy on IMDB (Wang & Manning 2012). Logistic regression is the other standard linear learner. Over character n-grams it served as a strong hate speech baseline (Waseem & Hovy 2016). A one-versus-rest variant with L2 regularization was used for a three-way hate, offensive and neither task (Davidson et al. 2017).

Convolutional networks

Kim's convolutional neural network (CNN) slides filters of widths 3, 4 and 5 over the matrix of word embeddings. It keeps each filter's maximum response and classifies with a softmax layer regularized by dropout (Kim 2014). Pretrained vectors helped much more than random initialization. On the MR movie-review set, the randomly initialized variant reached only 76.1% accuracy, against 81.5% for the variant that fine-tunes word2vec vectors (Kim 2014).

Character-level ConvNets took the opposite route. They used a 70-symbol alphabet with no notion of words (Zhang et al. 2015). Their main finding concerned data size. TF-IDF n-gram models stayed strong up to a few hundred thousand training samples, and the character model pulled ahead only at millions (Zhang et al. 2015). VDCNN (very deep convolutional neural network) extended this idea with up to 49 convolutional layers. Accuracy improved up to 29 layers, and the deepest networks needed residual shortcuts (Conneau et al. 2017).

Rie Johnson and Tong Zhang fed one-hot vectors directly into the convolution, so the network learned embeddings of whole text regions. They report this as the first successful use of word order for topic classification (Johnson & Zhang 2015).

Recurrent, hierarchical and attention models

Recurrent neural networks (RNNs) read a text word by word and carry a hidden state forward. For long documents, a two-level design proved effective. Duyu Tang, Bing Qin and Ting Liu built sentence vectors with a CNN or a long short-term memory (LSTM) network, then composed them with a gated recurrent network. They found that a plain document-level RNN worked poorly, while gating helped a great deal (Tang et al. 2015).

The Hierarchical Attention Network adds attention at both the word and the sentence level. The weights are scored against learned context vectors, and they can be shown to readers as an explanation of the decision (Yang et al. 2016). Label-embedding attention places labels in the same space as words and weights each word by how well it matches each label (Wang et al. 2018). For a fixed two-level taxonomy, HDLTex (Hierarchical Deep Learning for Text classification) trains a parent network for the broad field and a child network for each subfield (Kowsari et al. 2017).

Aspect-level sentiment became a testbed for target-aware architectures:

• recursive networks over parses re-rooted at the target (Dong et al. 2014);
• target-dependent LSTMs (Tang et al. 2016);
• aspect embeddings that steer attention (Wang et al. 2016);
• multi-hop memory networks (Tang et al. 2016);
• multi-grained attention (Fan et al. 2018);
• target-specific transformations (Li et al. 2018);
• graph convolutions over parses produced by dependency parsing (Zhang et al. 2019; Tang et al. 2020).

Against this growing complexity, the deep averaging network made a deflationary point. It averages the word embeddings, applies a few nonlinear layers, and comes close to syntax-aware models while training in minutes (Iyyer et al. 2015).

Fine-tuned transformers

BERT pretrains a bidirectional Transformer with two objectives: masked-token prediction and next-sentence prediction. For classification, the representation of [CLS], a special token added to the input for this purpose, is passed to a new output layer, and the whole network is fine-tuned (Devlin et al. 2019). Chi Sun and colleagues studied the choices this leaves open (Sun et al. 2019). They recommend:

• first continuing pretraining on in-domain text, which reduced error by 18.57% on average relative to standard fine-tuning without that extra stage;
• optionally fine-tuning on several related tasks together;
• finally fine-tuning on the target task, with learning rates that decrease toward the lower layers.

Other methods build on the pretrained encoder. One example is MixText, which interpolates hidden states of pairs of texts for semi-supervised training (Chen et al. 2020).

Few-shot and zero-shot classification

Pattern-Exploiting Training (PET), from Timo Schick and Hinrich Schütze, rewrites each input as a cloze question. A verbalizer then maps every label to one vocabulary word, so the masked language model's own prediction head acts as the classifier (Schick & Schütze 2021). PET fine-tunes one model per pattern on a handful of labelled examples. The ensemble then labels unlabelled text, and a final classifier is distilled from those soft labels (Schick & Schütze 2021). This line of work belongs to few-shot learning and prompt engineering.

For the zero-shot case, Wenpeng Yin, Jamaal Hay and Dan Roth cast each candidate label as a hypothesis, such as "the text is about sports". They then asked a model trained on textual entailment whether the input supports it (Yin et al. 2019).

Learning with few labels, other domains and other languages

Many methods target the label bottleneck without large pretrained models:

• EM with naive Bayes: on newsgroups, adding 10,000 unlabelled articles cut the labelled data needed for 70% accuracy from 2,000 articles to about 600 (Nigam et al. 2000).
• Transductive SVMs: on WebKB, the average precision/recall breakeven rose from 57.2 with inductive SVMs to 62.4 (Joachims 1999).
• Virtual adversarial training on word embeddings: it regularizes LSTM classifiers and exploits unlabelled text (Miyato et al. 2016).
• Easy Data Augmentation (EDA): four simple word edits. Models trained with it on half the data matched baselines trained on all of it, although the benefit faded with pretrained models (Wei & Zou 2019).
• Distant supervision: emoticons served as labels for about 10 million tweets, used to learn sentiment-specific embeddings (Tang et al. 2014).

Domain shift is a related problem. A classifier trained on book reviews loses accuracy on kitchen appliances. Structural correspondence learning with mutual-information pivots addressed such shifts. On average it reduced the relative error they cause by 46% compared with a classifier without adaptation, and by 30% compared with the original method (Blitzer et al. 2007). Adversarial multi-task learning keeps shared and task-private features apart when many small datasets are trained together (Liu et al. 2017).

Cross-lingual transfer, part of multilingual natural language processing, has two prominent approaches. One is co-training over machine-translated English and Chinese views of each review (Wan 2009). The other is an adversarial averaging network that learns features hiding the input language (Chen et al. 2018). Both help in low-resource languages.

Efficiency

Linear models remain attractive where speed and memory matter. Product quantization of fastText's embedding tables shrinks models to well under a megabyte with little loss in accuracy (Joulin et al. 2016). This connects text classification to work on efficient models.

Datasets and benchmarks

Collections

Shared collections defined much of the field's progress. They are covered more broadly under evaluation and benchmarks.

RCV1 was assembled because older test collections were small, poorly documented and overused, so that a whole community could overfit to them. Its authors also documented coding errors and released a cleaned version (Lewis et al. 2004). Sentence-level suites became the standard testbed for short-text models (Kim 2014):

• MR (movie-review sentiment);
• SST (the Stanford Sentiment Treebank);
• Subj (subjectivity);
• TREC (question type);
• CR (customer reviews);
• MPQA (opinion polarity).

The large datasets of Zhang, Zhao and LeCun became the standard for document-level models (Conneau et al. 2017). For zero-shot evaluation, Yin and colleagues fixed splits of seen and unseen classes across topic, emotion and situation labels (Yin et al. 2019).

Evaluation measures

For one class, every prediction falls into one of four cells. A true positive (TP) is a text correctly assigned to the class, and a false positive (FP) is a text wrongly assigned to it. A false negative (FN) is a member of the class that the classifier missed, and a true negative (TN) is a non-member correctly left out. The standard measures, in the textbook form popularized by van Rijsbergen (1979), are:

accuracy  = (TP + TN) / (TP + TN + FP + FN)
precision = TP / (FP + TP)
recall    = TP / (FN + TP)
F1        = 2 · precision · recall / (precision + recall)

Precision is the share of predicted members that are correct. Recall is the share of true members that are found. F1 is their harmonic mean, which is low whenever either of the two is low.

With many categories, the measures can be averaged in two ways. Microaveraging pools the counts of all categories before computing the measure, so frequent classes dominate the result. Macroaveraging computes the measure per class and then averages, so every class counts equally. Older work often reported the precision/recall breakeven point, where the two measures are equal (Joachims 1998). Twitter sentiment work commonly reports macro-F1 (Tang et al. 2014). Multi-label clinical coding adds ranking measures, such as the area under the receiver operating characteristic curve (AUC) and precision at 5 (Wang et al. 2018).

Class imbalance

Accuracy misleads when one class is rare, since always predicting the majority class already scores high. Several examples show how common this is:

• In the Reuters binary tasks of Nigam and colleagues, the positive class made up between 1% and 30% of documents (Nigam et al. 2000).
• In a lexicon-sampled Twitter corpus, only about 5% of tweets were majority-labelled as hate speech (Davidson et al. 2017).
• On that same corpus, overall precision and recall were around 0.9, yet hate-class precision was only 0.44 (Davidson et al. 2017).
• The racist class of the Waseem and Hovy corpus came from only 9 users (Waseem & Hovy 2016).

Per-class and macro-averaged scores, tuned decision thresholds (Lewis et al. 2004) and deliberately balanced data construction (Hartvigsen et al. 2022) are the usual responses.

Label noise

Gold labels are themselves uncertain. Sebastiani notes that trained human indexers disagree with one another (Sebastiani 2002). In hate speech, who does the labelling matters. Unscreened crowd workers flagged hate more often than expert annotators did. In cross-validation within each label set, the best features reached about 91.2–91.5 F1 on expert labels, against about 86.3–86.5 on crowd labels (Waseem 2016). Dialect can bias labels as well. African American English tweets were disproportionately marked offensive, and classifiers trained on those labels reproduced the disparity (Sap et al. 2019). Labels obtained by distant supervision, such as emoticons, are cheap but noisy by construction (Tang et al. 2014).

People and institutions

Bo Pang and Lillian Lee, with Shivakumar Vaithyanathan, wrote the movie-review sentiment study described under History (Pang et al. 2002). Lee is at Cornell University.

At Harbin Institute of Technology, Ting Liu and Bing Qin worked with Duyu Tang on a run of neural sentiment classifiers: sentiment-specific embeddings, gated document models, target-dependent LSTMs and deep memory networks (Tang et al. 2014; Tang et al. 2015; Tang et al. 2016; Tang et al. 2016). Tang also co-authored the adaptive recursive network with Furu Wei and Ming Zhou (Dong et al. 2014). Wei is at Microsoft.

Yann LeCun co-authored both the character-level ConvNet and its very deep successor (Zhang et al. 2015; Conneau et al. 2017). Zichao Yang and Diyi Yang worked together on hierarchical attention networks and on MixText (Yang et al. 2016; Chen et al. 2020). Both were at Carnegie Mellon University at the time of the hierarchical attention work, and Diyi Yang is now at Stanford University. Xuanjing Huang of Fudan University co-authored adversarial multi-task learning and the BERT fine-tuning study (Liu et al. 2017; Sun et al. 2019). Lidong Bing contributed transformation networks for target-oriented sentiment (Li et al. 2018).

Timo Schick and Hinrich Schütze of Ludwig-Maximilians-Universität München developed PET (Schick & Schütze 2021). Claire Cardie of Cornell co-authored adversarial cross-lingual sentiment classification (Chen et al. 2018).

On abusive language, Zeerak Waseem built the early Twitter hate speech corpora and studied how annotators shape them (Waseem & Hovy 2016; Waseem 2016). Maarten Sap and Noah A. Smith exposed dialect bias in toxicity labels (Sap et al. 2019). Smith is at the Allen Institute for Artificial Intelligence. Sap later co-authored the ToxiGen dataset (Hartvigsen et al. 2022).

Earlier foundations came from several researchers:

• Thorsten Joachims on SVMs (Joachims 1998);
• Andrew McCallum and Kamal Nigam on naive Bayes and EM (McCallum & Nigam 1998; Nigam et al. 2000);
• Fabrizio Sebastiani's survey (Sebastiani 2002);
• David D. Lewis and Yiming Yang on RCV1 (Lewis et al. 2004).

Open questions

Robustness

High benchmark scores do not mean a classifier is robust. TextFooler swaps influential words for synonyms. It pushed most target models, BERT included, below 10% accuracy while changing under a fifth of the words (Jin et al. 2019). BAE (BERT-based Adversarial Examples) uses a masked language model to propose substitutions and insertions that fit the context. Its outputs were judged more natural than TextFooler's and remained effective (Garg & Ramakrishnan 2020). Adversarial retraining helped only partly (Jin et al. 2019). How to build classifiers whose decisions hold up under meaning-preserving edits remains open.

Fairness and what a label means

For socially sensitive labels, the text alone may not settle the answer. Dialect and speaker identity change how annotators judge offensiveness (Sap et al. 2019). Merely offensive language is easily confused with hate speech (Davidson et al. 2017). Classifiers often treat any mention of a minority group as a toxic signal (Hartvigsen et al. 2022). A shared definition and comparable datasets are still missing in this area (Schmidt & Wiegand 2017). These questions overlap with work on bias, fairness and safety.

Scarce labels and domain shift

PET (Schick & Schütze 2021) and entailment-based zero-shot classification (Yin et al. 2019) reduce the need for annotation. Their sensitivity to the wording of patterns and label descriptions is not settled. Neither is the question of when a small fine-tuned model is still the better choice. Domain shift persists even with pretraining. Continued in-domain pretraining still brought large gains in one controlled study (Sun et al. 2019).

Interpretability and the value of baselines

Attention weights are offered as explanations (Yang et al. 2016; Wang et al. 2018). Whether they reflect how the model actually decides belongs to the wider study of interpretability.

Finally, the field keeps rediscovering that simple models are hard to beat:

• TF-IDF n-grams on medium-sized data (Zhang et al. 2015);
• NBSVM on sentiment (Wang & Manning 2012);
• averaged embeddings with a few nonlinear layers (Iyyer et al. 2015).

Each new architecture is therefore expected to show a clear margin over well-tuned baselines.

The library in figures