🔮 Semantics

About 17 minutes to read · every paper named here has its own page in this library

What it is

Every sentence has a form, made of sounds, letters or signs, and a meaning. Semantics studies the meaning side. Take "Every dog barks." A speaker of English knows which situations would make it true and which would make it false. They know that, as long as there are dogs at all, it says something stronger than "Some dog barks." They know that "dog" names a kind of animal and that "barks" names something such animals do. Semantics tries to state that knowledge explicitly.

The field has a close neighbour, pragmatics, and the border between them is a long-running theme. Semantics is usually taken to cover the conventional, literal meaning of an expression. Pragmatics covers what a speaker conveys by using it in a particular context. "Can you pass the salt?" is literally a question about ability but is used as a request. It is a classic indirect speech act, studied in speech act theory. Saying "Some of the students passed" usually conveys that not all of them did, although the sentence would still be true if all had passed. Such extra, context-driven meaning is called implicature, after Grice (1975).

Work since the 2010s sits right on this border. Kennedy gives numerals such as "six" an exact literal meaning. He then derives the sense of speaker ignorance carried by "at least six", but not by "more than six", from ordinary pragmatic reasoning about weaker and stronger alternatives (Kennedy 2015). Moss proposes a different kind of meaning for epistemic words like "might" and "probably": a sentence is paired with a set of probability measures rather than a set of possible worlds. Combined with a pragmatics of assertion, this explains why some classically valid argument forms fail with such words (Moss 2015).

Two broad branches organise the subject:

• Lexical semantics studies the meanings of words. It covers their senses, how senses relate (synonymy, antonymy, hyponymy), and how a word's meaning shifts with context.
• Compositional semantics studies how word meanings combine, guided by syntax, into the meanings of phrases and sentences.

In computational linguistics and natural language processing, semantics also names a family of engineering tasks:

• choosing word senses;
• labelling who did what to whom;
• mapping sentences to formal meaning representations;
• learning vector representations of words.

Together these serve natural language understanding.

How it works

Word meaning

Most words are polysemous: they have several related senses. Dictionaries, the products of lexicography, list these senses. The lexical database WordNet (Miller, 1995) organises English nouns, verbs, adjectives and adverbs into synonym sets. These sets are linked by relations such as hypernymy ("dog" is a kind of "animal"). WordNet became the standard sense inventory for computational work. Word sense disambiguation is the task of choosing the right sense for a word in a given context.

Polysemy has practical consequences. Wu and Palmer studied how English "break" should be translated into Mandarin. Mandarin has no general verb for breaking and must name both the action and the resulting state. On 157 Brown corpus sentences with "break", the commercial transfer system TranStar translated only 30 correctly, while a human translator used 68 distinct Chinese expressions for the same sentences (Wu & Palmer 1994). Their remedy places verb concepts in a hierarchy so that semantic similarity between concepts can be measured along it. A system can then pick a good translation even when no exact lexical match exists (Wu & Palmer 1994).

Whether senses are discrete at all is itself contested. Untrained speakers were asked to rate how well each WordNet sense fits an occurrence, and the same annotator often rated more than one sense highly. Conventional WordNet-based single-sense tagging reaches only about 67–78% inter-annotator agreement (Erk et al. 2013). Pustejovsky's generative lexicon goes further. It argues that word meanings are built and adjusted productively in context rather than simply listed (Pustejovsky 1991).

Composing meaning

The central principle of compositional semantics is often attributed to Frege, though he never stated it in this form. It says that the meaning of a complex expression is a function of the meanings of its parts and the way they are combined. In truth-conditional semantics, the meaning of a declarative sentence is identified with its truth conditions, that is, the conditions under which it would be true. "Every dog barks" can be written in first-order logic:

∀x (dog(x) → barks(x))

The symbols read as follows:

• ∀x reads "for every individual x".
• dog(x) and barks(x) are predicates that hold of an individual if it is a dog or if it barks.
• → is material implication ("if … then").

The formula is true in a situation exactly when nothing in it is a dog that fails to bark.

Composition can be made mechanical with the lambda calculus, as Montague (1970) did. The determiner "every" is given a meaning that waits for two properties:

⟦every⟧ = λP. λQ. ∀x (P(x) → Q(x))
⟦every⟧(⟦dog⟧)(⟦barks⟧) = ∀x (dog(x) → barks(x))

The double brackets ⟦ ⟧ stand for "the meaning of". λP and λQ mark slots for properties (functions from individuals to truth values). Applying the determiner to ⟦dog⟧ and then to ⟦barks⟧ fills the slots, a step called beta reduction, and yields the formula above.

Replacing ⟦every⟧ with ⟦some⟧ = λP. λQ. ∃x (P(x) ∧ Q(x)), where ∃ means "there is" and ∧ means "and", gives "Some dog barks" with no other change. This is how a theory relates the two sentences. The second follows from the first only once we assume, as speakers normally do, that there are dogs at all. Without that assumption of existential import, the universal formula is true in a situation with no dogs, while the existential one is false.

Meaning from distribution

A third tradition starts from the distributional hypothesis of Harris (1954) and Firth (1957): words found in comparable contexts usually have related meanings. Distributional semantics counts contexts in a corpus. It represents each word as a vector of those counts or of weights derived from them. Turney and Pantel organise this field by the shape of the underlying matrix: term–document, word–context or pair–pattern. They make derivation from corpus event frequencies the defining property of such models (Turney & Pantel 2010). Similarity between two word vectors u and v is typically the cosine of the angle between them:

cos(u, v) = (u · v) / (‖u‖ ‖v‖)

Here u · v is the dot product (the sum of the products of matching components), and ‖u‖ is the length of u. Values near 1 mean the words keep similar company. These models perform strongly on word-level tests. Rapp's (2003) model answered 92.5% of TOEFL synonym questions correctly. The average score of human TOEFL test-takers, who are learners of English, is 64.5% (Turney & Pantel 2010).

History

The modern study of meaning began in logic and philosophy:

• Frege (1892) distinguished sense from reference.
• Tarski (1933) defined truth for formal languages.
• Montague (1970, 1973) argued that natural languages could be treated with the same model-theoretic tools, which founded formal semantics.
• Grice (1975) separated what is said from what is implicated.
• Kamp (1981) introduced Discourse Representation Theory to handle pronouns and meaning across sentences.

Linguistics also produced alternatives to the logical programme. Fillmore's frame semantics (1976, 1982) holds that words evoke structured scenes, or frames, with participants. Cognitive linguistics and construction grammar pushed in related directions, as did the generative lexicon (Pustejovsky 1991). Frame semantics was later made computational in the Berkeley FrameNet database. The German SALSA corpus reused FrameNet's inventory of about 600 frames and found that most carried over to German (Burchardt et al. 2006).

From the late 1990s a statistical turn took hold. Researchers increasingly learned semantic parsers from data instead of writing grammars by hand, as in systems from 2008 and 2011 (Lu et al. 2008; Liang et al. 2011). They built vector models of word meaning in context (Erk & Padó 2008; Reisinger & Mooney 2010). They also proposed general corpus-based semantic memories that serve many tasks at once (Baroni & Lenci 2010).

Around 2010–2014 a burst of work tried to unite the distributional and compositional traditions. It produced:

• learned composition functions for adjective–noun phrases (Guevara 2010; Zanzotto et al. 2010);
• neural composition over parse trees (Socher et al. 2012; Socher et al. 2013);
• fast predictive word embeddings (Mikolov et al. 2013; Mikolov et al. 2013).

In 2013 Abstract Meaning Representation (AMR) offered a single sentence-level semantic annotation for English. It was modelled on the way treebanks had unified syntax (Banarescu et al. 2013). An active line of parsing and generation research followed.

After 2018, contextual encoders based on the Transformer and then large language models changed the questions. Researchers began probing what such models know about word meaning (Vulić et al. 2020). They also debated whether a model trained only on text can learn meaning at all (Bender & Koller 2020). Others measured how well these models compose (Press et al. 2023; Dziri et al. 2023).

Methods

Semantic role labelling and frames

Semantic role labelling identifies the predicate of a clause and labels its arguments with roles such as agent, patient, instrument, time or place. Inventories differ in style:

• FrameNet uses frame-specific roles. SALSA added such roles to the German TIGER treebank, covering about 20,000 predicate instances in its first release. It needed new roles, for example for German dative possessors (Burchardt et al. 2006).
• PropBank uses numbered, verb-specific arguments, and AMR inherits these as :arg0 to :arg5 (Banarescu et al. 2013).
• Universal Decompositional Semantics drops categorical labels altogether, building on Dowty's proto-roles (White et al. 2016). It asks ordinary speakers simple questions about each participant, such as whether it acted deliberately or changed state. The answers are layered on Universal Dependencies treebanks (White et al. 2016).

Semantic parsing and meaning representations

Semantic parsing maps a sentence to a formal representation of its meaning. Early statistical systems targeted database queries. Lu and colleagues generated sentence and meaning jointly as a hybrid tree. They reached an F-score of 85.2 for predicted meaning representations on the GEOQUERY geography questions (Lu et al. 2008). Liang, Jordan and Klein learned dependency-based compositional semantics, a representation designed for answering questions over databases (Liang et al. 2011).

Broad-coverage meaning representations aim higher, at the meaning of arbitrary sentences:

• AMR writes a sentence's meaning as a rooted, directed, labelled graph. Concepts are nodes, and roughly 100 relation types label the edges. Pronouns become reused variables, and a verb and its nominalisation get the same graph. AMR deliberately leaves the text-to-graph mapping unspecified and does not claim to be an interlingua (Banarescu et al. 2013).
• Discourse Representation Structures (DRSs), built by the Boxer tool, follow Kamp's theory. They underlie the Parallel Meaning Bank. That corpus projects English annotations onto German, Italian and Dutch translations, on the assumption that a faithful translation has the same logical form (Abzianidze et al. 2017).

Parsing into AMR became a benchmark problem of its own. Because the sembank gives no word-to-node links, these first had to be learned. One method adapts statistical translation alignment to graph–string pairs and reaches an alignment F-score of 83.1 on hand-aligned test data (Pourdamghani et al. 2014). Parsers are scored with Smatch, which counts matching relation triples between a predicted and a gold graph. Damonte and colleagues argued that a single Smatch figure hides differences between parsers. They released separate metrics for named entities, negation, reentrancy and other subtasks (Damonte et al. 2017).

The scores come from different data releases, so they show a trajectory rather than a strict ranking. With gold concepts, JAMR scored 80 Smatch F1, against an inter-annotator agreement of 83. This located most of its error in the first stage (Flanigan et al. 2014). Later work extended AMR parsing to languages other than English (Damonte & Cohen 2018).

Graph representations also support generation and downstream tasks:

• Synchronous hyperedge replacement grammars pair graph and string derivations, so a single grammar can both parse and generate. This enabled a pilot semantics-based translation system (Jones et al. 2012).
• AMR graphs of news clusters can be merged and pruned for multi-document text summarization (Liao et al. 2018).
• AMR can support question answering over knowledge bases (Kapanipathi et al. 2021).
• DRS graphs can guide sentence simplification (Narayan & Gardent 2014).

Logic-based representations make inference possible. The Meaning Factory was an entry to SemEval-2014 Task 1, whose data are described in A SICK cure for the evaluation of compositional distributional semantic models. The system converted DRSs to first-order logic and called a theorem prover and model builders, with WordNet supplying background axioms. It reached about 82% accuracy on the task's three-way entailment labels (Bjerva et al. 2014).

A semantic layer was also explored early for question answering. Webclopedia matched parse trees and expected answer types instead of relying on word overlap alone (Hovy et al. 2001).

Embeddings and composition

Predictive embeddings replaced counting with learning. The skip-gram model predicts the neighbours of a word, and CBOW predicts a word from its neighbours. Both are shallow log-linear models, cheap enough to learn good vectors from a 1.6-billion-word corpus in under a day (Mikolov et al. 2013). Negative sampling trains a classifier to tell a real context word from random noise words, and frequent-word subsampling speeds training further (Mikolov et al. 2013). A typical negative-sampling objective for a word w and a context word c is:

log σ(v_c · v_w) + Σ_{i=1..k} log σ(−v_{n_i} · v_w)

The terms are as follows:

• v_w and v_c are the vectors of the word and of a true context word.
• n_1 … n_k are k noise words drawn from a smoothed unigram distribution.
• σ is the logistic function, which turns a score into a probability.

The objective rewards a high score for the true pair and low scores for the noise pairs. The authors recommend k between 5 and 20 for small corpora and between 2 and 5 for large ones (Mikolov et al. 2013).

A single vector per word blurs its senses. Multi-prototype models cluster a word's occurrences and keep one vector per cluster (Reisinger & Mooney 2010). The structured vector space model adjusts a word's vector by the selectional preferences of the word it combines with (Erk & Padó 2008). Contextual encoders compute a new vector for every occurrence. Even these encoders retain substantial type-level lexical knowledge, mostly in their lower layers (Vulić et al. 2020).

For phrases, composition functions range from fixed to learned:

• Fixed operations add or multiply word vectors. For noun compounds, addition outperformed multiplication (Reddy et al. 2011).
• Learned linear maps can be fitted to corpus-observed phrase vectors (Guevara 2010), or to dictionary definitions (Zanzotto et al. 2010).
• The matrix-vector recursive network gives every word both a vector and a matrix, so that operators like "not" can transform their arguments. It scored 79.1 F1 on SemEval-2010 relation classification without extra features (Socher et al. 2012). A related recursive model was trained on a treebank annotated for sentiment at every phrase (Socher et al. 2013).
• Explicit compositional inference can be combined with learning. This gave 90.7% accuracy on expression-level sentiment polarity (Choi & Cardie 2008).
• Composition can extend beyond the sentence:
  • to dialogue acts (Kalchbrenner & Blunsom 2013);
  • to implicit discourse relations that also need entity representations (Ji & Eisenstein 2015);
  • across languages, either from parallel text alone (Hermann & Blunsom 2014) or by also drawing alignment signal from monolingual corpora (Ouyang et al. 2020).

Distributional vectors can also support entailment. Entailment between quantifier phrases, as in "many dogs" entails "some dogs", turned out to need different signals from entailment between nouns. Formal type theory predicts this split (Baroni et al. 2012).

A further line grounds word meaning in perception:

• Visual vectors built from tagged images help most on perceptual properties such as colour (Bruni et al. 2012).
• Autoencoders fuse textual and visual attribute vectors (Silberer & Lapata 2014).
• Sentence and image vectors can share one space (Socher et al. 2014).
• Detected object tags can anchor image regions to words in vision-and-language pre-training, a strand of multimodal research (Li et al. 2020).

Datasets and benchmarks

Semantic resources fall into two kinds. Sembanks annotate meaning structure. Judgement datasets collect human intuitions about similarity, sense or inference.

Some benchmarks were designed to defeat shortcuts. In WiC both sentences contain the same word, so a static embedding cannot beat chance. BERT-large scored roughly 64% there, against about 80% for human annotators (Pilehvar & Camacho-Collados 2019). The Decomp word-sense layer covers 18,054 arguments. Each was rated by several crowd workers, who could choose more than one WordNet sense (White et al. 2016). One grounded similarity set rates 7,576 noun pairs separately for semantic and visual similarity (Silberer & Lapata 2014).

Analogy tests ("Paris is to France as Rome is to …") became the signature evaluation of embeddings. They were often, though far from always, answered by simple vector arithmetic (Mikolov et al. 2013). The best 300-dimensional negative-sampling models reached about 61% accuracy on such a test (Mikolov et al. 2013). Entailment pairs scored for both inference label and graded relatedness tested formal and distributional systems side by side in SemEval-2014 Task 1 (Bjerva et al. 2014).

Semantics overlaps with several related research topics:

• lexical semantics and word embeddings;
• semantic parsing and meaning representation;
• reasoning;
• multiword expressions.

People and institutions

Several research groups shaped the computational side of the field.

AMR was a joint effort. Its sembanking paper lists among its contributors Martha Palmer, Kevin Knight, Ulf Hermjakob and Nathan Schneider (Banarescu et al. 2013). Palmer had earlier co-authored the conceptual verb-similarity work on lexical selection (Wu & Palmer 1994). Knight and Hermjakob, with first author Nima Pourdamghani, built the first learned aligner between AMR graphs and English strings (Pourdamghani et al. 2014). Knight also co-authored graph-grammar translation (Jones et al. 2012), and Hermjakob co-authored semantics-based answer extraction (Hovy et al. 2001). Schneider is now at Georgetown University, and Hermjakob is at the University of Southern California.

On AMR parsing and meaning representations:

• Jeffrey Flanigan and Chris Dyer co-authored JAMR (Flanigan et al. 2014). Flanigan is now at the University of California, Santa Cruz.
• Nianwen Xue, now at Brandeis University, co-developed transition-based AMR parsing, described in A Transition-based Algorithm for AMR Parsing.
• Johan Bos, of the University of Groningen, led work on the Parallel Meaning Bank and on character-level AMR parsing. He also co-authored the logic-based Meaning Factory (Abzianidze et al. 2017; Noord & Bos 2017; Bjerva et al. 2014).
• Stephan Oepen (University of Oslo) and Dan Flickinger (Stanford University) are associated with grammar-based semantic representations such as English Resource Semantics. They are also linked to cross-framework parsing, as in MRP 2020: The Second Shared Task on Cross-Framework and Cross-Lingual Meaning Representation Parsing.
• Benjamin Van Durme, now at Johns Hopkins University and Microsoft, co-authored Universal Decompositional Semantics (White et al. 2016).

Distributional semantics has its own lineage:

• Marco Baroni, now at ICREA and Universitat Pompeu Fabra, co-developed Distributional Memory with Alessandro Lenci, now at the University of Pisa (Baroni & Lenci 2010). Baroni also worked on multimodal and entailment-oriented distributional models (Bruni et al. 2012; Baroni et al. 2012).
• Sebastian Padó, at the University of Stuttgart, co-authored the structured vector space model and the SALSA corpus (Erk & Padó 2008; Burchardt et al. 2006).
• Mirella Lapata, at the University of Edinburgh, co-authored grounded meaning representations (Silberer & Lapata 2014).
• Raymond J. Mooney, at The University of Texas at Austin, co-authored multi-prototype models of word meaning (Reisinger & Mooney 2010) and early learned semantic parsers.

Charles J. Fillmore, the originator of frame semantics, co-authored the Berkeley FrameNet project papers, among them The Berkeley FrameNet Project. More recently, Shira Wein has examined how well AMR carries across languages. One example is Assessing the Cross-linguistic Utility of Abstract Meaning Representation.

Open questions

Can form alone yield meaning?

Bender and Koller define meaning as a relation between expressions and something outside language. That something is either communicative intents or conventional meanings that can be checked against a world. They argue that a system trained only to predict strings receives no signal about that relation (Bender & Koller 2020). Their octopus thought experiment became a reference point for the debate. In it, an eavesdropper learns message patterns but cannot respond to new situations. They write the relation compactly:

M ⊆ E × I

E is the set of linguistic expressions and I the set of communicative intents. Understanding, on this view, means recovering an intent i from an expression e.

Several lines of work are sometimes cited against this view. They suggest that text carries more than surface form:

• distributional structure;
• multimodal grounding (Bruni et al. 2012; Li et al. 2020);
• lexical knowledge recoverable from encoders (Vulić et al. 2020).

Whether this amounts to meaning, or only to an increasingly good correlate of it, remains open. The question links semantics to interpretability and to the philosophy of language.

Do neural models compose?

Compositionality is the property that formal semantics builds in and neural models must learn. The evidence is mixed. Natural language inference models scored well on standard tests even with shuffled training words. Yet they failed badly when subject and object were swapped. The DAM model labelled about 99.7% of 971 such swapped SNLI pairs, constructed as contradictions, as entailment (Nie et al. 2018).

Large transformer models handle shallow multi-step problems but degrade as depth grows. On three-digit by three-digit multiplication, GPT-4 reached only about 59% accuracy (Dziri et al. 2023). Language models also often answer the sub-questions of a compositional question correctly while failing the whole question (Press et al. 2023).

Part of the difficulty is definitional. Compositionality breaks into several distinct generalisation abilities that can be tested separately (Hupkes et al. 2019). Idioms and non-literal compounds also show that natural language is only partly compositional (Reddy et al. 2011).

How universal can a meaning representation be?

AMR was designed for English and says so (Banarescu et al. 2013). Annotation projection assumes that translation preserves meaning (Abzianidze et al. 2017). Frame inventories needed adaptation even between two closely related languages (Burchardt et al. 2006). Cross-lingual parsing helps (Damonte & Cohen 2018). Still, there is no agreed account of which distinctions a universal meaning representation should keep, such as tense, aspect, quantifier scope or modality.

Discrete or graded senses?

Graded human judgements (Erk et al. 2013) and context-sensitive benchmarks (Pilehvar & Camacho-Collados 2019) suggest that word meaning is not a choice among fixed dictionary entries. Yet much of the field's machinery, from WordNet to role inventories, presupposes discrete categories. Reconciling vector-based and category-based descriptions of the lexicon is unfinished work for knowledge representation and lexicography alike.

Where does semantics end and pragmatics begin?

Accounts of numerals (Kennedy 2015) and of epistemic modals (Moss 2015) show that the division of labour between literal meaning and contextual reasoning is still being negotiated, phenomenon by phenomenon. Computational systems face the same problem in practical form. A meaning representation that records only literal content will miss what speakers convey. One that records everything conveyed will no longer be a representation of the sentence alone.

The library in figures