Elamur Research
☰
home / papers / 2017 / Ordinal Common-sense Inference

Ordinal Common-sense Inference

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/Q17-1027.pdf

Summary

Classic textual entailment asks whether a hypothesis strictly follows from a premise. Much of everyday comprehension, however, depends on inferences that are only probable, and these are rarely written down. Because of this reporting bias, corpus statistics poorly reflect real-world likelihood. Zhang et al. reframe the problem as ordinal common-sense inference: given a context sentence, a system must rate how likely a statement is on a five-level scale, from very likely down to impossible.

To build data, the authors mine and abstract common-sense knowledge from text and use it to automatically generate candidate hypotheses (AGCI). They also train a seq2seq model that can score and generate inferences. Crowd workers then rate the resulting pairs, and the same protocol is applied to portions of existing resources such as SNLI. Together, these annotations make up a new dataset called JOCI, short for JHU Ordinal Common-sense Inference.

A feature-based ordinal regression baseline outperforms trivial and SVM baselines, but its errors remain large. Auto-generated hypotheses turn out to be harder than human-elicited ones. The work matters because it separates plausibility from strict entailment and provides a graded benchmark for reasoning with world knowledge.

Key points

  • Rating scale: a 5-point ordinal scale (very likely, likely, plausible, technically possible, impossible). It replaces binary or three-way entailment labels and captures cases like a coin flip landing heads, which is plausible but not entailed.
  • Release: the JOCI corpus is freely available at decomp.net.
  • Evaluation subsets:
    • Subset "A" contains 2,976 SNLI-train pairs with human-elicited hypotheses, relabeled ordinally.
    • Subset "B" contains 6,375 pairs that reuse the same SNLI premises but pair them with auto-generated hypotheses.
  • Baseline results: the ordinal regression model reaches an MSE of 1.96 on A-test and 2.74 on B-test, with Spearman's ρ of .40 and .27. The authors hope future models will push MSE below 1.0.
  • Features and ablation: the model uses bag-of-words overlap, GoogleNews word2vec similarity, seq2seq log-probabilities (five SNLI-trained variants), and length cues. Ablation shows that similarity and BOW features matter most on A, whereas seq2seq features are about equally important on B. This points to statistical differences between elicited and generated hypotheses.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages · Natural language processing

Cited by

83 papers of our library cite it (of the 140673 whose bibliographies we read); 12 of them have a page here, 71 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.1162/tacl_a_00068

@article{zhang2017ordinal,
  title = {{Ordinal Common-sense Inference}},
  author = {Zhang, Sheng and Rudinger, Rachel and Duh, Kevin and Durme, Benjamin Van},
  journal = {Transactions of the Association for Computational Linguistics},
  year = {2017},
  doi = {10.1162/tacl\_a\_00068},
  url = {https://doi.org/10.1162/tacl_a_00068}
}

References

10 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. component in filling in the gaps between the telegraphic style of natural language statements. We are able to convey considerable information in a relatively sparse channel, presumably owing to a partially shared model at the start of any discourse. 2 Common-sense inference – inferences based on common-sense knowledge – is possibilistic: things everyone more or less would expect to hold in a given context, but without the necessary strength of logical entailment. 3 Because natural language corpora exhibits human reporting bias (Gordon and Van Durme, 2013), systems that derive knowledge exclusively from such corpora may be more accurately considered models of language, rather than of the find
  2. Researchers in Artificial Intelligence and (Computational) Linguistics have long-cited the requirement of common-sense knowledge in language understanding. 1 This knowledge is viewed as a key Schank (1975): It has been apparent ... within ... natural language understanding ... that the eventual limit to our solution ... would be our ability to characterize world knowledge. find
  3. McCarthy (1959): a program has common sense if it automatically deduces for itself a sufficiently wide class of immediate consequences of anything it is told and what it already knows. 3 Many of the bridging inferences of Clark (1975) make use of common-sense knowledge, such as the following example of “Probable part”: I walked into the room. The windows looked out to the bay. To resolve the definite reference the windows, one needs to know that rooms have windows is probable. find
  4. Transactions of the Association for Computational Linguistics, vol. 5, pp. 379–395, 2017. Action Editor: Mark Steedman. Submission batch: 12/2016; Revision batch: 3/2017; Published 11/2017. c 2017 Association for Computational Linguistics. Distributed under a CC-BY 4.0 license. world (Rudinger et al., 2015). Facts such as “A person walking into a room is very likely to be blinking and breathing” are usually unstated in text, so their real-world likelihoods do not align to language model probabilities. 4 We would like to have systems capable of reading a sentence that describes a real-world situation and inferring how likely other statements about that situation are to hold true in the real world, e.g. This capability is subtly but crucially distinct from the ability to predict other sentences reported in the same text, as a language model may be trained to do. find
  5. We therefore propose a model of knowledge acquisition based on first deriving possibilistic statements from text. As the relative frequency of these statements suffers the mentioned reporting bias, we then follow up with human annotation of derived examples. Since we initially are uncertain about the real-world likelihood of the derived common-sense knowledge holding in any particular context, we pair it with various grounded context and present to humans for their own assessment. As these examples vary in assessed plausibility, we propose the task of ordinal common-sense inference, which embraces a wider set of natural conclusions arising from language comprehension (see Fig 1). In what follows, we describe prior efforts in common-sense and textual inference (§2). We then state our position on how ordinal common-sense inference should be defined (§3), and detail our own framework for large-scale extraction and abstraction, along with a crowdsourcing protocol for assessment (§4). This includes a novel neural model for forward generation of textual inference statements. Together these methods are applied to contexts derived from various prior textual inference resources, resulting in the JHU Ordinal Common-sense Inference (JOCI) corpus, a large collection of diverse common-sense inference examples, judged to hold with varying levels of subjective likelihood (§5). We provide baseline results (§6) for prediction on the JOCI corpus. 5 For further background see discussions by Van Durme (2010), Gordon and Van Durme (2013), Rudinger et al. (2015) and Misra et al. (2016). 5 The JOCI corpus is released freely at: http://decomp.net/. ↗ our card
  6. Common-sense Tasks Many textual inference tasks have been designed to require some degree of common-sense knowledge, e.g., the Winograd Schema Challenge discussed by Levesque et al. (2011). The data for these tasks are either smaller, carefully constructed evaluation sets by professionals, following efforts like the F RA C A S test suite (Cooper et al., 1996), or they rely on crowdsourced elicitation (Bowman et al., 2015). Crowdsourcing is scalable, but elicitation protocols can lead to biased responses unlikely to contain a wide range of possible common-sense inferences. Humans can generally agree on the plausibility of a wide range of possible inference pairs, but they are not likely to generate them from an initial prompt. 6 The construction of SICK (Sentences Involving Compositional Knowledge) made use of existing paraphrastic sentence pairs (descriptions by differ-McRae et al. (2005): Features such as <is larger than a tulip> or <moves faster than an infant>, for example; although logically possible, do not occur in [human responses] [...] Although people are capable of verifying that a <dog is larger than a pencil>. ent people of the same image), which were modified through a series of rule-based transformations then judged by humans (Marelli et al., 2014). As with SICK, we rely on humans only for judging provided examples, rather than elicitation of text. Unlike SICK, our generation is based on a process targeted specifically at common sense (see §4.1.1). find
  7. Plausibility Researchers in psycholinguistics have explored a notion of plausibility in human sentence processing, where, for instance, arguments to predicates are intuitively more or less “plausible” as fillers to different thematic roles, as reflected in human reading times. For example, McRae et al. (1998) looked at manipulations such as: (a) The boss hired by the corporation was perfect for the job. (b) The applicant hired by the corporation was perfect for the job. where the plausibility of a boss being the agent – as compared to patient – of the predicate hired might be measured by looking at delays in reading time in the words following the predicate. This measurement is then contrasted with the timing observed in the same positions in (b). 7 Rather than measuring according to predictions such as human reading times, here we ask annotators explicitly to judge plausibility on a 5-point ordinal scale (See §3). Further, our effort might be described in this setting as conditional plausibility, 8 where plausibility judgments for a given sentence are expected to be dependent on preceding context. Further exploration of conditional plausibility is an interesting avenue of potential future work, perhaps through the measurement of human reading times when using prompts derived from our ordinal common-sense inference examples. Computational modeling of (unconditional) semantic plausibility has been explored by those such as Padó et al. (2009), Erk et al. (2010) and Sayeed et al. (2015). find our card
  8. Textual Entailment A multi-year source of textual inference examples were generated under the Recognizing Textual Entailment (RTE) Challenges, introduced by Dagan et al. (2006): This notion of thematic plausibility is then related to the notion of verb-argument selectional preference (Zernik, 1992; Resnik, 1993; Clark and Weir, 1999), and sortal (in)correctness (Thomason, 1972). 8 Thanks to the anonymous reviewer for this connection. find
  9. We say that T entails H if, typically, a human reading T would infer that H is most likely true. This somewhat informal definition is based on (and assumes) common human understanding of language as well as common background knowledge. This definition strayed from the more strict notion of entailment as used by linguistic semanticists, such as those involved with F RA C A S. While Giampiccolo et al. (2008) extended binary RTE with an “unknown” category, the entailment community has primarily focused on issues such as “paraphrase” and “monotonicity”. An example of this is the Natural Logic implementation of MacCartney and Manning (2007). Language understanding in context is not only understanding the entailments of a sentence, but also the plausible inferences of the sentence, i.e. the new posterior on the world after reading the sentence. A new sentence in a discourse is almost never entailed by another sentence in the discourse, because such a sentence would add no new information. In order to successfully process a discourse, there needs to be some understanding of what new information can be, possibly or plausibly, added to the discourse. Collecting sentence pairs with ordinal entailment connections is potentially useful for improving and testing these language understanding capabilities that would be needed by algorithms for applications like storytelling. find
  10. Garrette et al. (2011) and Beltagy et al. (2017) treated textual entailment as probabilistic logical inference in Markov Logic Networks (Richardson and Domingos, 2006). However, the notion of probability in their entailment task has a subtle distinction from our problem of common-sense inference. The probability of being an entailment given by a probabilistic model trained for a binary classification (being an entailment or not) is not necessarily the same as the likelihood of an inference being true. For example: find