📄 From Speculation Detection to Trustworthy Relational Tuples in Information Extraction

Summary

The paper argues that relations extracted by Open Information Extraction (OIE) should not all be treated as facts, because many are hedged or only intended. Classic speculation detection marks cues and scopes for a whole sentence, but a single sentence can produce several tuples and only some of them are uncertain. The authors therefore define a new task: deciding, for each OIE tuple, whether it is speculative and of which kind. They found ready-made labels in LSOIE, an OIE corpus derived from QA-SRL Bank 2.0. In that corpus, modal auxiliaries taken from the crowdsourced questions survive inside tuple relations. They analyse these labels and sort them into easy, medium and hard cases. They also propose SpecTup, a baseline that combines two signals. The first is a BERT encoding of the relation word, made relation-aware with an added embedding. The second is a relation-based GCN over that word's immediate dependency neighbours. Results show the task is feasible, but detecting the rare speculative classes remains hard. This work connects factuality analysis with structured knowledge extraction.

Key points

• The labels come from six modals in LSOIE tuple relations: might, can, will, would, should, had. These are read as possibility, ability, intention or certainty, suggestion, and past event. Only 43.3% of these modals actually occur in the source sentence; the remaining 56.7% were inferred by annotators.
• After removing Sci sentences duplicated from Wiki, the data has 50,162 sentences and 101,166 tuples. Of these, 11,031 (10.9%) are speculative, with a higher share in Sci (about 13%) than in Wiki (about 9%).
• SpecTup takes gold LSOIE tuples as input, so its results are not tied to any particular extractor. It only needs the relation word, which makes it fairly robust given that the best LSOIE extraction F1 is about 0.71.
• Class imbalance dominates the results. Non-speculative tuples reach 95.6 F1, while speculative tuples as a whole reach 54.9. Within the speculative types, 'can' scores 61.9 and 'had' scores 49.6.
• Cues such as "plan to" can mean either likelihood or intention, and such label ambiguity is common among hard cases. Using the full dependency tree instead of the local neighbourhood made results worse.

Interesting ideas

• I1. The authors built a benchmark from annotation that was never meant for this task: modal slots in QA-SRL question templates became speculation labels.
• I2. Speculation is attached to each extracted tuple rather than to the whole sentence, so a sentence can contain both a fact and a hedge.
• I3. Syntactic context is limited to the predicate's direct dependents, because the whole tree added noise.
• I4. Judging by the examples, the test cases are graded by how directly the text signals the label. Easy cases contain the modal verbatim next to the verb, medium cases have it separated, and hard cases convey it only through another cue such as "plans to".
• I5. The model's decision depends only on the relation word, not on fully correct arguments, so argument extraction errors do not spread into the speculation label.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (learn): Existing resources such as QA-SRL and PropBank could be scanned for side-channel annotations that yield silver labels for modality or negation. These labels could then be checked against witness parsers.
• I2 (world): Each event or state stored in the text's world could carry a modality tag such as possible, intended or advised. Then "The UN plans to release a report" would not be asserted as an actual release.
• I3 (frames): When building a frame, crate prag could read modality from aux, mark and xcomp-governing heads directly attached to the verb. This keeps the decision local and easy to explain.
• I4 (learn): Evaluations of modality and factuality could be split into these difficulty bands. This would show whether Mova only matches literal modals or truly understands paraphrased hedges.
• I5 (frames): Modality assignment could be tied to the frame's predicate alone. Frames would then still get correct certainty flags even when role filling is partial.