📄 Evidentiality for Text Trustworthiness Detection

Summary

The paper asks whether the way a writer marks the evidence behind a claim can help a machine judge how trustworthy a text is. Such marks include hedges, epistemic modals and verbs of perception or belief. The setting is collaborative question answering (CQA), where only a handful of answers exist per question. Redundancy-based tricks, such as counting how often an answer co-occurs with the question, break down there. User-profile signals are also easy to game. The authors build a taxonomy of English lexical evidentials and use them as features in a machine-learning text classifier that separates best answers from the rest. The classes are adverbs, lexical verbs, auxiliaries and epistemic adjectives, graded on four levels: Absolute, High, Moderate and Low. Evidential-only representations beat a bag-of-words baseline and a content-word baseline, with an F1 gain of up to 14.85%, while using a tiny feature space. Fine-grained category features did not help. The best behaviour came from treating every evidential as a single signal that an answer is unreliable. The authors explain this through Gricean maxims of Quality and Quantity. The work matters because it shows that a linguistically motivated, compact cue set can rival generic lexical models on a credibility task.

Key points

• Taxonomy: four trust levels (Absolute, High, Moderate, Low) crossed with four word classes.
  • Attributive/modal adverbs, e.g. certainly, apparently, probably, perhaps.
  • Lexical verbs, e.g. believe, seem, doubt, wonder.
  • Auxiliary verbs, e.g. must, should, might.
  • Epistemic adjectives, e.g. definite, likely, doubtful.
• Baselines: Baseline1 is bag-of-words and Baseline2 uses content words. They were compared under Boolean and TF weighting on a balanced training/test set, scored on the best-answer class.
• Boolean results: the "Evidential'" feature set reached F1 62.13% (recall 90.12%), against 47.94% for Baseline1 and 47.28% for Baseline2. Its precision was 47.40%, below Baseline2's 59.58%. Features split by evidential category (Evid.cat2, Evid.cat4) scored below the baselines.
• Dimensionality: the feature space shrank from 218,328 dimensions (Baseline1, in one case) to 51 evidential features.
• Single-category setting: collapsing all evidentials into one category (Evid_cat1: P 59.42%, R 61.59%, Acc 59.76%, F1 60.49%) improved all four metrics over the baseline. Read together with the authors' discussion, this supports treating evidential markers as negative evidence of reliability.

Interesting ideas

• I1. A small, theory-driven set of hedge and evidence markers can stand in for a huge lexical vector, keeping the model compact and its decisions readable.
• I2. The authors expected evidence markers to signal reliable answers, but they ended up signalling weaker ones; reading this through Gricean Quantity turns a surprising sign into a pragmatic insight.
• I3. Separating evidence source (observation, hearsay, inference, memory) from degree of certainty yields two independent dimensions instead of one blurry modality score.
• I4. Finer categories that make sense to humans did not help the classifier, while a single merged class did; label granularity should be tested empirically, not assumed.
• I5. Where redundancy is scarce, such as a few answers per question, content-internal cues are more robust than frequency voting or author reputation.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (frames): the prag crate could tag each clause with an evidential level (Absolute/High/Moderate/Low) and its source type (seen, heard, inferred, believed), giving every frame an explainable certainty attribute.
• I2 (frames): implicature rules could treat an unprompted hedge in an answer as a cue that the speaker is unsure, not as a strengthening of the claim.
• I3 (world): facts in the text world could store both who vouches for a state and how firmly, so later reasoning can drop or downweight hearsay and guesses.
• I4 (learn): evaluation runs could compare fine versus collapsed versions of any new feature inventory as a built-in ablation before adopting the fine one.
• I5 (answer): when choosing between candidate answers in fable or school QA, the reader could prefer plainly asserted statements over hedged ones as a tie-breaker.