📄 Scoring Both Directions: LLMs realize the MRS they cannot reliably parse

Summary

The paper asks whether large language models that write good English from a formal meaning graph can also build that graph from English. It uses the English Resource Grammar (ERG) and its processor ACE, which map sentences to Minimal Recursion Semantics (MRS) and back. After reconstructing the 10K-item Redwoods test split from Hajdik et al. (2019), the author tests two models, Claude Opus 5 and Claude Sonnet 4.5, on both directions of the task. Each model gets three fixed examples and no fine-tuning, and ACE serves as the reference system. For generation (MRS to text), Opus beats a seq2seq system trained on 72k pairs and roughly ties one trained on about a million silver pairs. Parsing the generated sentences back with ACE shows they mostly keep the input meaning. For parsing (text to MRS), both models fall well short of ACE and almost never reproduce the gold graph. They tend to pick the right words but miss the grammar's predicate names and argument attachments. The main conclusion is that strong generation scores are not evidence that a model has mastered a formal semantic representation.

Key points

• Benchmark: ERG-1214 gold profiles converted with Hajdik et al.'s released script, giving 10,201 test items from 37 Redwoods profiles. The tools are ACE 0.9.31 and ERG 1214, scored with SacreBLEU, EDM F1, exact match and Smatch on EDS.

• Realization (BLEU):

  • Opus 5 scores 76.34 and Sonnet 4.5 scores 65.65.
  • The trained systems score 66.11 (72k gold pairs) and 77.17 (gold plus about 1M silver pairs).
  • ACE realizes only 79.2% of items, at 61.81.

• Grammar as candidate source:

  • With Sonnet choosing among ACE's 10-best (and its own output where ACE fails), the score reaches 69.64.
  • A pooled judge that adds Opus's sentence to the candidates reaches 76.95.
  • The oracle pick over ACE's candidates plus Opus's sentence reaches 83.07.

• Parsing (998 sentences, spans-off EDM F1 and exact match):

  System	EDM F1	Exact match
  ACE	91.0	50.5%
  Opus	65.5	1.4%
  Sonnet	57.2	0.6%

  Exact predicate names are recovered for about 93% (ACE), 74% (Opus) and 65.5% (Sonnet) of gold predicates.

• Controls:

  • Swapping a content noun in 140 MRS/reference pairs lowers BLEU by 3.17. This is a modest memorization effect.
  • A gold graph with every predicate name replaced by nonsense still earns 79.2 Smatch.

Interesting ideas

• I1. Round-trip faithfulness check: Parse each generated sentence back and compare the result with the input graph, so that a high overlap score is confirmed to reflect kept meaning, not surface luck.
• I2. Calibration rows for metrics: Score deliberately broken outputs, such as gold with junked names or one constant graph, to show how much a metric rewards bare structure.
• I3. Generator plus LLM judge: A rule system proposes an n-best list and a language model picks the most natural candidate. The model's own output is used when the grammar abstains or is added to the pool.
• I4. Splitting errors into vocabulary and attachment: Separate lemma-level matches, exact predicate-name matches and edge recovery between correctly named nodes, showing whether a model knows the words but not the conventions.
• I5. Content-substitution control for memorization: Replace one content noun consistently in both input and reference, then see how much the score drops, to estimate how much success relies on recalled public text.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (back): The text → graph → text compressor could re-parse every output and report graph agreement next to fluency, exposing generations that read well but drop roles.
• I2 (learn): Evaluation with negative controls could include these degenerate baselines for every tree and frame metric, so a seemingly decent score can be compared against its trivial floor.
• I3 (back): Mova's generator could emit several explainable realizations and let a judge choose among them, keeping every candidate traceable to the meaning graph.
• I4 (learn): When LLM teacher tiers produce silver frames, Mova could measure name inventory and attachment separately, and map LLM labels onto the VerbNet/WordNet inventory before trusting them.
• I5 (answer): QA on well-known fables and tales could be rerun with swapped characters or objects, checking that readers reason over the text's world rather than remembered stories.