📄 Cross-domain Generalization for AMR Parsing

Summary

AMR parsers score above 84% SMATCH on their home test set, but performance outside that domain had barely been studied. The authors run a zero-shot study. Five parsers representing different design families are trained on the standard source data and tested on several other domains: biomedical text, a novel, and wiki-style questions. The parsers are JAMR (two-stage), AMRGS (graph-based), StructBART (transition-based), SPRING (seq2seq) and AMRBART (AMR-specific pre-training).

Every parser loses accuracy, in the worst case up to 45.5%. Named entities and wiki links suffer most. AMRBART transfers best, and the transition-based parser holds up better than the seq2seq one.

The authors then relate accuracy loss to measured distribution shift. Shift in output-side features, especially AMR concepts, predicts the damage better than shift in the input words.

They test two fixes that leave the model architecture unchanged:

• continued BART pre-training on raw text from the target domain;
• training on a mix of gold data and silver target-domain graphs produced by an existing parser.

Both help, and silver data helps more. This shows that closing the gap on the output side pays off most.

Key points

• Five parsers are compared: JAMR, AMRGS, StructBART, SPRING and AMRBART. The evaluation is zero-shot, with no labelled target-domain data. Out-of-domain test sets include Bio, TLP (a novel) and QALD-9 (wiki questions).
• Biomedical SMATCH falls below 65%, against more than 84% in-domain. The worst relative drop reaches 45.5%, and named entities and wiki links degrade most.
• Domain shift is measured with Jensen-Shannon divergence and OOV rate. These measures are correlated with accuracy loss over 100 resampled test sets of 2,000 examples each, drawn from a pool of 3,147.
• Unigram shift matters more than bigram or trigram shift, and concept shift matters more than relation shift. Overall, output AMR features matter more than input text features.
• Domain-adaptive pre-training (span masking, 15% of tokens) gives modest gains of +0.8 to +1.3 SMATCH on Bio. Silver target data gives +2.4 (StructBART) and +3.3 (SPRING) on Bio, and about +1.1 to +1.2 on TLP.

Interesting ideas

• I1. The authors explain the accuracy drop by correlating measured feature divergence with each parser's error increase, using many resampled test sets as data points rather than one score per domain.
• I2. They separate input-side shift (words) from output-side shift (predicted concepts and structure) and find the output side to be the bigger obstacle.
• I3. Parsing raw target-domain text with an existing parser and mixing those silver graphs with gold data improves structural accuracy most, at the cost of some noise.
• I4. Continuing self-supervised pre-training on raw in-domain text before fine-tuning gives a cheap gain without architecture changes.
• I5. Fine-grained metrics revealed that the two fixes help different things. Silver data improves concepts, reentrancy and SRL, while pre-training on raw text improves NER and negation.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (learn): We could resample our evaluation sets in the same way and correlate per-feature divergence with error. This would show which shifted feature (lemmas, relations, frame types) actually drives failures on a new text genre.
• I2 (tree): Before adopting a treebank for a new genre, we could compare its distribution of deprels and head-dependent patterns with gparse output on target text, not just vocabulary overlap.
• I3 (learn): This supports adding witness-agreed silver trees from fables and school texts to gold UD training. The mix ratio should be monitored, since noise hurt the text-level metrics.
• I4 (tree): gparse's transformer encoder could be further pre-trained on unlabeled tales and textbook prose before treebank fine-tuning, to narrow the vocabulary gap.
• I5 (frames): Frame evaluation could likewise be split into roles, negation, coreference and entities, so that each data or training change is credited to the component it actually improves.