📄 LMN at SemEval-2022 Task 11: A Transformer-based System for English Named Entity Recognition

Summary

The paper tackles recognition of complex named entities in English: names that are not simple nouns but titles, gerunds, infinitives or whole clauses (e.g. I Capture The Castle), which context alone often fails to disambiguate. The author's system for the English track of SemEval-2022 Task 11 (MultiCoNER) treats the problem as BIO sequence labeling over six entity types: PER, LOC, GRP, CORP, PROD and CW. A pretrained Transformer encoder produces a vector for each token, and a feedforward layer with softmax predicts that token's tag. The whole model is fine-tuned with cross-entropy, and no CRF layer is used. BERT-large, RoBERTa-large and ALBERT-xxlarge were compared on the development set. ALBERT-xxlarge trained for 20 epochs came out best and was used for the final submission, which reached 72.50% macro F1 on the test set and placed 12th of 30 teams. The author also tried enlarging the training data by linking entity mentions to Wikidata and swapping in other entities of the same type. This did not raise scores, but it is documented as a reusable recipe. The work shows that a plain fine-tuned encoder is a solid baseline for hard NER.

Key points

• Task: English track of SemEval-2022 Task 11 (MultiCoNER), with labels {B,I}-PER, LOC, GRP, CORP, PROD, CW plus O in BIO format.
• Model: a Transformer encoder with a token-wise FFNN and softmax, and no CRF. The author cites Chen et al. (2019), who found that a CRF adds little on top of pretrained encoders.
• Setup: Huggingface Transformers, learning rate 2e-5, batch size 32, and 10 or 20 epochs. The encoders were bert-large-uncased, roberta-large and albert-xxlarge-v2.
• Development F1 at 10 epochs was 86.68 for RoBERTa-large and 87.14 for BERT-large. ALBERT-xxlarge at 20 epochs was the best development model.
• Test result: 72.50% macro F1, ranked 12th of 30. Entity-linking augmentation via Wikidata, using the EL-Dockers linker, brought no gain.

Interesting ideas

• I1. Type-preserving entity swapping: a mention is linked to Wikidata and replaced by a different entity of the same type, which yields new sentences whose structure is unchanged.
• I2. Skipping the CRF: with a strong pretrained encoder, a per-token softmax was judged sufficient, which keeps the model simple.
• I3. Names with internal syntax: many entities are gerunds, infinitives or full clauses, so their inner words look like ordinary syntax and mislead context-based recognizers.
• I4. Publishing a non-improving augmentation: the author reports the augmentation experiment even though it failed, which documents what was tried and why.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (learn): use such swaps to make silver variants whose UD trees and frames should stay identical apart from the name, and flag any witness parser whose output changes as a cheap consistency check.
• I2 (tree): before adding structured decoding layers to gparse's labeling heads, measure whether they actually help on top of the transformer features.
• I3 (tree): detect title-like spans first and attach them as a single flat name unit, so a clause such as I Capture The Castle is not read as the sentence's main predicate.
• I4 (learn): log every augmentation or repair rule together with its measured effect against a negative control, so that ineffective tricks are not silently retried.