📄 IdentifyMe: A Challenging Long-Context Mention Resolution Benchmark for LLMs

Summary

Traditional coreference evaluation asks models to reproduce span clusters. This penalizes LLMs whenever their mention boundaries or entity definitions differ from a dataset's own conventions, so their real grasp of reference gets blurred. The authors reframe the task as mention resolution in multiple-choice form. A long literary document contains one marked mention, and the model must pick the entity it denotes from a list of frequent entities plus a "None of the Above" (NoA) option.

Questions come from LitBank and FantasyCoref annotations. Each entity cluster gets a representative name, drafted by GPT-4o-mini and then reviewed by hand. Only nominal and pronominal mentions are kept. Heuristics first remove cases with obvious surface or pronoun-agreement cues, then rank the rest by difficulty.

The resulting 1800 questions are genuinely hard:

• GPT-4o leads at 81.9%.
• Sub-10B open models hover around 50%.
• Pronouns are harder than nominals.
• NoA items and nested mentions produce characteristic failures.

The benchmark offers a format-neutral, finely diagnosable probe of referential understanding.

Key points

• Size and splits: IdentifyMe has 1800 questions from 159 documents (64 LitBank, 95 FantasyCoref), with 600 for validation and 1200 for test. Documents average about 1700–2000 words, and those under 1000 words were dropped.
• Filtering:
  • Nominal mentions with fuzzy similarity above 75% to their entity's name are discarded.
  • Pronouns with a non-positive Net-Distractor-Score (based on LingMess gender/number/animacy categories) are discarded.
  • The remaining criteria are merged via Copeland's method.
• NoA items: in 10% of questions the gold entity is removed, making NoA correct. Open models almost never get these right (Mistral-7B 1.7%, Llama-3.1-8B 0.8%), and Gemini-1.5-Flash reaches only 30.3%.
• Main results:
  • Test accuracy is 81.9% for GPT-4o and 53.3% for Llama-3.1-8B, against a random baseline of about 8%.
  • Chain-of-thought helped on validation (+9.5 for Llama-3.1-8B, +3.7 for GPT-4o-mini).
  • On a 50-question subset, humans scored 92–94% versus GPT-4o's 70%.
• Filtering check and nested mentions: questions on randomly sampled mentions were easier by 9.5 points (Mistral-7B) and 7.2 points (GPT-4o-mini). Of 352 nested-mention cases, roughly 40% of model errors picked the entity of an overlapping inner mention.

Interesting ideas

• I1. Pronoun difficulty can be scored by subtracting supportive nearby pronouns (same category, same entity) from confusing ones (same category but another entity, or same entity but a different category); this gives a cheap, explainable ambiguity measure.
• I2. Grading reference by which entity was chosen, rather than by exact antecedent spans, removes noise from annotation conventions; a NoA option additionally tests whether the system knows when no candidate fits.
• I3. Several weak difficulty signals (surface similarity, distance to nearest mention, distance to nearest nominal mention) can be fused into one ranking with pairwise Copeland voting, with no need to tune weights.
• I4. Nested phrases like a possessive inside a noun phrase cause models to resolve the outer mention to the inner possessor, so this is worth targeting as its own error class.
• I5. Checking a "hard" benchmark against an equal-sized random sample proves the filtering really adds difficulty instead of just asserting it.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (frames): the coreference resolver could compute this score per pronoun and flag high-score cases for deeper world-based reasoning or for an explicit "uncertain" explanation.
• I2 (learn): coreference evaluation could ask "which world entity is this?" and include gold-removed items as negative controls, checking that Mova abstains instead of guessing.
• I3 (learn): the same voting could rank silver examples by difficulty, so the hardest disagreements are sent to higher LLM teacher tiers.
• I4 (frames): since the UD tree already separates the head noun from its nmod:poss dependent, a dedicated test set of such phrases could verify that the resolver links the head's entity, not the possessor's.
• I5 (answer): when building challenge sets for fable/tale QA, always report scores on a random-sample control set alongside them, so that claims about difficulty are backed by evidence.