📄 Parsing Indian English News Headlines

Summary

Newspaper headlines bend ordinary grammar. Editors omit subjects, auxiliaries and sometimes main verbs, use odd capitalization and punctuation, and rely on cultural context, so parsers trained on standard prose mislabel them. This ICON 2020 student paper outlines doctoral work on parsing headlines from Indian English newspapers.

The authors collected roughly 40,000 headlines (about 300,000 words) from three papers. They ran the Stanford and AllenNLP constituency parsers over them and catalogued the tagging and bracketing mistakes. A linguistic study then sorted headlines into fragments and complete clauses. This study produced transformation guidelines, a syntactico-semantic feature model, and a parallel corpus that pairs raw headlines with rewritten full sentences.

A small check showed that the same parsers give correct analyses once the missing grammatical material is restored. The planned system therefore rewrites headlines into canonical sentences first and parses them afterwards. It combines hand-written grammar rules with a translation-style statistical model. The paper reports no quantitative scores; its value lies in the resource and the error diagnosis.

Key points

• Corpus: about 40,000 Indian English news headlines (around 3 lakh, i.e. ~300,000 words) drawn from three newspapers; a contrastive comparison with British English headlines was also carried out.
• Error analysis with the Stanford and AllenNLP constituency parsers showed typical mistakes, such as verbs tagged as plural nouns, nouns as adjectives and common nouns as proper nouns. One example is "Boat capsize toll touches 21", where touches was read as NNS.
• Typology: headlines are either fragments (mostly noun phrases) or complete clauses. Complete clauses are further split into declarative, imperative, interrogative and exclamatory types.
• Resources built: transformation guidelines, a syntactico-semantic feature model, and a parallel corpus of raw headlines and grammatical rewrites. A headline grammar and the parsing module are still in progress.
• Planned automation is hybrid. The rule-based side uses context-free grammar rules, with LALR parsers considered. The statistical side treats normalization as machine translation trained on the parallel corpus.

Interesting ideas

• I1. Normalize, then parse: rather than retraining a parser on telegraphic text, restore the elided pieces (subject, auxiliary, verb) and hand the result to an ordinary parser.
• I2. Conservative rewriting rule: when expanding a fragment, add only material that is certain from the text and never drop anything meaningful. This keeps normalization from inventing content.
• I3. Hypothesis test by repair: confirm the cause of parser errors by feeding repaired versions of the failing inputs and seeing whether the output becomes correct.
• I4. Headline normalization framed as translation over a parallel corpus of raw and canonical forms. Seen this way, the mapping is learnable in both directions.
• I5. A structural typology (fragment vs. complete clause, with sentence types) used as a feature model to drive annotation and rules. It lets each headline type get its own handling.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (tree): A headline/caption preprocessing step could reinsert dropped auxiliaries or subjects as explicitly marked inserted nodes before parsing. The resulting UD tree would still show exactly what was added and why.
• I2 (learn): Apply this as a hard check when an LLM teacher rewrites fragments into silver training sentences, rejecting rewrites that add new entities or lose content words.
• I3 (learn): Pair failing fragments with repaired forms as a diagnostic control. Success on the repair attributes the error to ellipsis rather than to a weakness of the parser model.
• I4 (back): The reverse direction (full sentence → headline style) could serve as a compact surface form for generation or for the text → graph → text compressor.
• I5 (tokens): A cheap fragment/clause classifier could switch the tagger into a headline mode. That mode would revisit noun/verb readings of -s words like touches when no finite verb is otherwise found.