📄 Exploring Chunk Based Templates for Generating a subset of English Text

Summary

The paper addresses tactical generation: producing a grammatical English sentence in isolation that satisfies given constraints, such as a partial bag-of-words, fixed token positions or required ordering. Rather than handcrafting templates, the authors mine them automatically from English Wikipedia. Each chunk becomes a template after abstraction. Named entities are replaced by category symbols, some POS classes are collapsed to their tags, and the remaining content words are mapped to KMeans clusters of token–POS embeddings. Stopwords and punctuation stay literal. Because every template is copied from real text, it is grammatical internally, so the open problem is only how to join templates.

Generation is treated as search. An evolutionary algorithm assembles template sequences under per-slot constraints. Its fitness function combines n-gram log-probabilities over chunk tags, chunk layouts, junctions, heads and lexical clusters. Finally, the cluster slots are filled with the requested words.

The evaluation consists only of example outputs, which are often syntactically plausible but semantically odd. The work matters as a corpus-driven, inspectable alternative to hand-written templates.

Key points

• Templates were extracted from English Wikipedia using Stanford CoreNLP, spaCy and LM-LSTM-CRF for tagging, NER and chunking. Clustering used lexvec vectors with k = 7500 token–POS clusters.
• The fitness score is a weighted sum of five length-normalized n-gram scores (chunk, construction, junction, head, lexical) with empirically chosen weights of 75, 10, 10, 5 and 2.
• The evolutionary search works as follows:
  • population of 1000 organisms, run for 100 generations;
  • elitist selection, with the first half of the population mutated into the second half;
  • 25% of the population re-initialized each generation;
  • at least 8% of organisms must have unique blank layouts and 15% unique extended POS layouts.
• Template selection uses Zipf dampening with exponent 0.4, because otherwise NP chunks dominate (about 40% of picks).
• Generation took about 150 seconds and 2.2 GB of memory. The authors note biases toward sentences opening with "the" NPs and chains of VPs/PPs. Questions only appear when the wh-word position is constrained.

Interesting ideas

• I1. Build sentences by joining corpus fragments that are already well-formed internally, so the scorer only has to judge the seams between them rather than every word.
• I2. Score a candidate with several separate n-gram models over different abstraction layers (chunk tags, chunk layouts, junction tokens, heads, lexical clusters) and combine them with explicit weights.
• I3. Flatten a skewed Zipf distribution with a dampening exponent, so rare categories still get sampled while frequency preferences survive.
• I4. Describe a chunk by its layout, with content words blanked or replaced by POS while function words and punctuation are kept, giving a compact construction signature.
• I5. Cluster word embeddings per token–POS pair instead of per word form, so that each syntactic use of an ambiguous word gets its own class.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (back): Generation from the meaning graph could stitch verified phrase-level realizations taken from parsed text, and focus checking effort on boundaries between them.
• I2 (back): Each sub-score explains which aspect of a generated sentence is weak, which fits Mova's requirement that every step be explainable.
• I3 (learn): When sampling sentences for silver data or self-play, damping keeps rare constructions from being drowned out by common NP-heavy patterns.
• I4 (learn): Such signatures could serve as keys for grouping parser errors and indexing self-induced repair rules by construction type.
• I5 (tokens): POS-split clusters could provide backoff classes for rare lemmas in the tagger and morphology, without merging noun and verb behavior.