📄 Exploring the Data-Driven Prediction of Prepositions in English

Summary

The paper asks how far one can get in choosing the right English preposition using only the raw words around it, with no POS tags, parses or WordNet classes. Earlier systems relied on rich feature sets. The authors instead treat the web as a huge training corpus. They query the Yahoo search engine with n-grams in which the masked slot is filled by each candidate preposition, and they pick the variant with the most hits. The model starts from a 7-gram window. When no evidence is found, it backs off to pairs of overlapping shorter n-grams.

The test material is section J of the BNC with the nine most frequent prepositions, the same setup as De Felice (2008). Test sentences already present on the web were removed so that the search index does not act as leaked training data. The approach reaches about 77% accuracy, which beats De Felice's 70.06% and is close to other state-of-the-art feature-based systems. The result shows that local surface context carries much of the needed signal when data is plentiful.

Key points

• Evaluation items are drawn from BNC section J, where prepositions are identified by the CLAWS-5 tags PRF/PRP, and are restricted to nine prepositions (alphabetically: at, by, for, from, in, of, on, to, with).
• The full pool of 522,313 tasks was sampled down to 10,000. Queries over Yahoo's 4000-character limit were dropped, as were sentences already found verbatim online. This left 8060 tasks.
• The first back-off step approximates a 7-gram by two overlapping 6-grams. It adds 1620 correct and 408 wrong predictions, giving 3551 correct and 672 incorrect, with recall 56.1% and precision 84.1%.
• Across back-off steps, recall climbs steeply while precision declines only slightly. The final accuracy is about 77%.
• For comparison, the reported results of other systems are:

Their test sets and preposition inventories differ, so these figures are only loosely comparable.

Interesting ideas

• I1. Leakage check. Before evaluating, the authors searched the "training corpus" for each whole test sentence and discarded any that already appeared there, which is a cheap guard against leakage whenever the training source is huge and opaque.
• I2. Back-off with tracked precision and recall. The model degrades from a long exact context to overlapping shorter windows, and the authors record precision and recall at each step to show what each fallback level costs.
• I3. One masked task per slot. Each preposition in a sentence becomes its own fill-in-the-blank instance, turning raw text into many self-supervised test items for free.
• I4. Separate functional and lexical prepositions. The authors propose reporting the two classes apart, because functional ones (dependent on) are fixed by nearby words, while lexical ones (put the bag on/under) need meaning that is not local.
• I5. Stabilization curve for sample size. Plotting accuracy against the number of predictions showed it settles after about 1000 items, which justifies a smaller test sample.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (learn): Before scoring a parser or reader, check every evaluation sentence against the silver data and the LLM-teacher outputs, and drop any that leaked in.
• I2 (tree): Repair rules and dictionary lookups could fire from most to least specific context, with each decision tagged by the level that produced it, so explanations show how confident the match was.
• I3 (answer): Mask function words or arguments in tale texts to build cloze-style checks of whether the reader's world model picks the right relation.
• I4 (frames): Score preposition and role decisions separately for valency-selected cases and spatial or semantic cases, and route the latter to world-level reasoning rather than local patterns.
• I5 (learn): Use running-accuracy curves to choose how many items each expensive evaluation (e.g., teacher-graded) really needs.