📄 Recognition of Classical Arabic Poems

Summary

The paper addresses finding classical Arabic poetry inside arbitrary text, such as web pages, forums and blogs. In these sources poems have no dedicated markup, follow inconsistent layouts and are mixed with prose. The authors build a rule-based detector around traits of the classical form:

• verses split into two hemistiches of roughly equal length;
• a single end-rhyme shared by every verse, with simple letter and vowel equivalences;
• three common layouts on the page;
• poets rarely reuse the same word at the start or end of verses.

The detector scans lines and groups consecutive ones of similar length. It splits lines that carry a mid-line separator and looks for runs that share a rhyme. It also repairs common typing defects and applies extra pruning to short candidates. The test set was built from 100 web pages, half containing poems and half not. On it the method reached 96.94% precision and 92.24% recall, measured on output lines. The detector was also plugged into a crawler to build a poem-only search engine. The work shows that a small set of transparent form constraints can reliably pull structured literary text out of noisy sources.

Key points

• Evaluation data: 50 positive pages (5 poets, the top 10 Google hits each) and 50 negative pages, converted to plain text with Apache Tika. The result is about 23K non-empty lines holding 161 poem instances and 4,740 half-verses.
• Results: 96.94% precision and 92.24% recall. Both are computed over half-verse lines, comparing the system output against a gold model.
• Word-reuse study (134 poems):
  • 22% of poems contain a repeated start word and 31% a repeated end word.
  • Per verse, the chance of a repeat is only about 3% for start words and 4% for end words.
• Pipeline and limits:
  • Lines are kept only if they have 2–20 words, and line lengths are compared with a 40% difference tolerance.
  • All layouts are normalized to one canonical shape.
  • Meter (Al-Khalil's 15+1 system) is not used.
  • Single-verse poems cannot be detected.
• Pruning and prototype:
  • Pruning was compared at levels 0, 2, 3, 4 and 4*. In 4*, the duplicate-word test is applied to every candidate rather than only to short ones.
  • The prototype search engine extends the HTML parser of Apache Nutch and indexes with Solr.

Interesting ideas

• I1. Repetition as the trigger: Detection fires only when a pattern recurs across consecutive units (shared rhyme, similar length), so evidence piles up with each extra verse and isolated coincidences are ignored.
• I2. Group consistency over local noise: When verses share a rhyme but their separators differ or are missing in some lines, the group's agreement decides whether those marks are separators or just punctuation.
• I3. Stricter tests only where evidence is thin: Extra checks are applied to short candidates (two to four verses), where chance matches are likelier, while long candidates pass on accumulated evidence.
• I4. Longest consistent solution wins: When several overlapping rhyme-based segmentations are possible, the one yielding the longest poems is chosen. This is a simple, explainable tie-break rule.
• I5. Negative pages in the test set: Half the evaluation pages contain no poems at all, so false positives on ordinary text are measured directly, and scoring is done per output line.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (tokens): Before tokenization, recognize lists, tables or verse blocks by repeated line-level patterns, so they are not parsed as ordinary run-on sentences.
• I2 (learn): Self-induced repair rules could treat an attachment or label that conflicts with a strongly consistent group of parallel structures as an error to fix.
• I3 (learn): Silver data could go through additional filters only for short sentences or items with weak witness-parser agreement, rather than filtering everything equally.
• I4 (tokens): Ambiguous multiword-expression or segmentation choices could use a similar maximal-coverage preference that is recorded as an explicit reason.
• I5 (learn): Detectors such as irony or speech-act recognizers could be evaluated with pools of texts known to lack the phenomenon, as negative controls.