Elamur Research⌕
☰
home / papers / 1999 / Mining the Web for Bilingual Text

Mining the Web for Bilingual Text

Paper · 1999 · Philip Resnik

Summary

Parallel text is central to statistical machine translation and cross-language retrieval, but existing bilingual corpora are often licensed, costly, narrow in domain, outdated, or missing for many language pairs. This paper develops STRAND, a system that finds translated document pairs on the Web, beyond its earlier small-scale prototype. STRAND first uses AltaVista queries on hyperlink anchor text to propose candidate page pairs. It then keeps a pair only if the two pages have similar HTML structure. The pages are reduced to markup tokens and text-chunk lengths, aligned with a diff-style algorithm, and tested for correlated chunk lengths. Because this check uses only lengths, it works for any language. The new version adds automatic language identification to reject pages that match structurally but are in the wrong languages. Date-restricted parent and sibling queries make candidate generation far more productive. Evaluation against human judges on English-French data produced a nearly noise-free corpus of 2491 document pairs. Apart from a small training sample for language identification, the method needs no per-language resources.

Key points

  • Pipeline: Candidate generation is followed by a structural filter (rejection if more than 20% of tokens are unmatched, length correlation at p < .05) and then a language-ID filter.
  • Preliminary English-Spanish test: 90 candidates were checked, of which 24 were true translations. STRAND reached 62.5% recall and 88.2% precision.
  • English-French scale-up (Nov 1998): The system produced 16,763 candidate pairs, an 18-fold increase. Filtering left 3376 pairs, with 79.5% precision and 70.3% recall.
  • Stricter language ID: Training on 10 European languages fixed errors from Dutch pages being taken as English. Precision rose to 100% with 64.1% recall, yielding 2491 pairs of about 1.5 million words per language.
  • Agreement and noise: Kappa was 0.95 between the two judges and 0.75 between STRAND and their agreed judgments. Corpus noise is estimated at 0–8%. Planned future work covers English-Korean, English-Italian and English-Greek, plus a dedicated Web crawler.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of Maryland, College ParkCollege Park · United States83 works · 23,784 citations

📚 Shelves 5 Computer Science · 9,086 works Artificial Intelligence · 8,889 works Arts and Humanities · 9,296 works Language and Linguistics · 9,206 works Information Systems · 523 works

Cite

Cite as Philip Resnik. 1999. Mining the Web for Bilingual Text. Proceedings of the 37th Annual Meeting of the Association for Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.3115/1034678.1034757

@inproceedings{resnik1999mining,
  title = {{Mining the Web for Bilingual Text}},
  author = {Philip Resnik},
  booktitle = {Proceedings of the 37th Annual Meeting of the Association for Computational Linguistics},
  year = {1999},
  doi = {10.3115/1034678.1034757},
  url = {https://doi.org/10.3115/1034678.1034757}
}

References

9 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. P. Brown, J. Cocke, S. Della Pietra, V. Della Pietra, F. Jelinek, R. Mercer, and P. Roossin. 1990. A statistical approach to machine translation. Computational Linguistics, 16(2):79-85. find
  2. Jean Carletta. 1996. Assessing agreement on classification tasks: the Kappa statistic. Computational Linguistics, 22(2):249-254, June. our card
  3. Mark Davis and Ted Dunning. 1995. A TREC evaluation of query translation methods for multi-lingual text retrieval. In Fourth Text Retrieval Conference (TREC-4). NIST. find
  4. Ted Dunning. 1994. Statistical identification of language. Computing Research Laboratory Technical Memo MCCS 94-273, New Mexico State University, Las Cruces, New Mexico. find
  5. David A. Hull and Douglas W. Oard. 1997. Symposium on cross-language text and speech retrieval. Technical Report SS-97-04, American Association for Artificial Intelligence, Menlo Park, CA, March. find
  6. Thomas K. Landauer and Michael L. Littman. 1990. Fully automatic cross-language document retrieval using latent semantic indexing. In Proceedings of the Sixth Annual Conference of the UW Centre for the New Oxford English Dictionary and Text Research, pages pages 31-38, UW Centre for the New OED and… find
  7. Douglas W. Oar& 1997. Cross-language text retrieval research in the USA. In Third DELOS Workshop. European Research Consortium for Informatics and Mathematics March. find
  8. Philip Resnik. 1998. Parallel strands: A preliminary investigation into mining the web for bilingual text. In Proceedings of the Third Conference of the Association for Machine Translation in the Americas, AMTA-98, in Lecture Notes in Artificial Intelligence, 1529, Langhorne, PA, October 28-31. our card
  9. E. M. Voorhees and D. K. Harman. 1998. The seventh Text REtrieval Conference (TREC-7). NIST special publication, Galthersburg, Maryland, November 9-11. http ://trec. nist. gov/pubs, html. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/resnik-1999-mining-web-bilingual-text/