Elamur Research
☰
home / papers / 2003 / The Web as a Parallel Corpus

The Web as a Parallel Corpus

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)" does not allow redistributing the text itself. Original — aclanthology.org/J03-3002.pdf

Summary

Bitexts are central to statistical machine translation, lexicon induction, cross-language retrieval and annotation projection. For most language pairs, however, they exist only in narrow, often licensed collections such as the Hansards, UN proceedings or religious texts. Resnik and Smith treat the World Wide Web as a large, untapped source of translated documents. They present an extended version of STRAND, a system that identifies two web pages whose contents translate each other. Its core insight is that bilingual site authors usually give both language versions the same HTML layout, so candidate pairs can be judged by markup alone.

The article adds three upgrades:

  • a supervised classifier over structural alignment features, replacing hand-tuned thresholds;
  • a content-based translational-equivalence measure that ignores markup yet performs comparably to the structural filter;
  • an adaptation to the Internet Archive for large-scale mining.

Combining structural and content matching, the authors build a sizable English-Arabic corpus of web document pairs. This shows that the approach can serve low-density language pairs. (The evaluation figures appear in a portion of the text not supplied here.)

Key points

  • STRAND works in three stages: locate pages likely to have translations, generate candidate pairs, then filter out non-translations by comparing structure.
  • Original page discovery used AltaVista queries for "parent" pages (links to several language versions) and "sibling" pages (a link to the same page in another language). A 10-line proximity filter checks that the language links appear close together in the HTML source.
  • A later spider component pairs pages by URL substitution rules (e.g., english → big5). Pairs can also be screened by length, assuming length(E) ≈ C·length(F) for a constant C specific to the language pair.
  • For structural filtering, each page is linearized into START/END tag tokens and text-chunk lengths. The two sequences are then aligned with dynamic programming (Hunt and McIlroy 1975), deliberately avoiding slower tree alignment.
  • The work builds on Resnik (1998, 1999) and Smith (2001, 2002). The final resource is an English-Arabic bitext mined from the Internet Archive.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

181 papers of our library cite it (of the 140673 whose bibliographies we read); 6 of them have a page here, 175 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.1162/089120103322711578

@article{resnik2003web,
  title = {{The Web as a Parallel Corpus}},
  author = {Resnik, Philip and Smith, Noah A.},
  journal = {Computational Linguistics},
  year = {2003},
  doi = {10.1162/089120103322711578},
  url = {https://doi.org/10.1162/089120103322711578}
}

References

44 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Abney, Steven. 2002. Bootstrapping. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics (ACL), Philadelphia, pages 360–367. find
  2. Ahuja, Ravindra K., Thomas L. Magnati, and James B. Orlin. 1993. Network Flows: Theory, Algorithms, and Applications. Prentice Hall, Englewood Cliffs, NJ. find
  3. Al-Onaizan, Yaser, Jan Curin, Michael Jahr, Kevin Knight, John Lafferty, I. Dan Melamed, Franz-Josef Och, David Purdy, Noah A. Smith, and David Yarowsky. 1999. Statistical machine translation. Technical report, Johns Hopkins University. Available at citeseer. nj.nec.com/al-onaizan99statistical.html… find
  4. Beesley, Kenneth R. 1988. Language identifier: A computer program for automatic natural-language identification of on-line text. In D. L. Hammond, editor, Language at the Crossroads: Proceedings of the 29th Annual Conference of the American Translators Association, Medford, NJ, pages 47–54. find
  5. Blum, Avrim and Tom Mitchell. 1998. Combining labeled and unlabeled data with co-training. In Proceedings of the Eleventh Annual Conference on Computational Learning Theory, pages 92–100, Madison, WI, July. find our card
  6. Broder, Andrei Z., Steven C. Glassman, Mark S. Manasse, and Geoffrey Zweig. 1997. Syntactic clustering of the Web. In Proceedings of the Sixth International World-Wide Web Conference, pages 391–404, Santa Clara, CA, April. find
  7. Brown, Peter F., John Cocke, Stephen A. Della Pietra, Vincent J. Della Pietra, Frederick Jelinek, John D. Lafferty, Robert L. Mercer, and Paul S. Roossin. 1990. A statistical approach to machine translation. Computational Linguistics, 16(2):79–85. find
  8. Cabezas, Clara, Bonnie Dorr, and Philip Resnik. 2001. Spanish language processing at University of Maryland: Building infrastructure for multilingual applications. In Proceedings of the Second International Workshop on Spanish Language Processing and Language Technologies (SLPLT-2). Available at ft… find
  9. Cavnar, William B. and John M. Trenkle. 1994. N-gram-based text categorization. In Proceedings of Third Annual Symposium on Document Analysis and Information Retrieval (SDAIR-94), pages 161–175, Las Vegas, NV. find
  10. Chen, Jiang and Jian-Yun Nie. 2000. Parallel Web text mining for cross-language information retrieval. In Recherche d’Informations Assistée par Ordinateur (RIAO), pages 62–77, Paris, April. find
  11. Church, Kenneth W. and Robert Mercer. 1993. Introduction to the special issue on computational linguistics using large corpora. Computational Linguistics, 19(1):1–24. find
  12. Dabbadie, Marianne, Anthony Hartley, Margaret King, Keith J. Miller, Widad Mustafa El Hadi, Andrei Popescu-Bellis, Florence Reeder, and Michelle Vanni. 2002. A hands-on study of the reliability and coherence of evaluation metrics. In Workshop on Machine Translation Evaluation: Human Evaluators Meet… find
  13. Darwish, Kareem. 2002. Building a shallow Arabic morphological analyser in one day. In Proceedings of the Workshop on Computational Approaches to Semitic Languages, pages 22–29, Philadelphia, July. find our card
  14. Davis, Mark and Ted Dunning. 1995. A TREC evaluation of query translation methods for multi-lingual text retrieval. In Fourth Text Retrieval Conference (TREC-4), pages 483–498. NIST, Gaithersburg, MD. find
  15. Diab, Mona and Philip Resnik. 2002. An unsupervised method for word sense tagging using parallel corpora. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics (ACL), Philadelphia, July. find
  16. Dunning, Ted. 1994. Statistical identification of language. Computing Research Laboratory Technical Memo MCCS 94-273, New Mexico State University, Las Cruces, NM. Available at http://citeseer.nj.nec.com/dunning94statistical. html. find
  17. Gale, William A. and Kenneth W. Church. 1991. Identifying word correspondences in parallel texts. In Fourth DARPA Workshop on Speech and Natural Language, pages 152–157, Asilomar, CA, February. find
  18. Hunt, James W. and M. Douglas McIlroy. 1975. An algorithm for differential file comparison. Technical Memorandum 75-1271-11, Bell Laboratories, October. find
  19. Hwa, Rebecca, Philip Resnik, Amy Weinberg, and Okan Kolak. 2002. Evaluating translational correspondence using annotation projection. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics (ACL), pages 392–399, Philadelphia, July. find
  20. Ingle, Norman C. 1976. A language identification table. The Incorporated Linguist, 15(4):98–101. find
  21. Landauer, Thomas K. and Michael L. Littman. 1990. Fully automatic cross-language document retrieval using latent semantic indexing. In Proceedings of the Sixth Annual Conference of the UW Centre for the New Oxford English Dictionary and Text Research, pages 31–38, Waterloo, Ontario, October. find
  22. Lin, Dekang. 1998. An information-theoretic definition of similarity. In Proceedings of the Fifteenth International Conference on Machine Learning (ICML-98), pages 296–304, Madison, WI. find
  23. Lopez, Adam, Michael Nossal, Rebecca Hwa, and Philip Resnik. 2002. Word-level alignment for multilingual resource acquisition. In Workshop on Linguistic Knowledge Acquisition and Representation: Bootstrapping Annotated Language Data, at the Third International Conference on Language Resources and Ev… find
  24. Ma, Xiaoyi and Mark Liberman. 1999. Bits: A method for bilingual text search over the Web. In Machine Translation Summit VII, September. Available at http://www.ldc.upenn.edu/Papers/MTSVII1999/BITS.ps. find
  25. MacKay, David and Linda Peto. 1995. A hierarchical Dirichlet language model. Journal of Natural Language Engineering, 1(3):1–19. find
  26. Melamed, I. Dan. 1997. Automatic discovery of non-compositional compounds in parallel data. In Proceedings of the 2nd Conference on Empirical Methods in Natural Language Processing (EMNLP-97), pages 97–108, Providence, RI, August. find
  27. Melamed, I. Dan. 2000. Models of translational equivalence among words. Computational Linguistics, 26(2):221–249. find
  28. Menezes, Arul and Stephen D. Richardson. 2001. A best-first alignment algorithm for automatic extraction of transfer mappings from bilingual corpora. In Proceedings of the ACL 2001 Workshop on Data-Driven Methods in Machine Translation, pages 39–46, Toulouse, France. find
  29. Nie, Juan-Yun and Jian Cai. 2001. Filtering noisy parallel corpora of Web pages. In IEEE Symposium on Natural Language Processing and Knowledge Engineering, pages 453–458, Tucson, AZ, October. find
  30. Oard, Douglas W. 1997. Cross-language text retrieval research in the USA. In Third DELOS Workshop on Cross-Language Information Retrieval, Zurich, March. European Research Consortium for Informatics and Mathematics, Sophia Antipolis, France. Available at http://www.ercim.org/publication/ws-proceedi… find
  31. Och, Franz-Josef and Hermann Ney. 2002. Discriminative training and maximum entropy models for statistical machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics (ACL), pages 295–302, Philadelphia, July. find
  32. Resnik, Philip. 1998. Parallel strands: A preliminary investigation into mining the Web for bilingual text. In Proceedings of the Third Conference of the Association for Machine Translation in the Americas (AMTA-98). Langhorne, PA, October 28-31. Lecture Notes in Artificial Intelligence 1529. Availa… find
  33. Resnik, Philip. 1999. Mining the Web for bilingual text. In Proceedings of the 37th Annual Meeting of the ACL, pages 527–534, College Park, MD, June. find
  34. Resnik, Philip and I. Dan Melamed. 1997. Semi-automatic acquisition of domain-specific translation lexicons. In Fifth Conference on Applied Natural Language Processing, Washington, DC. find
  35. Resnik, Philip, Douglas Oard, and Gina Levow. 2001. Improved cross-language retrieval using backoff translation. In Proceedings of the First International Conference on Human Language Technology Research (HLT-2001), pages 153–155, San Diego, CA, March. find
  36. Resnik, Philip, Mari Broman Olsen, and Mona Diab. 1999. The Bible as a parallel corpus: Annotating the “Book of 2000 Tongues.” Computers and the Humanities, 33:129–153. find
  37. Riloff, Ellen, Charles Schafer, and David Yarowsky. 2002. Inducing information extraction systems for new languages via cross-language projection. In Nineteenth International Conference on Computational Linguistics (COLING-2002), pages 828–834, Taipei, Taiwan, August. find
  38. Shannon, Claude E. 1948. A mathematical theory of communication. Bell Systems Technical Journal, 27:379–423, 623–656. find
  39. Smith, Noah A. 2001. Detection of translational equivalence. Undergraduate honors thesis, University of Maryland College Park. Available at http://nlp.cs.jhu.edu/∼nasmith/cmsc-thesis.ps. find
  40. Smith, Noah A. 2002. From words to corpora: Recognizing translation. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 95–102, Philadelphia, July. find
  41. Tiedemann, Jörg. 1999. Automatic construction of weighted string similarity measures. In Joint SIGDAT Conference on Empirical Methods in Natural Language Processing and Very Large Corpora, pages 213–219, College Park, MD, June. find
  42. Yarowsky, David. 1995. Unsupervised word sense disambiguation rivaling supervised methods. In Proceedings of the 33rd Annual Meeting of the Association for Computational Linguistics (ACL), pages 189–196, Cambridge, MA. find our card
  43. Yarowsky, David and Grace Ngai. 2001. Inducing multilingual POS taggers and NP bracketers via robust projection across aligned corpora. In Proceedings of the Second Meeting of the North American Association for Computational Linguistics (NAACL-2001), pages 200–207, Pittsburgh, PA, June. find
  44. Yarowsky, David, Grace Ngai, and Richard Wicentowski. 2001. Inducing multilingual text analysis tools via robust projection across aligned corpora. In Proceedings of the First International Conference on Human Language Technology Research (HLT-2001), pages 161–168, San Diego, CA, March. find