Elamur Research⌕
☰
home / papers / 2010 / Cross-Language Text Classification Using Structural Correspondence Learning

Cross-Language Text Classification Using Structural Correspondence Learning

Paper · 2010 · Peter Prettenhofer and Benno Stein

Summary

The paper addresses cross-language text classification: training a classifier for a target language that has no labeled data, using labeled documents from a source language. The authors adapt structural correspondence learning (SCL), originally a domain adaptation method, to this setting and call it CL-SCL. Each language is treated as its own domain. A small set of pivots, which are word pairs translated by a simple word translation oracle, links the two languages. For each pivot, a linear classifier is trained on unlabeled documents from both languages to predict whether the pivot occurs. The learned weights are then compressed with SVD into a shared low-dimensional representation. Experiments cover sentiment classification on corpora the authors compiled, transferring from English into three target languages: German, French, and Japanese. CL-SCL comes close to a machine-translation baseline (CL-MT) for the European languages and beats it for Japanese. This matters because the method needs only unlabeled text and a few hundred word translations, with no parallel corpus or MT system.

Key points

  • The pivots are 100–500 translated word pairs. The vocabularies on each side are on the order of 10^5 words.
  • Default hyperparameters were m = 450 pivots, k = 100 dimensions, and minimum pivot support φ = 30.
  • The monolingual upper bound averages about 82% accuracy. CL-MT falls 9.9% below it for Japanese but only 5.3% below it for German and French.
  • German and French: CL-MT usually wins, but mostly by less than 1%. The gap reaches about 2% only for French book and music reviews.
  • Japanese: CL-SCL beats CL-MT by roughly 3% on most tasks. It does even better with fewer pivots (150 < m < 250).
  • Unlabeled data: more unlabeled data helps, but gains level off at a labeled-to-unlabeled ratio of about 10 across all nine tasks. A ratio of 1 corresponds to 2,000 unlabeled documents per language.
  • Dimensionality: k between 75 and 150 works equally well regardless of task.

Interesting ideas

  • I1. Language transfer as domain adaptation. Treating each language as a separate domain lets existing domain adaptation tools work across languages. This could help low-resource languages such as Ukrainian, where parallel data is scarce.
  • I2. Pivot-occurrence prediction as auxiliary tasks. Predicting whether a translated word pair appears in a document turns unlabeled text into many cheap supervised problems. The shared structure learned from these problems becomes a bridge between the languages, and the same trick could be reused for any feature set with a few known anchors.
  • I3. Choosing pivots for the task. Pivots are selected with the target task in mind, so the induced correspondences capture task-specific usage, not just dictionary meaning. For example, "beautiful" in book reviews correlates with illustrations and poetry. Pivots chosen this way can encode domain pragmatics that general bilingual resources miss.
  • I4. Checking learned correspondences qualitatively. Listing the words most correlated with each pivot, and separating meaning-based from usage-based correlations, is a simple way to see what a representation has learned.
  • I5. Comparing against MT and a monolingual upper bound. Reporting both shows where translation quality limits performance. In this paper, the MT gap was much larger for Japanese, and that is exactly where CL-SCL helped most. This kind of comparison is a good diagnostic for distant language pairs.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Bauhaus-Universität WeimarWeimar · Germany9 works · 3,111 citations

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 9,086 worksArtificial Intelligence · 8,889 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 works

🧭 Topics 1Research subjects and languages the work is about.Multilingual & cross-lingual NLP · 467 works

Cite

Cite as Peter Prettenhofer and Benno Stein. 2010. Cross-Language Text Classification Using Structural Correspondence Learning. Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{prettenhofer2010cross,
  title = {{Cross-Language Text Classification Using Structural Correspondence Learning}},
  author = {Peter Prettenhofer and Benno Stein},
  booktitle = {Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics},
  year = {2010},
  url = {https://aclanthology.org/P10-1114/}
}

References

27 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Rie-K. Ando and Tong Zhang. 2005a. A framework for learning predictive structures from multiple tasks and unlabeled data. J. Mach. Learn. Res., 6:1817– 1853. find
  2. Rie-K. Ando and Tong Zhang. 2005b. A high-performance semi-supervised learning method for text chunking. In Proceedings of ACL-05, pages 1– 9, Ann Arbor. find
  3. Mikhail Bautin, Lohit Vijayarenu, and Steven Skiena. 2008. International sentiment analysis for news and blogs. In Proceedings of ICWSM-08, pages 19–26, Seattle. find
  4. Nuria Bel, Cornelis H. A. Koster, and Marta Villegas. 2003. Cross-lingual text categorization. In Proceedings of ECDL-03, pages 126–139, Trondheim. find
  5. John Blitzer, Ryan McDonald, and Fernando Pereira. 2006. Domain adaptation with structural correspondence learning. In Proceedings of EMNLP-06, pages 120–128, Sydney.
  6. John Blitzer, Mark Dredze, and Fernando Pereira. 2007. Biographies, bollywood, boom-boxes and blenders: Domain adaptation for sentiment classification. In Proceedings of ACL-07, pages 440–447, Prague.
  7. Hal Daumé III. 2007. Frustratingly easy domain adaptation. In Proceedings of ACL-07, pages 256–263, Prague.
  8. Susan T. Dumais, Todd A. Letsche, Michael L. Littman, and Thomas K. Landauer. 1997. Automatic cross-language retrieval using latent semantic indexing. In AAAI Symposium on CrossLanguage Text and Speech Retrieval. find
  9. Jenny-R. Finkel and Christopher-D. Manning. 2009. Hierarchical bayesian domain adaptation. In Proceedings of HLT/NAACL-09, pages 602–610, Boulder. find
  10. Blaž Fortuna and John Shawe-Taylor. 2005. The use of machine translation tools for cross-lingual text mining. In Proceedings of the ICML Workshop on Learning with Multiple Views. find
  11. Alfio Gliozzo and Carlo Strapparava. 2005. Cross language text categorization by acquiring multilingual domain models from comparable corpora. In Proceedings of the ACL Workshop on Building and Using Parallel Texts. find
  12. Alfio Gliozzo and Carlo Strapparava. 2006. Exploiting comparable corpora and bilingual dictionaries for cross-language text categorization. In Proceedings of ACL-06, pages 553–560, Sydney. find
  13. Kanayama Hiroshi, Nasukawa Tetsuya, and Watanabe Hideo. 2004. Deeper sentiment analysis using machine translation technology. In Proceedings of COLING-04, pages 494–500, Geneva. find
  14. Jing Jiang and Chengxiang Zhai. 2007. A two-stage approach to domain adaptation for statistical classifiers. In Proceedings of CIKM-07, pages 401–410, Lisbon. find
  15. Victor Lavrenko, Martin Choquette, and W. Bruce Croft. 2002. Cross-lingual relevance models. In Proceedings of SIGIR-02, pages 175–182, Tampere. find
  16. Yaoyong Li and John S. Taylor. 2007. Advanced learning algorithms for cross-language patent retrieval and classification. Inf. Process. Manage., 43(5):1183–1199. find
  17. Xiao Ling, Gui-R. Xue, Wenyuan Dai, Yun Jiang, Qiang Yang, and Yong Yu. 2008. Can chinese web pages be classified with english data source? In Proceedings of WWW-08, pages 969–978, Beijing. find
  18. Douglas W. Oard. 1998. A comparative study of query and document translation for cross-language information retrieval. In Proceedings of AMTA-98, pages 472–483, Langhorne. find
  19. J. Scott Olsson, Douglas W. Oard, and Jan Hajič. 2005. Cross-language text classification. In Proceedings of SIGIR-05, pages 645–646, Salvador. find
  20. Bo Pang, Lillian Lee, and Shivakumar Vaithyanathan. 2002. Thumbs up?: sentiment classification using machine learning techniques. In Proceedings of EMNLP-02, pages 79–86, Philadelphia.
  21. Martin Potthast, Benno Stein, and Maik Anderka. 2008. A wikipedia-based multilingual retrieval model. In Proceedings of ECIR-08, pages 522–530, Glasgow. find
  22. Ariadna Quattoni, Michael Collins, and Trevor Darrell. 2007. Learning visual representations using images with captions. In Proceedings of CVPR-07, pages 1–8, Minneapolis. find
  23. Leonardo Rigutini, Marco Maggini, and Bing Liu. 2005. An em based training algorithm for cross-language text categorization. In Proceedings of WI-05, pages 529–535, Compiègne. find
  24. Shai Shalev-Shwartz, Yoram Singer, and Nathan Srebro. 2007. Pegasos: Primal estimated sub-gradient solver for svm. In Proceedings of ICML-07, pages 807–814, Corvalis. find
  25. Yoshimasa Tsuruoka, Jun’ichi Tsujii, and Sophia Ananiadou. 2009. Stochastic gradient descent training for l1-regularized log-linear models with cumulative penalty. In Proceedings of ACL/AFNLP-09, pages 477–485, Singapore. find
  26. Xiaojun Wan. 2009. Co-training for cross-lingual sentiment classification. In Proceedings of ACL/AFNLP-09, pages 235–243, Singapore.
  27. Tong Zhang. 2004. Solving large scale linear prediction problems using stochastic gradient descent algorithms. In Proceedings of ICML-04, pages 116– 124, Banff. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/prettenhofer-2010-cross-language-text-classification-structural/