Elamur Research⌕
☰
home / papers / 2014 / Adaptation of Long-Unit-Word Analysis System to Different Part-Of-Speech Tagset

Adaptation of Long-Unit-Word Analysis System to Different Part-Of-Speech Tagset

Paper · 2014 · Shunsuke Kozawa, Kiyotaka Uchimoto and Yasuharu Den

Summary

The paper examines how to reuse an existing analyzer for Japanese long-unit words (LUWs) on a corpus with a different part-of-speech scheme. The authors take an LUW analyzer originally built for the Corpus of Spontaneous Japanese (CSJ), which uses a flat school-grammar tagset, and adapt it to the Balanced Corpus of Contemporary Written Japanese (BCCWJ), which uses the hierarchical UniDic tagset. Assuming correct short-unit-word (SUW) input, they make four changes. First, they add hierarchical features generalized to higher POS levels. Second, they replace rule-based rewriting with SVM-based models that predict POS, conjugation type and conjugation form. Third, they add BCCWJ-specific cues, namely word origin and circled-character markers. Fourth, they redefine the chunking labels so that category information is tied to the last SUW of each LUW. In experiments, the improved method beat the original approach on BCCWJ and also helped on CSJ. The work matters because it identifies what must change in an analyzer when the tagset changes. The authors also release the open-source tool Comainu.

Key points

  • Tagset differences:
    • CSJ has 15 POS, 59 conjugation types and 8 conjugation forms, all in a flat scheme.
    • BCCWJ/UniDic uses a 4-level POS hierarchy with 54 tags at the finest level, 115 conjugation types and 36 conjugation forms.
  • Original method: Uchimoto & Isahara's approach treats LUW identification as sequence chunking over SUWs. It uses the labels Ba/Ia/B/I and applies rewrite rules as post-processing.
  • Tools: The chunker was trained with CRF++, and the category-estimation models with YamCha (SVM, polynomial kernel of degree 3, one-versus-rest).
  • Comainu:
    • It is trained on the BCCWJ core data: 45,342 sentences, 828,133 LUWs and 1,047,069 SUWs.
    • It accepts plain text, which it pre-processes with MeCab and UniDic, or SUW sequences as input.
    • The chunking model can use either CRF or SVM.
  • Missing results: The exact accuracy scores were in the part of the paper omitted from the provided text, so they are not reported here.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🔬National Institute of Information and Communications TechnologyTokyo · Japan23 works · 2,090 citations🎓Chiba UniversityChiba · Japan3 works · 10 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.Part-of-speech tagging · 126 works

Cite

Cite as Shunsuke Kozawa, Kiyotaka Uchimoto and Yasuharu Den. 2014. Adaptation of Long-Unit-Word Analysis System to Different Part-Of-Speech Tagset. Journal of Natural Language Processing.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.5715/jnlp.21.379

@article{kozawa2014adaptation,
  title = {{Adaptation of Long-Unit-Word Analysis System to Different Part-Of-Speech Tagset}},
  author = {Shunsuke Kozawa, Kiyotaka Uchimoto and Yasuharu Den},
  journal = {Journal of Natural Language Processing},
  year = {2014},
  doi = {10.5715/jnlp.21.379},
  url = {https://doi.org/10.5715/jnlp.21.379}
}

References

6 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. 近藤泰弘 (2012). 日本語通時コーパスの設計について . 国語研プロジェクトレビュー , 3, pp. 84–92. find
  2. Kudo, T., Yamamoto, K., and Matsumoto, Y. (2004). “Applying Conditional Random Fields to Japanese Morphological Analysis.” In Proceedings of the 2004 Conference on Empirical Methods in Natural Language Processing, pp. 230–237.
  3. 前川喜久雄 (2004). 『日本語話し言葉コーパス』の概要 . 日本語科学 , 15, pp. 111–133. find
  4. Maekawa, K. (2008). “Balanced Corpus of Contemporary Written Japanese.” In Proceedings of the 6th Workshop on Asian Language Resources, pp. 101–102. find
  5. 中野洋 (1998). 言語情報処理 . 岩波講座 「言語の科学」 , 9, pp. 149–199. find
  6. Neubig, G., Nakata, Y., and Mori, S. (2011). “Pointwise Prediction for Robust, Adaptable find

Updated

This page in Markdown → https://md.research.elamur.ai/free/en/papers/kozawa-2014-adaptation-long-unit-word-analysis/