📄 ASPEC: Asian Scientific Paper Excerpt Corpus

Summary

Before ASPEC, no sizeable parallel corpus spanned many scientific disciplines. Earlier automatic extractions from portals such as MEDLINE or Elsevier were small and limited to narrow subject areas, which hampered machine translation of research literature written in languages other than English. The authors describe ASPEC, a resource built by JST and NICT during a 2006–2010 Japanese-Chinese MT project. It has two parts. ASPEC-JE holds roughly 3 million Japanese-English sentence pairs, automatically aligned from existing bilingual paper abstracts. ASPEC-JC holds about 0.68 million Chinese-Japanese pairs, created by human translation of abstracts and body paragraphs from Japanese papers. Both parts come with fixed development, dev-test and test splits. The corpus is free for non-commercial use and serves as the official data for the WAT evaluation campaign. In WAT2015, the best systems reached about 4 on the 5-point JPO adequacy scale for every language pair except ASPEC-JC, which the authors take as evidence of corpus quality. The resource matters because it enables domain-specific MT and NLP research for Asian scientific texts.

Key points

• ASPEC-JE
  • Splits: 3,008,500 training pairs; dev 1,790, dev-test 1,784 and test 1,812 sentences, each drawn from 400 documents.
  • Source: about 2 million JST-owned abstracts, aligned with the Utiyama and Isahara (2007) method.
  • Each pair carries a similarity score, and the data is sorted by that score.
• ASPEC-JC
  • Splits: 672,315 training pairs; dev 2,090, dev-test 2,148 and test 2,107 sentences.
  • Construction: human translation from Japanese into Chinese, sentence by sentence, using papers from JDreamII and J-STAGE.
  • Coverage: 8 fields only; Medicine and Information each make up 28%.
• Field labels: every JE pair is tagged with a field letter (A–Z). Medicine dominates at about 33.8% of the training data.
• Copyright handling: for ASPEC-JE, the Japanese side is the original, and the English abstracts were translated manually from it.
• WAT human evaluation:
  • A crowdsourced pairwise comparison against a baseline, with 5 workers per sentence.
  • JPO adequacy scoring of the top 3 systems by experts, with 2 annotators on 200 sentences.

Interesting ideas

• I1. Keep the alignment confidence score and sort by it. Users can then pick their own trade-off between data size and cleanliness, for example training only on the best 1M pairs. This would work for any automatically mined Ukrainian parallel corpus.
• I2. Choose texts for human translation by vocabulary coverage. ASPEC-JC selected passages iteratively to cover as many nouns, verbs and case frames as possible, rather than sampling at random. This makes a limited translation budget go further for low-resource pairs.
• I3. Separate splits at the paper level. No paper contributes sentences to both training and test sets. This prevents leakage of document-specific terminology and mirrors real use, where new documents arrive unseen.
• I4. Release test sets that allow document reconstruction. Only 1-to-1 aligned documents were included, and field proportions were balanced across splits. This keeps document-level evaluation possible and makes per-field results comparable.
• I5. Aggregate noisy crowd judgments robustly. Each of 5 votes counts as +1, 0 or −1, and a win or loss is declared only if the sum reaches ±2. The final score is 100·(W−L)/(W+L+T), a cheap and noise-tolerant scheme for human MT evaluation by non-experts.