Elamur Research⌕
☰
home / papers / 2006 / Formal v. Informal: Register-Differentiated Arabic MT Evaluation in the PLATO Paradigm

Formal v. Informal: Register-Differentiated Arabic MT Evaluation in the PLATO Paradigm

Paper · 2006 · Keith J. Miller and Michelle Vanni

Summary

The paper asks whether a set of fine-grained linguistic quality measures can show how suitable machine translation output is for particular analyst tasks, and whether that suitability depends on the register of the source text. Within the PLATO program, the authors compared five Arabic-English MT engines on 10 input texts. Half were formal Modern Standard Arabic newswire. The other half were informal web writing, a mixture of dialect and genre features drawn from discussion lists, weblogs and online editorials. Ten assessors applied eight tests covering clarity, coherence, morphology, syntax, untranslated words, name rendering, domain terminology and adequacy, which produced 800 score sets. The engines handled vocabulary coverage about equally well on both registers. Their scores split apart on clarity, name rendering and domain terms. The authors link each test to a downstream task. On that basis, the systems look usable for triage on either kind of input, but relevance judgment, information extraction and topic identification need register taken into account. The study also shows that per-dimension score profiles can tell apart text a system was presumably tuned for from text it was not.

Key points

  • Setup: 5 Arabic-English MT systems, 10 source texts in two registers (formal MSA newswire vs. informal web text), 8 tests and 10 assessors, giving 800 data sets.
  • Clarity is judged without any source or reference text, on a 0–4 scale. Adequacy follows the 1994 DARPA test and compares output with a reference on a 1–5 scale.
  • Several scores are computed as one minus an error ratio:
    • Morphology: word-formation fixes divided by the number of inflectable words.
    • Syntax: minimal edits needed to reach grammaticality, divided by sentence length (similar in spirit to TER).
    • Coherence: a binary judgment of whether some RST function can plausibly be assigned to the sentence in context.
  • Dictionary Update, Name Rendering and Domain Terminology are checked against a human reference using exact match, which keeps rater agreement near maximal and makes automation easier.
  • Untranslated-word rates were similar across registers. Clarity, name rendering and domain terms diverged. Earlier refinement had already brought Clarity inter-rater reliability to about .8.

Interesting ideas

  • I1. The evaluation output is a profile of separate linguistic scores, a "signature", and each score is tied to the downstream task it predicts (lexical coverage for triage, names for extraction). A single number would hide which uses the output is fit for.
  • I2. Test sets are split by register so that each system faces input close to its training data and input far from it. The gap between the two profiles then exposes brittleness that a pooled average would blur.
  • I3. Raters give a quick reference-free clarity judgment first and only later run the analytic tests, an ordering meant to prevent bias and training effects. Fixing the order of human judgments protects the cheap intuitive signal from being shaped by the detailed ones.
  • I4. Coherence is reduced to a yes/no question: can any RST relation reasonably link this sentence to its context? This turns a vague discourse notion into a constrained, checkable decision.
  • I5. Name and term accuracy is scored by exact match against a list extracted from a human reference. The authors chose this to make scoring deterministic and easy to automate.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Mova's evaluation could report separate scores for tokens, tree, frames and world, each mapped to the reader tasks that depend on it, so a regression points to the specific capability it breaks.
  • I2 (tree): Parser accuracy could be reported separately for edited prose and informal web or forum text, so we can see where the UD dialect or training data stops generalizing.
  • I3 (back): When people rate generated or round-tripped text, a fluency-only judgment could be collected before they see the source graph, keeping the readability signal independent of the fidelity check.
  • I4 (frames): The same test could flag sentences whose frames attach to nothing in the preceding discourse, giving an explainable sign of a failed coreference or relation analysis.
  • I5 (world): Entity lists built from gold annotations could be matched exactly against the entities in Mova's text world, giving a cheap, unambiguous check on entity extraction and naming.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢MitreBedford · United States10 works · 2,631 citations🔬DEVCOM Army Research LaboratoryAdelphi · United States14 works · 2,292 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 2Research subjects and languages the work is about.Machine translation · 1,087 worksSemitic languages · 2,908 works

Cite

Cite as Keith J. Miller and Michelle Vanni. 2006. Formal v. Informal: Register-Differentiated Arabic MT Evaluation in the PLATO Paradigm. Proceedings of the Fifth International Conference on Language Resources and Evaluation (LREC'06).

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{miller2006formal,
  title = {{Formal v. Informal: Register-Differentiated Arabic MT Evaluation in the PLATO Paradigm}},
  author = {Keith J. Miller and Michelle Vanni},
  booktitle = {Proceedings of the Fifth International Conference on Language Resources and Evaluation (LREC'06)},
  year = {2006},
  doi = {10.63317/5m5ph6wf35d7},
  url = {https://doi.org/10.63317/5m5ph6wf35d7}
}

References

25 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Aikmajian, A., R. A. Demers and R. M. Harnish. (1988). Linguistics: An Introduction to Language and Communication, Second Edition. Cambridge, MA: MIT Press. 521. find
  2. Church, K. and E. Hovy. (1993). Good Applications for Crummy Machine Translation. Machine Translation 8: 239-258. ↗
  3. Crystal, D. (2001). Language and the Internet. Cambridge: CUP. ↗
  4. Freeman, A. (2002). In Search of a koine in Sana’a. Unpublished Ph.D. Dissertation. Departments of Linguistics and Near Eastern Studies. University of Michigan. find
  5. Henderson, J. (2004). The Text Interpretability Rating Scale. Presentation at Panel on Trends in MT Evaluation. Conference of the Association for Machine Translation in the Americas (AMTA) Washington, DC. find
  6. Hovy, E., M. King and A. Popescu-Belis. (2003). Principles of Context-Based Machine Translation Evaluation. Machine Translation 17:1. 43–75. ↗
  7. International Standards for Language Engineering. (2000). ISLE Classification MT Evaluations. http://www.isi.edu/natural-language/mteval. In Proceedings of Hands-on Workshop on Machine Translation Evaluation. Association for Machine Translation in the Americas, Cuernavaca, Mexico. ↗
  8. Jones, D. and G. Rusk. 2000. Toward a Scoring Function for Quality-Driven Machine Translation. In Proceedings of COLING-2000. find
  9. Lavie, L., K. Sagae and S. Jayaraman. (2004). The Significance of Recall in Automatic Metrics for MT Evaluation. In Machine Translation: From Real Users to Research, Proceedings of the Sixth Conference of the Association for Machine Translation in the Americas (AMTA-2004), Washington, D.C. find
  10. Lyons, J. (1981). Language and Linguistics: An Introduction. Cambridge:CUP. 292. find
  11. Mann, W.C. and S.A. Thompson. (1988). Rhetorical Structure Theory: Toward a functional theory of text organization. Text 8:3. pp. 243-281. ↗
  12. Melamed, I, R. Green, and J. Turian (2003) Precision and Recall of Machine Translation. In Proceedings on NAACL/HLT, Edmonton, Canada.
  13. Miller, K.J. and M. Vanni (2002). Scaling the ISLE Taxonomy: Development of Metrics for Multi-Dimensional Characterization of Machine Translation Quality. In Proceedings of MT Summit VIII. Santiago de Campostela, Spain. find
  14. Miller, K. J. and M. Vanni. (2005). Inter-rater Agreement Measures and the Refinement of Metrics in the PLATO MT Evaluation Paradigm. In Proceedings of MT Summit X, Phuket, Thailand. ↗
  15. Papineni, K., S. Roukos, T. Ward and W. Zhu. (2001). BLEU: A Method for Automatic Evaluation of Machine Translation, IBM Research Report, RC22176, September 2001.
  16. Papineni, K., S. Roukos, T. Ward, J. Henderson, and F. Reeder. (2002). Corpus-Based Comprehensive and Diagnostic MT Evaluation: Initial Arabic, Chinese, French, and Spanish Results. Proceedings of the Human Language Technology Conference. find
  17. Ryding, K.C. and D.J. Mehall. (2005). Formal Spoken Arabic. Washington, DC: Georgetown University Press. find
  18. Snover, M., B.J. Dorr, R. Schwartz, J Makhoul, L. Micciula and R. Weischedel. (2005). A Study of Translation Error Rate with Targeted Human Annotation. Technical Report LAMP-TR-126, CS-TR-4755, UMIACS-TR-2005-58, University of Maryland, College Park. find
  19. Tate, C., C. Voss, B. J. Dorr and E. Slud. (2005). Toward a Predictive Statistical Model of Task-based Performance Using Automatic MT Evaluation Metrics. In Proceedings of the Association for Computational Linguistics Workshop on Intrinsic and Extrinsic Evaluation Measures for MT and/or Summarization. Ann Arbor, MI. find
  20. Taylor, K. and J. White. (1998). Predicting What MT is Good for: User Judgments and Task Performance. In: Proceedings of the 1998 conference of the Association of Machine Translation in the Americas. 364-373. find
  21. Vanni, M. (1998). Evaluating MT Systems: Testing the Feasibility of a Task-Diagnostic. Proceedings of the Association for Information Management (ASLIB): Translating and the Computer 20. find
  22. Vanni, M. & Miller, K.J. (2002). Scaling the ISLE Framework: Use of Existing Corpus Resources for Validation of MT Metrics across Languages. In: Proceedings of the Third International Conference on Language Resources and Evaluation (LREC). Las Palmas de Gran Canaria, Spain. 1254-1262. find
  23. "Genre." Merriam-Webster On-Line Dictionary. (2006). http://www.m-w.com/dictionary/genre. (27 Feb 2006). ↗
  24. Weinberg, A. (2005). Machine Translation Evaluation at the Center for Advanced Study of Language. Workshop on Intelligence Analysis, Mitre Corporation. find
  25. White, J. and O’Connell T.A. (1996). The ARPA MT evaluation methodologies: evolution, lessons and future approaches. Proceedings of the 1994 Conference, of the Association for Machine Translation in the Americas.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/miller-2006-formal-v-informal-register-differentiated/