📄 PET: a Tool for Post-editing and Assessing Machine Translation

Summary

Post-editing machine translation is increasingly common in the translation industry. Existing environments, however, are mostly proprietary and hard to modify, and they are usually tied to one or a few MT systems. They also record little about how the editing actually happened. The authors present PET, a free, standalone Java tool that lets people revise output from any MT system, or translate from scratch. While they work, it logs effort indicators for each segment, such as editing time, keystrokes, HTER and optional quality judgements.

Jobs are defined in XML and the tool is configured through simple files. An API allows new attributes, constraints and indicators, for example a per-unit time limit. The paper shows two uses: comparing MT systems and contrasting post-editing with translation, and collecting training labels for quality estimation (QE). In the QE study, translators worked through 600 sentences in one hour for each condition. When sentences were ordered by any QE model, they processed more words per second than with random order. A model trained on post-editing time gave the highest throughput. This suggests that time logged by PET is a useful, low-cost QE label.

Key points

• PET is written with standard Java-6 libraries, so it runs anywhere with a JVM.

• The interface has two columns (source and editable target) and shows neighbouring units for context.

• Optional boxes can show sub-sentential information such as paraphrases, dictionary entries or links, supplied as key-value XML.

• Built-in indicators:

  • editing time
  • assessing time
  • assessment tag
  • keystrokes grouped by key type
  • HTER, computed with tokenization, case ignored and uniform edit costs

• Constraints are attached through attributes. For example, a "maxtime" attribute activates a "Deadline" constraint, which triggers an "EndTaskByForce" event.

• QE training data came from WMT news sets translated by a Moses phrase-based system: news-test2009 (2,525 fr-en sentences) and the first 1,000 sentences of news-test2010 (en-es). Each sentence received three labels: HTER, a 1–4 effort score, and seconds per word.

• Post-editing throughput in words per second:

  Ordering	fr-en	en-es
  HTER model	0.96	0.41
  Effort model	0.91	0.43
  Time model	1.09	0.57
  Unsorted	0.75	0.32

Interesting ideas

• I1. Task-based evaluation of QE models. Instead of reporting correlation, the authors rank sentences with each model and measure how much text a human fixes in a fixed hour. This directly tests practical usefulness and could be applied to any ranking or filtering model.
• I2. Implicit labels from effort logging. Recording editing time without extra effort from annotators produced the best QE labels. Passively logged behaviour can be a cheaper and more informative supervision signal than explicit scores.
• I3. Hiding a unit until editing begins. Annotators cannot think about a segment before the clock starts, which makes time measurements cleaner. This is a simple trick for any timed annotation study.
• I4. System-agnostic input with a "producer" attribute. Each draft records which system or person produced it, so outputs from several systems can be mixed and compared within one job. This is a good design for blind comparative evaluations.
• I5. Constraints and events declared as attributes. New behaviours, such as time limits, become plug-in classes switched on from the input file. This keeps an annotation tool flexible without rewriting its core.