📄 A Study of Performance Evaluation for GA-ILMT Using Travel English

Summary

The paper tests GA-ILMT, an English-to-Japanese translator that learns its rules only from example sentence pairs, with no grammar or analytical knowledge. It combines inductive learning with genetic algorithms. The system breeds new example pairs by crossover and mutation. It derives sentence and word rules by comparing the shared and differing substrings of pairs. User corrections score each rule, and rules that perform badly are dropped. The authors test it on travel dialogue drawn from ten English phrasebooks written for Japanese readers. The data covers four scenes, and each run starts from an empty dictionary. They compare GA-ILMT with two rule-based systems, A and B. Overall accuracy is about equal to B and clearly above A. Unlike the fixed systems, GA-ILMT gets better as it sees more sentences. Worked examples suggest its output suits the conversational setting better, for instance rendering "anything cheaper" as a cheaper room when checking in. The main claim is that a learner using only examples can reach rule-based quality in a narrow domain and keep improving.

Key points

• The data covers four travel scenes: on the plane (456 sentences), at the airport (296), on the check-in (345) and on the telephone (607). Average sentence length is about 5.3–5.8 words.
• The learning setup uses one-point and two-point crossover anchored at substrings that two examples share, plus random mutation at 2.0%. A rule is removed once it has been used more than five times with fitness below 25.0%.
• Fitness is the share of a rule's uses that produced a correct translation. When several outputs compete, the system prefers rules that are closer to the input and have higher fitness.
• The books were fed in three orders: forward (α), reverse (β) and interleaved (γ). The strict scoring counts an output as correct only if it is identical or very close to a human translation.
• Correct rates were 17.8–26.0% for GA-ILMT, 6.7–16.9% for system A and 18.6–22.0% for system B.

Interesting ideas

• I1. New training pairs are made by swapping material between two aligned examples at a point they have in common, so a few seeds grow into many variants.
• I2. Each rule carries a running success ratio, and it is retired only after a minimum number of uses with a low score. This avoids judging a rule from too little evidence.
• I3. Templates and lexical rules are pulled out by diffing pairs of examples. The shared part becomes a frame with variables, and the differing part becomes a word-level mapping.
• I4. A generated example that is wrong as a whole can still contain a correct local piece, and that piece is kept as a rule.
• I5. Because the system learns online, it is evaluated under several data orderings and with learning curves, not with a single final score.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (learn): Swapping aligned subtrees between silver sentences that share a head or frame slot could cheaply multiply training data for the parser and frame layer.
• I2 (learn): Self-induced repair rules could follow the same policy, being dropped only when repeated applications show a poor hit rate against witness agreement.
• I3 (back): Diffing meaning graphs of paired sentences could yield generation templates with slots, making each text realization traceable to the examples behind it.
• I4 (learn): Partially wrong silver parses could be mined for locally consistent arcs instead of being thrown out whole.
• I5 (learn): Mova's incremental training evaluations should report results across shuffled curriculum orders, so that gains caused by a lucky ordering do not pass as real improvements.