📄 Annotating Clinical Risk and Variation in Haitian Creole Medical Translation

Summary

Haitian Creole (HC) is badly underserved by medical machine translation. Existing systems also ignore its internal variation, which runs from French-influenced mesolectal speech to basilectal, Creole-exclusive speech. As a result, a translation can be literally right yet confusing, stigmatizing, or clinically risky for monolingual patients, for example in Miami's diaspora.

The authors cleaned and normalized the CMU Haitian Creole medical corpus and designed a compact annotation schema for small expert teams. The schema has:

• binary fields for acceptability, severity of possible misunderstanding, and foreign-influence cues;
• error tags that apply only when quality is flagged and are compatible with MQM;
• a field for recording variety-specific variants.

After three annotation-and-adjudication rounds, they released a gold set of EN→HC sentence pairs. They then tested how learnable the labels are with a simple, interpretable classifier–labeller.

Acceptability and severity proved reliable for both annotators and the model. Foreign influence stayed hard to learn because it is rare in the data. The labels can already support triage of patient-facing translations, and they could later serve as reward signals for preference-based MT and LLM tuning.

Key points

• The gold dataset contains 152 adjudicated EN→HC medical sentence pairs, built from a normalized version of the CMU Haitian Creole medical corpus. Inter-annotator agreement stabilized over three rounds.
• The baseline used TF–IDF and simple text-shape features with five-fold cross-validation. It outputs per-field probabilities and suggests error tags.
• Severity scores worked well for ranking: checking just the items the model rated most likely to be severe caught much of the genuinely high-risk material.
• Foreign influence was the weakest dimension (F1 = 0.356 ± 0.216 across folds; label prevalence 11.8%). The authors attribute this to limited data rather than a flaw in the schema.
• Qualitative analysis found three recurring error types:
  • lexical misselection, such as stigmatizing tafyatè;
  • morphosyntactic interference from English or French;
  • register or variety mismatches.