📄 Style Transfer as Bias Mitigation: Diffusion Models for Synthetic Mental Health Text for Arabic

Summary

Mental health text corpora from social media often under-represent some demographic groups. The authors address this for gender in CARMA, an Arabic mental health corpus in which posts by men outnumber posts by women about three to one. They do not sample new posts from a large language model. Instead, they treat augmentation as rewriting existing male posts in a female voice. This keeps the diversity of the real data and avoids biases carried in from web-scale pretraining.

With advice from native speakers, they used gpt-4o to build five paired datasets of male source posts and female-voiced rewrites. The datasets range from purely grammatical gender agreement to stronger emotional expression. They then trained a separate DiffuSeq-style diffusion model, without pretraining, on each dataset.

On held-out data, the outputs keep the meaning of the source (BERTScore F1 about 0.93–0.95) while changing much of the wording (ROUGE below 0.07). The authors present this as a flexible template for reducing bias in sensitive, low-resource domains.

Key points

• Corpus. CARMA, an Arabic social-media corpus labelled through self-reported diagnoses, with six conditions plus controls. The selected subsets are:
  • control: 4,086 users and about 270K posts;
  • depression: 171 users and about 21K posts.
• Five transfer datasets, each with 2,265 pairs split 90/5/5:
  • D1: pronoun and gender-agreement switching;
  • D2: shift in adjective and degree-modifier preference;
  • D3: more expressive first-person framing;
  • D4: politeness and hedging;
  • D5: amplified emotion and affect.
• Target generation. Targets were produced by gpt-4o at temperature 0 through the batch API. The diffusion models themselves start from no pretrained weights and use an AraBERT-compatible tokenizer.
• Training objective. Gaussian noise is added to the target embeddings under a linear schedule. The noisy targets are concatenated with the source embeddings, and a transformer is trained with an ℓ2 loss to reconstruct the clean targets.
• Results.
  • BLEU ranges widely, from 9.00 to 56.99. The highest value belongs to the pronoun-only setting, which is the most conservative rewrite.
  • ROUGE-2 is near zero for most runs.
  • An LLM-based gender classifier is reported, only qualitatively, to read the outputs as female-styled.

Interesting ideas

• I1. Organise rewrite tasks as a graded ladder, from changing only morphological agreement up to pragmatic and affective shifts, so each level isolates one linguistic layer of the change. — For Mova (back): Generation from the meaning graph could be tested with graded rewrites (agreement only, then hedging, then affect) to check that each layer can be changed without disturbing the others.
• I2. Report a semantic-similarity score next to a surface-overlap score, so that "meaning kept, wording changed" can be told apart from trivial copying or topic drift. — For Mova (learn): The text → graph → text compressor could be scored on both axes, with an identity-copy baseline as a negative control that should score high on overlap but show no real transformation.
• I3. Condition every synthetic example on a real item instead of sampling freely, which keeps the diversity and distribution of the original data rather than collapsing toward a generator's favourite outputs. — For Mova (learn): Silver training data could be built by perturbing real treebank sentences (gender, tense, politeness) rather than generating sentences from scratch, which keeps the natural syntactic variety.
• I4. Hedged and unhedged versions of the same claim (for example, "this decision is wrong" versus "this decision may be wrong") form minimal pairs that differ in strength of assertion but not in stance. — For Mova (frames): Such pairs could serve as test cases for the speech-act and implicature module, checking that it lowers speaker commitment while keeping who-claims-what intact.
• I5. Use a strong LLM only to create paired supervision, then train a small model from scratch on those pairs, so the deployed transformer carries no pretrained weights. — For Mova (learn): The LLM teacher tiers could produce paired examples of repairs or conversions, from which compact, inspectable rules or models are learned that do not depend on the teacher at run time.