📄 Polite Chatbot: A Text Style Transfer Application

Summary

The paper tackles a practical gap in dialogue systems: making a chatbot answer politely while staying on topic, even though no parallel corpus pairs ordinary replies with polite ones. The authors use a three-stage pipeline. First, they finetune BART as a politeness transfer model. Its training pairs are synthetic: following Madaan et al. (2020), politeness markers found via TF-IDF are deleted from very polite Enron sentences, and the model learns to restore them. Unlike the earlier tagger-plus-generator design, this happens in one seq2seq step. Second, they run this model over DailyDialog utterances to build a corpus of contexts paired with polite next turns. Third, they finetune GPT-2, DialoGPT and BlenderBot on that corpus. The result is an end-to-end polite responder that needs no post-processing. Both automatic metrics (a BERT politeness classifier, BLEU, Sentence-BERT similarity) and a small human study show higher politeness than baselines, with only a modest loss in coherence to context. The value of the work is that a simple data-centric recipe can replace more complex multi-step or reinforcement-learning approaches.

Key points

• Politeness transfer results: on the Madaan et al. (2020) Enron test data, the single-step BART transfer model beats the two-module baseline on all three metrics.
  • Polite Score: 8.68 vs 7.01
  • BLEU: 71.65 vs 60.16
  • Content Similarity: 93.25 vs 87.86
• Synthetic DailyDialog corpus: the authors' transfer model lifts the Polite Score from 5.41 (source data) to 7.95, compared with 6.37 for Madaan et al. Content similarity to the source utterances stays high: BLEU 70.21, CS 89.07.
• Data used:
  • DailyDialog has 13,118 dialogues, split into 11,118 train, 1,000 validation and 1,000 test, with about 7.9 turns per dialogue.
  • The Enron data is sorted into ten politeness buckets, P0–P9, and only the most polite bucket (P9) is used to create training pairs.
• Politeness classifier: the BERT-based scorer reaches 83.27% accuracy. Its errors fall mostly into adjacent buckets, with an average error of 0.98 buckets.
• Human evaluation: 50 test contexts were rated on BlenderBot outputs using a 1–5 Likert scale.
  • Politeness: 4.26 for the authors' model, the highest of all systems; plain DailyDialog finetuning scored 3.90.
  • Coherence to context: 2.94, below plain DailyDialog finetuning at 3.74.
  • Fluency: 4.30.

Interesting ideas

• I1. Parallel training data for a style can be manufactured by deleting the phrases that statistically separate a styled text from a plain one and training a model to restore them; this needs no human-paired corpus.
• I2. Phrases whose mean TF-IDF is much higher in polite texts than in non-polite ones act as an automatically mined lexicon of politeness markers.
• I3. The desired style is baked into the training corpus up front, so the final model is end-to-end and needs no post-editing step at inference time.
• I4. Politeness is scored on ordered buckets, and errors are reported as the average distance between predicted and true bucket rather than raw accuracy alone, which shows that most mistakes are near misses.
• I5. Pairing lenient automatic metrics (BLEU-1/2, embedding similarity) with human ratings exposed a trade-off that the automatic scores hid: higher politeness came with lower coherence to context.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (back): the same delete-and-restore trick could produce training pairs for turning a bare, meaning-faithful generated sentence into a register-appropriate one, without hand-written parallel text.
• I2 (frames): such a mined marker list could feed the speech-act and pragmatics layer in crate prag as explainable cues for politeness or indirectness, rather than an opaque classifier.
• I3 (learn): when silver data is prepared, normalisations or repairs can be applied to the data once, rather than bolted on as runtime rules, keeping the deployed pipeline simpler.
• I4 (learn): graded judgments such as confidence levels or parse-quality tiers could be evaluated by distance-aware error, separating harmless near misses from real failures.
• I5 (answer): reader and QA evaluations should include a human or control check of relevance, so that gains in surface quality are not mistaken for better answers.