📄 Improving “Email Speech Acts” Analysis via N-gram Selection

Summary

Carvalho and Cohen revisit the task of tagging work email with "email speech acts", the intents a message carries, such as asking for something, promising to do something, proposing a meeting or sending data. An earlier study (Cohen et al., 2004) used only single words as features. This paper argues that short word sequences capture intent much better.

The authors used 1716 hand-labeled messages from five student teams in the CSpace corpus. They cleaned the messages and replaced numbers, times, days, pronouns, wh-words and file extensions with class symbols. They then extracted every n-gram of length one to five and ranked the n-grams per act with Information Gain. Classification used a linear SVM under 5-fold cross-validation.

Combining unigrams, bigrams and trigrams with this normalization cut error by 26.4% on average relative to the 2004 baseline. Adding 4-grams and 5-grams brought no further gain. The top-ranked n-grams closely matched linguistic intuition about each act. The authors also release Ciranda, an open-source toolkit for predicting email speech acts.

Key points

• Data: about 15,000 CSpace emails came from a 14-week CMU management course in which 277 MBA students in roughly 50 teams ran simulated companies.
  • Five teams (1F3, 2F2, 3F2, 4F4, 11F1) supplied 1716 labeled messages.
  • Earlier annotator agreement reached Kappa 0.72–0.85.
• Labels: each message is tagged with verb–noun pairs, so one email can carry several acts.
  • The verbs are Request, Propose, Commit and Deliver.
  • The nouns are Meeting and dData.
• Features: n-grams of length 1–5 gave more than 347,500 features.
  • The classifier was LIBSVM with default settings.
• Per-act results: relative error reductions over the 2004 baseline were:
  • Request 24.7%
  • Commit 33.3%
  • Deliver 23.7%
  • Propose 38.3%
  • Meeting 9.2%
  • dData 29.1%
• Ablation findings:
  • More labeled data (1716 vs 1354 messages) lowered unigram error.
  • Normalization alone barely helped unigrams.
  • The big jump came from adding 2- and 3-grams.
  • 4- and 5-grams did not help, which the authors attribute to too little labeled data.

Interesting ideas

• I1. Abstraction can be tuned per target: for Commit only, the generic person symbol leaves out the first-person "I", because sender-self phrases like "I will" are the strongest cue for a promise.
• I2. Replacing dates, clock times, weekdays, numbers and file types with class symbols lets rare surface strings pool into general patterns such as "[day] at [hour] [pm]".
• I3. Each act's top-ranked features are shown next to linguistic intuition, which serves as a sanity check that the model learned meaningful cues.
• I4. The experiment is an ablation ladder that adds one factor per step (more data, then normalization, then longer n-grams), so each gain can be attributed to its cause.
• I5. Intents are encoded as compositional verb–noun pairs (an act plus its object type), and one message may hold several of them.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (frames): the speech-act detector in prag should treat the speaker's own agency (a first-person subject with a future or modal verb) as a separate commitment signal rather than merging all persons.
• I2 (tokens): the tagger could attach normalized class labels to time, date and number tokens, so later stages and rule induction can generalize over them without memorizing literal values.
• I3 (learn): self-induced repair rules or learned cues could be ranked and audited for plausibility, and an implausible top cue would flag a data artifact.
• I4 (learn): evaluations of new training data, preprocessing or features should change one factor at a time, so that improvements stay explainable.
• I5 (frames): speech-act frames could combine an act type with a typed object slot (meeting, data, task) and allow multiple acts per utterance, linking them to entities in the world stage.