📄 Participation d'EDF R&D à DEFT 2018 (Here the title in English)

Summary

EDF R&D entered the DEFT 2018 challenge on French tweets about public transport in Île-de-France. They took part in two tasks: telling transport tweets apart from unrelated ones (T1), and assigning an overall polarity to transport tweets (T2: positive, negative, neutral or mixed). They used only the official "simple" corpus of about 54,000 tweets, split 80/20 for training and validation, with no outside data.

They compared three approaches:

• a Word2Vec model trained with category labels inserted as pseudo-words, from which similarity-based features feed WEKA classifiers;
• LSTM networks;
• CNN networks.

For the LSTM and CNN runs, tweets were first enriched with polarity tags from small hand-built lexicons. Every submitted run scored above the campaign mean. On T1 the Word2Vec pipeline was narrowly best. On T2 the LSTM led with F≈0.80. The team placed 5th of 11 on T1 and 4th of 12 on T2, the top non-academic entrant. The authors see the methods as easy to reuse for other short-text classification needs, such as separating EDF the utility from EDF the national football team.

Key points

• Data handling: The ~14,000-tweet "batch_b" corpus was discarded because every item carried the label "inconnu". Final predictions were made on 7,816 evaluation tweets.

• Word2Vec setup: skip-gram, window of 5, 200 dimensions, hierarchical softmax, minimum word frequency 30, 1000 iterations, plus a stop-list. Similarity statistics gave 14 features for T1 and 44 for T2. The features were fed to Logistic Regression and Random Forest.

• Polarity lexicons: 121 positive and 291 negative entries, emoticons included. The tags they injected improved training accuracy by roughly 5%.

• LSTM settings:

  • T1: embedding 150, padding 40, hidden size 60.
  • T2: embedding 300, padding 40, hidden size 100, with 20% dropout.

• CNN settings: embedding 300, filter and pooling windows of 5 words.

• T2 F-measures:

  Run	F-measure
  LSTM	0.80249
  CNN	0.80067
  LSTM with reduced vocabulary	0.79957
  Word2Vec + Logistic Regression	0.7435
  Word2Vec + Random Forest	0.73969
  Campaign mean	0.7293
  Campaign max	0.82288

• T1 result: the Word2Vec run scored 0.90124, against a campaign mean of 0.88813 and a max of 0.90785.

Interesting ideas

• I1. Wrapping each training sentence in its class label as an extra pseudo-token puts labels and words in one vector space, so similarity to a label becomes a cheap category signal.
• I2. Noisy text is normalised by collapsing repeated letters and replacing URLs, dates, clock times, durations and emoticons with typed placeholder tokens, which shrinks the vocabulary and absorbs many spelling variants.
• I3. To detect mixed polarity, the authors use the largest and smallest jumps in label similarity between successive words, treating a sharp internal contrast as evidence of mixed sentiment.
• I4. Lexicon matches are written straight into the text as inline tags, so a learned model receives symbolic hints without any architectural change, and this alone gave a measurable gain.
• I5. Each hyperparameter configuration was trained ten times, and both mean validation accuracy and mean loss were compared, rather than trusting a single lucky run.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (world): candidate verb-class or frame memberships for rare verbs could be proposed this way and then kept only after explicit checks against VerbNet/WordNet entries.
• I2 (tokens): typed placeholders like DATE/DURATION could be emitted with a recorded original span, giving the tagger stable tokens while keeping the measure value for later stages.
• I3 (frames): a polarity-contrast signal across clauses could serve as one explainable cue for irony or concessive speech acts.
• I4 (learn): explicit lexicon or rule annotations could be injected as marker tokens into gparse training input, with a negative-control run without them to measure their real effect.
• I5 (learn): repeated seeded runs with averaged metrics would make parser or reader comparisons less noisy before a configuration is adopted.