📄 YNU-HPCC at IJCNLP-2017 Task 4: Attention-based Bi-directional GRU Model for Customer Feedback Analysis Task of English

Summary

The paper presents the YNU-HPCC system for IJCNLP-2017 Task 4, which assigns English customer feedback sentences to six categories: comment, complaint, request, bug, meaningless, and undetermined. The authors treat it as short-text classification. They compare bag-of-n-gram baselines (one-vs-all linear SVM and multinomial Naïve Bayes) with a neural model, Bi-GRUATT. This model encodes a padded sentence with a bidirectional GRU, collapses the hidden states into one vector using learned attention weights, and classifies it with a softmax layer. The paper's main experimental variable is the word representation: word2vec trained on review text the authors crawled themselves, the public GloVe vectors, or both concatenated. A sample-weighting option compensates for heavy class imbalance. The concatenated embeddings give the strongest results, reaching 71.0% accuracy and up to 49.9 macro-F1, compared with 63.0% accuracy for the SVM. The work shows that, on a small and skewed dataset, the choice of embedding and the handling of imbalance matter at least as much as the architecture.

Key points

• Data

  • English portion only: 3,065 training, 500 dev, and 500 test sentences.
  • Classes are very skewed. Comment has 1,758 training instances, while undetermined has only 22 (about 0.7%).
  • Sentences with several tags were split into separate copies, one per label.

• Embeddings

  • The authors trained their own word2vec on roughly 26.1M tokens of crawled Booking and Amazon app-store reviews.
  • They also used the public glove.840B.300d vectors.
  • The combined variant concatenates both into a 600-dimensional vector.
  • Unknown words and padding are zero vectors, and the input length is fixed at L = 117 tokens.

• Setup

  • Keras with a TensorFlow backend and Unitok tokenization.
  • Regularization: recurrent dropout of 0.2, plus dropout of 0.3 and 0.5 before the attention and output layers.
  • Training: rmsprop with batch size 32, and early stopping at around 20 epochs.

• Main results (Acc / micro-F1 / macro-F1)

  Model	Acc	micro-F1	macro-F1
  SVM	63.0	65.4	40.3
  Naïve Bayes	62.6	64.8	36.1
  Bi-GRUATT, word2vec, weighted	65.0	67.3	44.1
  Bi-GRUATT, GloVe, unweighted	71.0	73.0	47.0
  Bi-GRUATT, word2vec+GloVe, unweighted	71.0	73.2	48.8
  Bi-GRUATT, word2vec+GloVe, weighted	68.6	70.7	49.9

  For the baselines, combining unigrams and bigrams beat using either one alone.

• Per-class F1 (weighted model, word2vec+GloVe)

  • Comment reaches 80.5, but bug reaches only 38.5.
  • No F1 score could be computed for undetermined.

Interesting ideas

• I1. Class-balancing sample weights consistently traded micro-F1 for macro-F1, so whether to use them depends on whether rare classes or overall accuracy matter more; reporting both metrics exposes this trade-off.
• I2. Concatenating a small in-domain embedding with a large general-purpose one beat either alone, which suggests the two sources carry complementary signals.
• I3. Multi-label sentences were turned into several single-label training copies with identical text, which is a cheap way to reuse a single-label learner.
• I4. Attention pooling replaces averaging or last-state reduction with learned per-token weights, so the sentence vector shows which tokens drove the decision.
• I5. The per-class breakdown showed that some labels (bug, request) stay weak whatever the embedding, which points to data scarcity rather than representation as the bottleneck.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (learn): Evaluations of rare frame or speech-act labels should always report macro next to micro scores, so that gains on frequent classes do not hide failures on rare ones.
• I2 (tokens): Lexical features in crate en could combine statistics from a domain corpus (school texts, fables) with general-corpus statistics instead of choosing one source.
• I3 (frames): When one utterance carries several speech acts, the prag classifier could train on duplicated examples and still output a set of labels at inference.
• I4 (answer): A reader that scores candidate answers could expose per-token weights as a partial explanation, while the UD tree stays the primary justification.
• I5 (learn): Per-label error tables could decide where to spend LLM-teacher or silver-data budget, targeting categories that representation changes do not fix.