Elamur Research⌕
☰
home / papers / 2019 / Gender Prediction from Tweets: Improving Neural Representations with Hand-Crafted Features

Gender Prediction from Tweets: Improving Neural Representations with Hand-Crafted Features

Paper · 2019 · Erhan Sezerer, Ozan Polatbilek and Selma Tekir

Summary

The paper tackles author profiling, specifically guessing a Twitter user's gender from a batch of their posts. The authors build RNNwA, a bidirectional GRU network over GloVe Twitter embeddings. It applies attention twice: once over the words inside each tweet, and once over the user's tweets to merge them into a single user vector. They then add classic sparse features. Word and character n-grams are tf-idf weighted, compressed with LSA, and appended to the neural user vector before the final classifier. This variant is called RNNwA + n-gram.

Experiments use the PAN 2018 author profiling data in English, Spanish and Arabic. The combined model sets a new best English accuracy and comes close to the strongest SVM system on the other two languages. It also beats the earlier top neural approach, which used max pooling. The work suggests that hand-engineered signals still add value on top of learned representations, and that learned weighting of tweets works better than plain averaging. A side finding is that tuning for per-tweet accuracy can hurt per-user accuracy.

Key points

  • Data: PAN 2018, with 100 tweets per user.
    • English: 3000 train / 1900 test users.
    • Spanish: 3000 train / 2200 test users.
    • Arabic: 1500 train / 1000 test users.
    • Training data was split 80/20 into train and validation sets for tuning.
  • Model setup:
    • 200-dimensional GloVe Twitter embeddings.
    • BiGRU with 150 cells for English and 100 for Spanish and Arabic.
    • Bahdanau-style attention at both word and tweet level.
    • Adam optimiser, cross-entropy loss and L2 regularisation.
  • Hand-crafted features:
    • Character 3–5-grams.
    • Word 1–2-grams (1–3 for English).
    • Items seen fewer than twice are dropped; the rest are tf-idf weighted and LSA-reduced to 300 dimensions.
  • Test accuracy of RNNwA + n-gram (Daneshvar & Inkpen SVM in brackets):
    • English: 82.31 (81.52).
    • Spanish: 80.22 (82.00).
    • Arabic: 80.50 (80.90).
    • Plain RNNwA scored 81.79, 78.23 and 78.50.
    • Takahashi et al. scored 79.68, 78.64 and 77.10.
  • Attention versus averaging: Replacing averaged per-tweet predictions with tweet-level attention raised user accuracy, for example from 79.3 to 81.8 for the English RNN. The one exception was the CNN on Arabic.
  • Tuning target matters: Configurations that were best at tweet-level accuracy lost about 4% at user level.
  • Reproducibility: The authors could not reproduce the SVM baseline's reported English score of 82.21 with its released code.

Interesting ideas

  • I1. Appending a compact, dimension-reduced block of engineered features to a learned vector, right before the output layer, cheaply injects signals the network may miss.
  • I2. Merging many sub-units with learned importance weights, rather than averaging their predictions, lets a few decisive pieces dominate, and the weights themselves show which pieces mattered.
  • I3. The level at which you tune need not match the level you care about: the best model on individual items was worse on the aggregate.
  • I4. Slight overfitting on individual items acted like implicit attention, over-trusting a few distinctive examples, which suggests aggregate metrics can hide odd per-item behaviour.
  • I5. Re-running a competitor's released code, and openly flagging when its claimed number cannot be reached, gives fairer comparison tables.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (tokens): Character n-gram or suffix features could be concatenated to the tagger's or lemmatizer's learned word vectors to help with rare and unknown words, while each feature stays inspectable.
  • I2 (answer): A reader answering questions about a fable could weight candidate supporting sentences this way and expose the weights as part of its explanation.
  • I3 (learn): Choose parser or frame-model checkpoints by downstream answer or world-building accuracy, not only by per-token LAS, and log both so the mismatch is visible.
  • I4 (learn): Evaluation should report per-item and per-document scores side by side, so that gains coming from overconfident shortcuts can be spotted and checked with negative controls.
  • I5 (learn): When benchmarking against external parsers or QA systems, rerun them on our splits and record both the reported and the reproduced scores.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

📚 Shelves 6 Computer Science · 9,086 works Artificial Intelligence · 8,889 works Arts and Humanities · 9,296 works Language and Linguistics · 9,206 works Psychology · 798 works Social Psychology · 129 works

Cite

Cite as Erhan Sezerer, Ozan Polatbilek and Selma Tekir. 2019. Gender Prediction from Tweets: Improving Neural Representations with Hand-Crafted Features. arXiv:1908.09919.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1908.09919 · arXiv 1908.09919

@misc{sezerer2019gender,
  title = {{Gender Prediction from Tweets: Improving Neural Representations with Hand-Crafted Features}},
  author = {Erhan Sezerer, Ozan Polatbilek and Selma Tekir},
  year = {2019},
  doi = {10.48550/arxiv.1908.09919},
  eprint = {1908.09919},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1908.09919}
}

References

21 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Alowibdi, J. S., U. A. Buy, and P. Yu (2013). Language independent gender classification on twitter. In Proceedings of the 2013 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining, ASONAM ’13, pp. 739–743. ACM. find
  2. Argamon, S., M. Koppel, J. W. Pennebaker, and J. Schler (2009, February). Automatically profiling the author of an anonymous text. Commun. ACM 52(2), 119–123. find
  3. Bahdanau, D., K. Cho, and Y. Bengio (2014). Neural machine translation by jointly learning to align and translate. In Proceedings of the 3rd International Conference on Learning Representations. ↗ our card
  4. Chung, J., Ç. Gülçehre, K. Cho, and Y. Bengio (2014). Empirical evaluation of gated recurrent neural networks on sequence modeling. CoRR. find
  5. Collobert, R. and J. Weston (2008). A unified architecture for natural language processing: deep neural networks with multitask learning. In ICML, Volume 307 of ACM International Conference Proceeding Series, pp. 160–167. ACM. find
  6. Daneshvar, S. and D. Inkpen (2018). Gender identification in twitter using n-grams and LSA: notebook for PAN at CLEF 2018. In Working Notes of CLEF 2018 - Conference and Labs of the Evaluation Forum, Avignon, France, September 10-14, 2018. find
  7. Deitrick, W., Z. Miller, B. Valyou, B. Dickinson, T. Munson, and W. Hu (2012, 01). Author gender prediction in an email stream using neural networks. Journal of Intelligent Learning Systems and Applications 04, 169–175. ↗
  8. Goldberg, Y. (2017). Neural Network Methods for Natural Language Processing. Synthesis Lectures on Human Language Technologies. Morgan & Claypool Publishers. ↗
  9. Kingma, D. P. and J. Ba (2014). Adam: A method for stochastic optimization. CoRR. ↗
  10. Kodiyan, D., F. Hardegger, S. Neuhaus, and M. Cieliebak (2017). Author profiling with bidirectional rnns using attention with grus. In CLEF. find
  11. Kucukyilmaz, T., B. B. Cambazoglu, C. Aykanat, and F. Can (2006). Chat mining for gender prediction. In T. Yakhno and E. J. Neuhold (Eds.), Advances in Information Systems, pp. 274–283. Springer Berlin Heidelberg. find
  12. Ljubešić, N., D. Fišer, and T. Erjavec (2017). Language-independent gender prediction on twitter. In Proceedings of the Second Workshop on NLP and Computational Social Science, pp. 1–6. Association for Computational Linguistics. find
  13. Miller, Z., B. Dickinson, and W. Hu (2012, 01). Gender prediction on twitter using stream algorithms with n-gram character features. International Journal of Intelligence Science 02, 143–148. ↗
  14. Pardo, F. M. R., P. Rosso, M. M. y Gómez, M. Potthast, and B. Stein (2018). Overview of the 6th author profiling task at pan 2018: Multimodal gender identification in twitter. In CLEF. find
  15. Pennington, J., R. Socher, and C. D. Manning (2014). Glove: Global vectors for word representation. In Empirical Methods in Natural Language Processing (EMNLP), pp. 1532–1543. our card
  16. Sayyadiharikandeh, M., G. L. Ciampaglia, and A. Flammini (2016, November). Cross-domain gender detection in twitter. In Proceedings of the Workshop on Computational Approaches to Social Modeling (ChASM 2016). find
  17. Sezerer, E., O. Polatbilek, O. Sevgili, and S. Tekir (2018). Gender prediction from tweets with convolutional neural networks. In Proceedings of the Ninth International Conference of the CLEF Association (CLEF 2018). find
  18. Sezerer, E., O. Polatbilek, and S. Tekir (2019, April). Gender prediction from turkish tweets with neural networks. In 2019 27th Signal Processing and Communications Applications Conference (SIU), pp. 1–4. find
  19. Sezerer, E., O. Polatbilek, and S. Tekir (2019, August). A Turkish dataset for gender identification of twitter users. In Proceedings of the 13th Linguistic Annotation Workshop, Florence, Italy, pp. 203–207. Association for Computational Linguistics. find
  20. Takahashi, T., T. Tahara, K. Nagatani, Y. Miura, T. Taniguchi, and T. Ohkuma (2018). Text and image synergy with feature cross technique for gender identification: Notebook for PAN at CLEF 2018. In Working Notes of CLEF 2018 - Conference and Labs of the Evaluation Forum, Avignon, France, September 10-14, 2018. find
  21. van der Goot, R., N. Ljubešić, I. Matroos, M. Nissim, and B. Plank (2018). Bleaching text: Abstract features for cross-lingual gender prediction. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pp. 383–389. Association for Computational Linguistics. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/sezerer-2019-gender-prediction-tweets-improving-neural/