Elamur Research⌕
☰
home / papers / 2019 / BERT-Based Arabic Social Media Author Profiling

BERT-Based Arabic Social Media Author Profiling

Paper · 2019 · Chiyu Zhang and Muhammad Abdul-Mageed

Summary

The paper describes a system for the APDA shared task on Arabic author profiling. Given 100 tweets from each user, it predicts three attributes: age group (3 classes), country-level dialect (15 classes), and gender (binary). The authors fine-tune multilingual BERT separately for each attribute. Classification happens one tweet at a time, and the tweet labels are then combined into a single label per user. A one-layer GRU serves as the baseline.

For dialect and gender, the shared-task training data is enlarged with in-house labelled data. Age gets no augmentation because no external age-labelled data was available. The final submission takes a majority vote over models trained under different data conditions and achieved the top official scores, reaching 54.72% on age, 93.75% on dialect, 81.67% on gender, and 40.97% in joint accuracy.

The work is useful mainly as a recipe. It shows that a pretrained encoder, extra labelled data, confidence-filtered aggregation from posts to users, and simple ensembling are enough to build a strong profiling system, without any feature engineering.

Key points

  • Data: The APDA training set covers 2,250 users with 100 tweets each, 225,000 tweets in total. The authors split it 90/10 into TRAIN and DEV, holding out 225 users for DEV. The official test set has 720 users.
  • Tweet-level DEV accuracy for age / dialect / gender:
    • GRU baseline: 42.48 / 37.50 / 57.81.
    • BERT-Base Multilingual Cased: 45.64 / 42.35 / 60.26.
    • BERT EXT (with augmented data): 44.53 for dialect and 61.01 for gender.
  • Augmentation data:
    • EXTENDED Gender adds 1,100 hand-labelled users, balanced 550 female and 550 male, contributing about 163K tweets.
    • EXTENDED Dialect adds up to 20,000 tweets per class from in-house gold data, about 299K tweets in total.
  • User-level DEV accuracy after aggregation:
    • BERT: 55.56 for age, 96.00 for dialect, 80.00 for gender.
    • BERT EXT: 95.56 for dialect, 84.00 for gender.
  • Official test results:
    • Submission 1 (BERT EXT): 38.75 joint accuracy.
    • Submission 2 (BERT EXT+DEV, which retrains with DEV included): 40.97 joint.
    • Submission 3 (majority vote): best results overall, with 93.75 for dialect and 40.97 joint.

Interesting ideas

  • I1. Confidence-gated aggregation: Each unit (a tweet) gets its own prediction, and only predictions whose softmax confidence clears a threshold vote on the group (user) label; the threshold is swept from 0.00 to 0.99 on DEV, which drops noisy, uninformative items without any extra model.
  • I2. Voting across data conditions: Models trained on different data mixes (base, augmented, augmented plus DEV) make partly independent errors, so a simple majority vote over them beat each single model on the official test.
  • I3. Balanced, capped augmentation: Extra dialect data was sampled with a fixed per-class cap, and the gender set was built exactly balanced, so the added data does not tilt the label distribution.
  • I4. Folding DEV back in: Once settings are fixed on DEV, retraining with DEV included gave the final models more data, which raised the gender result and joint accuracy on test.
  • I5. Measuring against a weak baseline: A small GRU anchors every reported gain, and augmentation is openly omitted for age because no suitable data existed.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (frames): document-level judgements such as irony or the dominant speech act could be built from sentence-level frame decisions, counting only confident sentences and tuning the cutoff on held-out data.
  • I2 (learn): witness parsers for silver data could be deliberately trained on different treebank mixtures, so their agreement carries more information than votes from near-identical models.
  • I3 (tree): when mixing additional or converted treebanks into gparse training, capping each genre or source keeps one large corpus from swamping the style of the target data.
  • I4 (learn): after hyperparameters are frozen, the held-out split can be merged back for the final model, while a separate untouched negative-control set is kept for honest evaluation.
  • I5 (learn): each new component's gain can be stated against a deliberately simple baseline, and any conditions that could not be run can be listed explicitly instead of being silently skipped.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksPsychology · 1,552 worksSocial Psychology · 306 works

🧭 Topics 1Research subjects and languages the work is about.Semitic languages · 2,908 works

Cite

Cite as Chiyu Zhang and Muhammad Abdul-Mageed. 2019. BERT-Based Arabic Social Media Author Profiling. arXiv:1909.04181.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1909.04181 · arXiv 1909.04181

@misc{zhang2019bert,
  title = {{BERT-Based Arabic Social Media Author Profiling}},
  author = {Chiyu Zhang and Muhammad Abdul-Mageed},
  year = {2019},
  doi = {10.48550/arxiv.1909.04181},
  eprint = {1909.04181},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1909.04181}
}

References

10 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. 1. Abdul-Mageed, M.: Subjectivity and sentiment analysis of Arabic as a morophologically-rich language. Ph.D. thesis, Indiana University (2015) 2. Abdul-Mageed, M.: Modeling arabic subjectivity and sentiment in lexical space. find
  2. Information Processing & Management (2017) 3. Abdul-Mageed, M., Alhuzali, H., Elaraby, M.: You tweet what you speak: A city-level dataset of arabic dialects. In: Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC-2018) (2018) 4. Alrifai, K., Rebdawi, G., Ghneim, N.: Arabic tweeps gender and dialect prediction. In: CLEF (Working Notes) (2017) 5. Argamon, S.E.: Register in computational language research. Register Studies 1(1), 100–135 (2019) 6. Bassiouney, R.: Arabic sociolinguistics.
  3. Edinburgh University Press (2009) 7. Bleidorn, W., Hopwood, C.J.: Using machine learning to advance personality assessment and theory. Personality and Social Psychology Review p. 1088868318772990 (2018) 8. Burger, J.D., Henderson, J., Kim, G., Zarrella, G.: Discriminating gender on twitter. In: Proceedings of the conference on empirical methods in natural language processing. pp. 1301–1309. Association for Computational Linguistics (2011) 9. Chen, J., Cheng, L., Yang, X., Liang, J., Quan, B., Li, S.: Joint learning with both classification and regression models for age prediction. In: Journal of Physics: Conference Series. vol. 1168, p. 032016.
  4. IOP Publishing (2019) 10. Colleoni, E., Rozza, A., Arvidsson, A.: Echo chamber or public sphere? predicting political orientation and measuring political homophily in twitter using big data. Journal of communication 64(2), 317–332 (2014) 11. Daneshvar, S., Inkpen, D.: Gender identification in twitter using n-grams and lsa. In: Proceedings of the Ninth International Conference of the CLEF Association (CLEF 2018) (2018) 12. Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 (2018) 13. Elaraby, M., Abdul-Mageed, M.: Deep models for arabic dialect identification on benchmarked data. In: Proceedings of the Fifth Workshop on NLP for Similar Languages, Varieties and Dialects (VarDial 2018). pp. 263–274 (2018) 14. Flekova, L., Preoţiuc-Pietro, D., Ungar, L.: Exploring stylistic variation with age and income on twitter. In: Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers). vol. 2, pp. 313–319 (2016) 15. Gamon, M.: Linguistic correlates of style: authorship classification with deep linguistic analysis features. In: Proceedings of the 20th international conference on Computational Linguistics. p. 611. Association for Computational Linguistics (2004) 16. Goswami, S., Sarkar, S., Rustagi, M.: Stylometric analysis of bloggers age and gender. In: Third international AAAI conference on weblogs and social media (2009) 17. Habash, N.Y.: Introduction to arabic natural language processing. Synthesis Lectures on Human Language Technologies 3(1), 1–187 (2010) 18. Hinds, J., Joinson, A.: Human and computer personality prediction from digital footprints. Current Directions in Psychological Science p. 0963721419827849 (2019) BERT-Based Arabic Social Media Author Profiling 19. Holes, C.: Modern Arabic: Structures, functions, and varieties. ↗
  5. Georgetown University Press (2004) 20. Holmes, D.I.: The evolution of stylometry in humanities scholarship. Literary and linguistic computing 13(3), 111–117 (1998) 21. Johnson, K., Goldwasser, D.: Classification of moral foundations in microblog political discourse. In: Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). pp. 720–730 (2018) 22. Kingma, D., Ba, J.: Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014) 23. Matz, S.C., Kosinski, M., Nave, G., Stillwell, D.J.: Psychological targeting as an effective approach to digital mass persuasion. Proceedings of the national academy of sciences 114(48), 12714–12719 (2017) 24. Mitrou, L., Kandias, M., Stavrou, V., Gritzalis, D.: Social media profiling: A panopticon or omniopticon tool? In: Proc. of the 6th Conference of the Surveillance Studies Network. ↗
  6. Barcelona, Spain (2014) 25. Mubarak, H., Darwish, K.: Using twitter to collect a multi-dialectal corpus of arabic. In: Proceedings of the EMNLP 2014 Workshop on Arabic Natural Language Processing (ANLP). pp. 1–7 (2014) 26. Nguyen, D., Smith, N.A., Rosé, C.P.: Author age prediction from text using linear regression. In: Proceedings of the 5th ACL-HLT Workshop on Language Technology for Cultural Heritage, Social Sciences, and Humanities. pp. 115–123. Association for Computational Linguistics (2011) 27. Palva, H.: Dialects: classification. Encyclopedia of Arabic Language and Linguistics 1, 604–613 (2006) 28. Pang, D., Eichstaedt, J.C., Buffone, A., Slaff, B., Ruch, W., Ungar, L.H.: The language of character strengths: Predicting morally valued traits on social media. Journal of personality (2019) 29. Potthast, M., Rosso, P., Stamatatos, E., Stein, B.: A decade of shared tasks in digital text forensics at pan. In: European Conference on Information Retrieval. pp. 291–300.
  7. Springer (2019) 30. Preoţiuc-Pietro, D., Liu, Y., Hopkins, D., Ungar, L.: Beyond binary labels: political ideology prediction of twitter users. In: Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). pp. 729–740 (2017) 31. Qwaider, C., Saad, M., Chatzikyriakidis, S., Dobnik, S.: Shami: A corpus of levantine arabic dialects. In: Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC-2018) (2018) 32. Rangel, F., Rosso, P., Charfi, A., Zaghouani, W., Ghanem, B., Snchez-Junquera, J.: Overview of the track on author profiling and deception detection in arabic. In: Mehta P., Rosso P., Majumder P., Mitra M. (Eds.) Working Notes of the Forum for Information Retrieval Evaluation (FIRE 2019). CEUR Workshop Proceedings. In: CEUR-WS.org, Kolkata, India, December 12-15 (2019) 33. Rangel, F., Rosso, P., Chugur, I., Potthast, M., Trenkmann, M., Stein, B., Verhoeven, B., Daelemans, W.: Overview of the 2nd author profiling task at pan 2014. In: CLEF 2014 Evaluation Labs and Workshop Working Notes Papers, Sheffield, UK, 2014. pp. 1–30 (2014) 34. Rangel, F., Rosso, P., Koppel, M., Stamatatos, E., Inches, G.: Overview of the author profiling task at pan 2013. In: CLEF Conference on Multilingual and Multimodal Information Access Evaluation. pp. 352–365.
  8. CELCT (2013) Zhang & Abdul-Mageed 35. Rangel, F., Rosso, P., Potthast, M., Stein, B.: Overview of the 5th author profiling task at pan 2017: Gender and language variety identification in twitter. Working Notes Papers of the CLEF (2017) 36. Rangel, F., Rosso, P., Verhoeven, B., Daelemans, W., Potthast, M., Stein, B.: Overview of the 4th author profiling task at pan 2016: cross-genre evaluations. In: Working Notes Papers of the CLEF 2016 Evaluation Labs. CEUR Workshop Proceedings/Balog, Krisztian [edit.]; et al. pp. 750–784 (2016) 37. Rangel Pardo, F.M., Celli, F., Rosso, P., Potthast, M., Stein, B., Daelemans, W.: Overview of the 3rd author profiling task at pan 2015. In: CLEF 2015 Evaluation Labs and Workshop Working Notes Papers. pp. 1–8 (2015) 38. Rao, D., Yarowsky, D., Shreevats, A., Gupta, M.: Classifying latent user attributes in twitter. In: Proceedings of the 2nd international workshop on Search and mining user-generated contents. pp. 37–44. find
  9. ACM (2010) 39. Sadat, F., Kazemi, F., Farzindar, A.: Automatic identification of arabic language varieties and dialects in social media. Proceedings of SocialNLP p. 22 (2014) 40. Salameh, M., Bouamor, H.: Fine-grained arabic dialect identification. In: Proceedings of the 27th International Conference on Computational Linguistics. pp. 1332– 1344 (2018) 41. Sap, M., Park, G., Eichstaedt, J., Kern, M., Stillwell, D., Kosinski, M., Ungar, L., Schwartz, H.A.: Developing age and gender predictive lexica over social media. In: Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP). pp. 1146–1151 (2014) 42. Schwartz, H.A., Eichstaedt, J.C., Kern, M.L., Dziurzynski, L., Ramones, S.M., Agrawal, M., Shah, A., Kosinski, M., Stillwell, D., Seligman, M.E., et al.: Personality, gender, and age in the language of social media: The open-vocabulary approach. PloS one 8(9), e73791 (2013) 43. Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I., Salakhutdinov, R.: Dropout: a simple way to prevent neural networks from overfitting. The Journal of Machine Learning Research 15(1), 1929–1958 (2014) 44. Verhoeven, B.: Two authors walk into a bar: studies in author profiling. Ph.D. thesis, University of Antwerp (2018) 45. Verhoeven, B., Daelemans, W., Plank, B.: Twisty: a multilingual twitter stylometry corpus for gender and personality profiling. In: Proceedings of the 10th Annual Conference on Language Resources and Evaluation (LREC 2016)/Calzolari, Nicoletta [edit.]; et al. pp. 1–6 (2016) 46. Versteegh, K.: The arabic language.
  10. Edinburgh University Press (2014) 47. Zhang, C., Abdul-Mageed, M.: No army, no navy: Bert semi-supervised learning of arabic dialects. In: Proceedings of the Fourth Arabic Natural Language Processing Workshop. pp. 279–284 (2019)

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zhang-2019-bert-based-arabic-social-media/