Elamur Research⌕
☰
home / papers / 2023 / Can Authorship Representation Learning Capture Stylistic Features?

Can Authorship Representation Learning Capture Stylistic Features?

Paper · 2023 · Andrew Wang, Cristina Aggazzotti, Rebecca Kotula et al.

Summary

Neural models trained to tell authors apart are often assumed to learn writing style. However, high attribution accuracy could also come from topic or other content signals that happen to correlate with authorship. The authors test this assumption on the Universal Authorship Representation (UAR), which is trained with a supervised contrastive objective so that texts by the same author map to nearby vectors.

They evaluate author retrieval by MRR on Reddit comments, Amazon reviews and fanfic stories, and apply three interventions:

  • masking content words during training;
  • automatically paraphrasing texts, which changes how things are said while keeping the meaning;
  • transferring the embeddings to unseen topic and coarse style classification tasks.

SBERT serves as a content-oriented reference throughout. Hiding content words barely changes retrieval quality, while paraphrasing sharply degrades it, and far more than it degrades SBERT. UAR also transfers poorly to topic discrimination but better to style discrimination. Together, these results indicate that authorship embeddings are largely stylistic. This implies some robustness to topic drift over time and potential value for applications such as style transfer.

Key points

  • The model studied is UAR (Rivera-Soto et al., 2021), with an open-source implementation called LUAR (github.com/LLNL/LUAR). It is trained with a supervised contrastive loss (Khosla et al., 2020).
  • Experiments cover three domains: Reddit, Amazon and fanfic. Each query or target is an "episode" of 16 consecutive comments, reviews or story paragraphs from one author.
  • Retrieval is scored with mean reciprocal rank (MRR), which ranges from 0 to 1. All models were trained on a single GPU for 20 epochs.
  • Masking content words had no significant effect on ranking, while paraphrasing caused a drastic drop that was much larger than SBERT's. This points to UAR's sensitivity to style.
  • The authors acknowledge three limitations: only UAR instances are tested, the analysis covers aggregate rather than local features, and style and content may never be fully separable.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Johns Hopkins UniversityBaltimore · United States240 works · 96,487 citations🏛️United States Department of DefenseWashington · United States4 works · 1,907 citations🔬Lawrence Livermore National LaboratoryLivermore · United States5 works · 456 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksPsychology · 1,552 worksSocial Psychology · 306 works

Cite

Cite as Andrew Wang, Cristina Aggazzotti, Rebecca Kotula, Rafael Rivera Soto, Marcus Bishop and Nicholas Andrews. 2023. Can Authorship Representation Learning Capture Stylistic Features?. Transactions of the Association for Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.1162/tacl_a_00610

@article{wang2023can,
  title = {{Can Authorship Representation Learning Capture Stylistic Features?}},
  author = {Andrew Wang, Cristina Aggazzotti, Rebecca Kotula, Rafael Rivera Soto, Marcus Bishop and Nicholas Andrews},
  journal = {Transactions of the Association for Computational Linguistics},
  year = {2023},
  doi = {10.1162/tacl\_a\_00610},
  url = {https://doi.org/10.1162/tacl_a_00610}
}

References

31 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Nicholas Andrews and Marcus Bishop. 2019. Learning invariant representations of social media users. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 1684–1695. Ass… find
  2. Jason Baumgartner, Savvas Zannettou, Brian Keegan, Megan Squire, and Jeremy Blackburn. 2020. The Pushshift Reddit dataset. In Proceedings of the 14th International AAAI Conference on Web and Social Media (ICWSM), volume 14, pages 830–839. https://doi.org/10.1609/icwsm.v14i1.7347 — [doi:10.1609/icwsm.v14i1.7347](https://doi.org/10.1609/icwsm.v14i1.7347) ↗
  3. Janek Bevendorff, Bilal Ghanem, Anastasia Giachanou, Mike Kestemont, Enrique Manjavacas, Ilia Markov, Maximilian Mayerl, Martin Potthast, Francisco Rangel Pardo, Paolo Rosso, Guenther Specht, Efstathios Stamatatos, Benno Stein, Matti Wiegmann, and Eva Zangerle. 2020. Overview of PAN 2020: Authorship… find
  4. Benedikt Boenninghoff, Steffen Hessler, Dorothea Kolossa, and Robert M. Nickel. 2019. Explainable authorship verification in social media via attention-based similarity learning. In 2019 IEEE International Conference on Big Data (Big Data), pages 36–45. https://doi .org/10.1109/BigData47090.2019.900… — [doi:10.1109/bigdata47090.2019.9005650](https://doi.org/10.1109/bigdata47090.2019.9005650) find
  5. Yiming Cui, Wanxiang Che, Ting Liu, Bing Qin, and Ziqing Yang. 2021. Pre-training with whole word masking for Chinese BERT. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 29:3504–3514. https://doi.org/10.1109/TASLP.2021.3124365 ↗
  6. Bill Dolan and Chris Brockett. 2005. Automatically constructing a corpus of sentential paraphrases. In Third International Workshop on Paraphrasing (IWP2005). Asia Federation of Natural Language Processing.
  7. Julien Hay, Bich-Lien Doan, Fabrice Popineau, and Ouassim Ait Elhara. 2020. Representation learning of writing style. In Proceedings of the 6th Workshop on Noisy User-generated Text (W-NUT 2020), pages 232–243. Association for Computational Linguistics. https://doi.org/10.18653/v1/2020.wnut-1.30 ↗
  8. Javier Huertas-Tato, Alvaro Huertas-Garcia, Alejandro Martin, and David Camacho. 2022. PART: Pre-trained Authorship Representation Transformer. cs.CL/2209.15373v1. https://doi.org/10.48550/arXiv.2209.15373 — [doi:10.48550/arxiv.2209.15373](https://doi.org/10.48550/arxiv.2209.15373) ↗
  9. Somayeh Jafaritazehjani, Gwénolé Lecorvé, Damien Lolive, and John D. Kelleher. 2020. Style versus content: A distinction without a (learnable) difference? In Proceedings of the 28th International Conference on Computational Linguistics, pages 2169–2180. International Committee on Computational Li… find
  10. Di Jin, Zhijing Jin, Zhiting Hu, Olga Vechtomova, and Rada Mihalcea. 2022. Deep learning for text style transfer: A survey. Computational Linguistics, 48(1):155–205. https://doi.org/10.1162/coli a 00426 ↗
  11. Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. 2020. Supervised contrastive learning. In Proceedings of the 34th International Conference on Neural Information Processing Systems (NeurIPS). find
  12. Jaehyeon Kim, Jungil Kong, and Juhee Son. 2021. Conditional variational autoencoder with adversarial learning for end-to-end text-to-speech. In Proceedings of the 38th International Conference on Machine Learning, pages 5530–5540. find
  13. Kalpesh Krishna, John Wieting, and Mohit Iyyer. 2020. Reformulating unsupervised style transfer as paraphrase generation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 737–762. Association for Computational Linguistics. https://doi.org/10 .1…
  14. David D. Lewis. 1997. Reuters-21578 text categorization test collection, Distribution 1.0. AT&T Labs-Research. find
  15. Juncen Li, Robin Jia, He He, and Percy Liang. 2018. Delete, retrieve, generate: A simple approach to sentiment and style transfer. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)…
  16. Aman Madaan, Amrith Setlur, Tanmay Parekh, Barnabás Poczós, Graham Neubig, Yiming Yang, Ruslan Salakhutdinov, Alan W. Black, and Shrimai Prabhumoye. 2020. Politeness transfer: A tag and generate approach. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, p…
  17. Frederick Mosteller and David L. Wallace. 1964. Inference and Disputed Authorship: The Federalist. Addison-Wesley, Reading, MA. find
  18. Jianmo Ni, Jiacheng Li, and Julian McAuley. 2019. Justifying recommendations using distantly-labeled reviews and fine-grained aspects. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (…
  19. John Pavlopoulos, Jeffrey Sorensen, Lucas Dixon, Nithum Thain, and Ion Androutsopoulos. 2020. Toxicity detection: Does context really matter? In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4296–4305. Association for Computational Linguistics. https:…
  20. Danish Pruthi, Rachit Bansal, Bhuwan Dhingra, Livio Baldini Soares, Michael Collins, Zachary C. Lipton, Graham Neubig, and William W. Cohen. 2022. Evaluating explanations: How much do explanations from the teacher aid students? Transactions of the Association for Computational Linguistics, 10:359–37…
  21. Peng Qi, Yuhao Zhang, Yuhui Zhang, Jason Bolton, and Christopher D. Manning. 2020. Stanza: A Python natural language processing toolkit for many human languages. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 101–108. Associat…
  22. Sudha Rao and Joel Tetreault. 2018. Dear Sir or Madam, may I introduce the GYAFC dataset: Corpus, benchmarks and metrics for formality style transfer. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Vo…
  23. Rafael A. Rivera-Soto, Olivia Elizabeth Miano, Juanita Ordonez, Barry Y. Chen, Aleem Khan, Marcus Bishop, and Nicholas Andrews. 2021. Learning universal authorship representations. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 913–919. Association f… find
  24. Chakaveh Saedi and Mark Dras. 2021. Siamese networks for large-scale author identification. Computer Speech & Language, 70:101241. https://doi.org/10.1016/j.csl.2021 .101241 — [doi:10.1016/j.csl.2021](https://doi.org/10.1016/j.csl.2021) ↗
  25. Yunita Sari, Mark Stevenson, and Andreas Vlachos. 2018. Topic or style? Exploring the most useful features for authorship attribution. In Proceedings of the 27th International Conference on Computational Linguistics, pages 343–353. Association for Computational Linguistics. find
  26. Efstathios Stamatatos. 2018. Masking topic-related information to enhance authorship attribution. Journal of the Association for Information Science and Technology, 69(3):461–473. https://doi.org/10.1002/asi.23968 — [doi:10.1002/asi.23968](https://doi.org/10.1002/asi.23968) ↗
  27. Mukund Sundararajan, Ankur Taly, and Qiqi Yan. 2017. Axiomatic attribution for deep networks. In Proceedings of the 34th International Conference on Machine Learning (Volume 70), pages 3319–3328. find
  28. Anna Wegmann and Dong Nguyen. 2021. Does it capture STEL? A modular, similarity-based linguistic style evaluation framework. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 7109–7130. Association for Computational Linguistics. https://doi.org/10 .1865… find
  29. Anna Wegmann, Marijn Schraagen, and Dong Nguyen. 2022. Same author or just same topic? Towards content-independent style representations. In Proceedings of the 7th Workshop on Representation Learning for NLP, pages 249–268. Association for Computational Linguistics. https://doi.org/10 .18653/v1/2022… find
  30. John Wieting and Kevin Gimpel. 2018. ParaNMT-50M: Pushing the limits of paraphrastic sentence embeddings with millions of machine translations. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, Volume 1 (Long Papers), pages 451–462. Association for Computati…
  31. Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2019. BERTScore: Evaluating text generation with BERT. cs.CL/1904.09675v3. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/wang-2023-can-authorship-representation-learning-capture/