Elamur Research⌕
☰
home / papers / 2022 / POTATO: The Portable Text Annotation Tool

POTATO: The Portable Text Annotation Tool

Paper · 2022 · Jiaxin Pei, Aparna Ananthasubramaniam, Xingyao Wang et al.

Summary

NLP projects increasingly need annotations for intricate tasks. Existing tools such as Label Studio, Doccano, LightTag and Prodigy each have drawbacks: some depend on external hosting, some offer little layout flexibility for complex designs, and some cost too much for small teams. The authors present POTATO, a free, MIT-licensed, self-hosted annotation system written in Python and refined over two years. One YAML file defines the data sources, annotation schemes and server settings. The interface is generated from editable HTML/Jinja templates, so it can show text, images or anything else a browser can render. The tool includes productivity aids (active learning, keyword-triggered highlighting, keyboard shortcuts, label tooltips) and quality-control mechanisms (attention checks, qualification tests, pre- and post-study surveys, a time tracker). It also connects to Prolific and Amazon Mechanical Turk. In experiments on two annotation tasks, these features sped up labeling, most of all for long documents and complex tasks. This gives low-budget research groups a transparent, adaptable alternative to commercial tools.

Key points

  • Eight built-in scheme types: multi-select checkboxes, single-select radio buttons, best-worst scaling, Likert scale, free text, span labels, numeric input and dropdowns. The figure advertises 10+ ready-made templates for tasks such as sentiment analysis, QA and image labeling.
  • Data is loaded from delimited files or newline-delimited JSON, including the formats returned by the Twitter and Reddit APIs. Deployers only need to name the instance-ID field and, usually, the text field. Extra columns are kept in the output.
  • The default active learning model periodically retrains a logistic regression on unigram and bigram features. It puts low-confidence items first while still mixing in a configurable share of random items.
  • POTATO has been used in real projects, for example a 27-class immigration-framing scheme (Mendelsohn et al., 2021) and empathy/condolence ratings of long Reddit comments.
  • Each running server handles one task, but one installation can store many task configurations. Annotators can log in by email or through a URL parameter carrying their crowdsourcing ID, and they resume at their latest unlabeled item.

Interesting ideas

  • I1. Decoy highlights: when keyword highlighting is enabled, the tool also highlights some random words at a set rate. This keeps annotators from trusting highlights blindly, and the same trick could be used with any assistive cue, such as model pre-labels.
  • I2. Uncertainty sampling plus a random share: mixing a fixed fraction of random items into the queue alongside low-confidence ones guards against a biased, self-reinforcing sample. This is a cheap safeguard for any active-learning pipeline.
  • I3. Separating task definition from presentation: schemes are declared in YAML and rendered into user-editable Jinja templates. New tasks can then be launched quickly while layouts are customized independently, which is a useful pattern for internal labeling tools.
  • I4. Interactive config generation: if no YAML file is supplied, the tool asks the deployer a series of questions and writes the config itself. This lowers the barrier for non-technical project owners.
  • I5. Time tracking to support fair pay: recording time spent per annotator makes hourly rather than per-item payment possible. The authors also dropped a countdown alert they had piloted because it added stress, a reminder to test whether a feature harms annotator wellbeing.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of California, BerkeleyBerkeley · United States127 works · 212,523 citations🎓University of Illinois Urbana-ChampaignUrbana · United States108 works · 67,432 citations🎓University of MichiganAnn Arbor · United States36 works · 12,181 citations🎓University of Illinois SystemUrbana · United States2 works · 89 citations

Also on the byline: The American College of Greece.

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 9,086 worksArtificial Intelligence · 8,889 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 worksSocial Sciences · 744 worksGeneral Social Sciences · 87 works

Cite

Cite as Jiaxin Pei, Aparna Ananthasubramaniam, Xingyao Wang, Naitian Zhou, Apostolos Dedeloudis, Jackson Sargent and David Jurgens. 2022. POTATO: The Portable Text Annotation Tool. Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing: System Demonstrations.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2022.emnlp-demos.33

@inproceedings{pei2022potato,
  title = {{POTATO: The Portable Text Annotation Tool}},
  author = {Jiaxin Pei, Aparna Ananthasubramaniam, Xingyao Wang, Naitian Zhou, Apostolos Dedeloudis, Jackson Sargent and David Jurgens},
  booktitle = {Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing: System Demonstrations},
  year = {2022},
  doi = {10.18653/v1/2022.emnlp-demos.33},
  url = {https://doi.org/10.18653/v1/2022.emnlp-demos.33}
}

References

45 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Rediet Abebe, Solon Barocas, Jon Kleinberg, Karen Levy, Manish Raghavan, and David G Robinson. 2020. Roles for computing in social change. In Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency, pages 252–260. find
  2. Jiajun Bao, Junjie Wu, Yiming Zhang, Eshwar Chandrasekharan, and David Jurgens. 2021. Conversations gone alright: Quantifying and predicting prosocial outcomes in online conversations. In Proceedings of the Web Conference 2021, pages 1134–1145. find
  3. Ruha Benjamin. 2019. Race after technology: Abolitionist tools for the new jim code. Social Forces. find
  4. Su Lin Blodgett, Solon Barocas, Hal Daumé III, and Hanna Wallach. 2020. Language (technology) is power: A critical survey of “bias” in nlp. arXiv preprint arXiv:2005.14050.
  5. Joy Buolamwini and Timnit Gebru. 2018. Gender shades: Intersectional accuracy disparities in commercial gender classification. In Conference on fairness, accountability and transparency, pages 77–91. PMLR. find
  6. Juan Miguel Cejuela, Peter McQuilton, Laura Ponting, Steven J Marygold, Raymund Stefancsik, Gillian H Millburn, Burkhard Rost, FlyBase Consortium, et al. 2014. Tagtog: interactive and text-mining-assisted annotation of gene mentions in plos full-text articles. find
  7. Explosion. 2017. Prodigy. https://prodi.gy. ↗
  8. Karën Fort, Gilles Adda, and K Bretonnel Cohen. 2011. Amazon mechanical turk: Gold mine or coal mine? Computational Linguistics, 37(2):413–420. find
  9. M van Gompel, K Sloot, Martin Reynaert, and APJ van den Bosch. 2017. Folia in practice. the infrastructure of a linguistic annotation format. find
  10. Mary L Gray and Siddharth Suri. 2019. Ghost work: How to stop Silicon Valley from building a new global underclass. Eamon Dolan Books. find
  11. Thilo Hagendorff. 2020. The ethics of ai ethics: An evaluation of guidelines. Minds and Machines, 30(1):99–120. find
  12. Alon Halevy, Peter Norvig, and Fernando Pereira. 2009. The unreasonable effectiveness of data. IEEE Intelligent Systems, 24(2):8–12. find
  13. Lilly C Irani and M Six Silberman. 2013. Turkopticon: Interrupting worker invisibility in amazon mechanical turk. In Proceedings of the SIGCHI conference on human factors in computing systems, pages 611– 620. find
  14. Jinja. JINJA template designer documentation. https://jinja.palletsprojects.com/ en/3.0.x/templates/. ↗
  15. Pratik Joshi, Sebastin Santy, Amar Budhiraja, Kalika Bali, and Monojit Choudhury. 2020. The state and fate of linguistic diversity and inclusion in the nlp world. arXiv preprint arXiv:2004.09095.
  16. Yanzeng Li, Bowen Yu, Li Quangang, and Tingwen Liu. 2021. FITAnnotator: A Flexible and Intelligent Text Annotation System. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies: Demonstrations, pages 35– 41. find
  17. Ninareh Mehrabi, Fred Morstatter, Nripsuta Saxena, Kristina Lerman, and Aram Galstyan. 2021. A survey on bias and fairness in machine learning. ACM Computing Surveys (CSUR), 54(6):1–35.
  18. Julia Mendelsohn, Ceren Budak, and David Jurgens. 2021. Modeling framing in immigration discourse on social media. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2219–2263.
  19. Margaret Mitchell, Simone Wu, Andrew Zaldivar, Parker Barnes, Lucy Vasserman, Ben Hutchinson, Elena Spitzer, Inioluwa Deborah Raji, and Timnit Gebru. 2019. Model cards for model reporting. In Proceedings of the conference on fairness, accountability, and transparency, pages 220–229. find
  20. Robert Munro Monarch. 2021. Human-in-the-Loop Machine Learning: Active learning and annotation for human-centered AI. Simon and Schuster. find
  21. Hiroki Nakayama, Takahiro Kubo, Junya Kamura, Yasufumi Taniguchi, and Xu Liang. 2018. doccano: Text annotation tool for human. Software available from https://github.com/doccano/doccano. ↗
  22. Alexandra Olteanu, Carlos Castillo, Fernando Diaz, and Emre Kıcıman. 2019. Social data: Biases, methodological pitfalls, and ethical boundaries. Frontiers in Big Data, 2:13. find
  23. Iroro Orife, Julia Kreutzer, Blessing Sibanda, Daniel Whitenack, Kathleen Siminyu, Laura Martinus, Jamiil Toure Ali, Jade Abbott, Vukosi Marivate, Salomon Kabongo, et al. 2020. Masakhane– machine translation for africa. arXiv preprint arXiv:2003.11529. — [arXiv:2003.11529](https://arxiv.org/abs/2003.11529) find
  24. Jiaxin Pei and David Jurgens. 2020. Quantifying intimacy in language. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 5307–5326. find
  25. Chen Sun, Abhinav Shrivastava, Saurabh Singh, and Abhinav Gupta. 2017. Revisiting unreasonable effectiveness of data in deep learning era. In Proceedings of the IEEE international conference on computer vision, pages 843–852. find
  26. Saiganesh Swaminathan, Kotaro Hara, and Jeffrey P Bigham. 2017. The crowd work accessibility problem. In Proceedings of the 14th International Web for All Conference, pages 1–4. find
  27. Jiaxin Pei and David Jurgens. 2021. Measuring sentence-level and aspect-level (un) certainty in science communications. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 9959–10011. find
  28. Maxim Tkachenko, Mikhail Malyuk, Nikita Shevchenko, Andrey Holmanyuk, and Nikolai Liubimov. 2021. Label studio: Data labeling software, 2020-2021. Open source software available from https://github.com/heartexlabs/label-studio. ↗
  29. Jiaxin Pei, Vítor Silva, Maarten Bos, Yozon Liu, Leonardo Neves, David Jurgens, and Francesco Barbieri. 2022. find
  30. Semeval 2023 task 9: Multilingual tweet intimacy analysis. arXiv preprint arXiv:2210.01108. find
  31. Xingyao Wang and David Jurgens. 2021. An animated picture says at least a thousand words: Selecting gif-based replies in multimodal dialog. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 3228–3257. find
  32. Tal Perry. 2021. Lighttag: Text annotation platform. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 20–27. find
  33. Max Wiechmann, Seid Muhie Yimam, and Chris Biemann. 2021. Activeanno: General-purpose document-level annotation tool with active learning integration. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies: De… find
  34. Hans Persson, Henrik Åhman, Alexander Arvei Yn-gling, and Jan Gulliksen. 2015. Universal design, inclusive design, accessible design, design for all: different concepts—one goal? on the concept of accessibility—historical, methodological and philosophical aspects. Universal Access in the Information… find
  35. Maarten Sap, Swabha Swayamdipta, Laura Vianna, Xuhui Zhou, Yejin Choi, and Noah A. Smith. 2022. Annotators with attitudes: How annotator beliefs and identities bias toxic language detection.
  36. Burr Settles. 2009. Active learning literature survey.
  37. Boaz Shmueli, Jan Fell, Soumya Ray, and Lun-Wei Ku. 2021. Beyond fair pay: Ethical implications of nlp crowdsourcing. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3758–3769. find
  38. Katta Spiel, Kathrin Gerling, Cynthia L Bennett, Emeline Brulé, Rua M Williams, Jennifer Rode, and Jennifer Mankoff. 2020. Nothing about us without us: Investigating the role of critical disability studies in hci. In Extended Abstracts of the 2020 CHI Conference on Human Factors in Computing Systems… find
  39. Pontus Stenetorp, Sampo Pyysalo, Goran Topić, Tomoko Ohta, Sophia Ananiadou, and Jun’ichi Tsujii. 2012. Brat: a web-based tool for nlp-assisted text annotation. In Proceedings of the Demonstrations at the 13th Conference of the European Chapter of the Association for Computational Linguistics, page…
  40. Jenna Wiens, Suchi Saria, Mark Sendak, Marzyeh Ghassemi, Vincent X Liu, Finale Doshi-Velez, Kenneth Jung, Katherine Heller, David Kale, Mohammed Saeed, et al. 2019. Do no harm: a roadmap for responsible machine learning for health care. Nature medicine, 25(9):1337–1340. find
  41. Alex C Williams, Gloria Mark, Kristy Milland, Edward Lank, and Edith Law. 2019. The perpetual work life of crowdworkers: How tooling practices increase fragmentation in crowdwork. Proceedings of the ACM on Human-Computer Interaction, 3(CSCW):1–28. find
  42. Dustin Wright, Jiaxin Pei, David Jurgens, and Isabelle Augenstein. 2022. Modeling Information Change in Science Communication with Semantically Matched Paraphrases. find
  43. Seid Muhie Yimam, Iryna Gurevych, Richard Eckart de Castilho, and Chris Biemann. 2013. Webanno: A flexible, web-based and visually supported system for distributed annotations. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 1–…
  44. Naitian Zhou and David Jurgens. 2020. Condolences and empathy in online communities. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 609– 626. find
  45. Kathryn Zyskowski, Meredith Ringel Morris, Jeffrey P Bigham, Mary L Gray, and Shaun K Kane. 2015. Accessible crowdwork? understanding the value in and challenge of microtask employment for people with disabilities. In Proceedings of the 18th ACM Conference on Computer Supported Cooperative Work & So… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/pei-2022-potato-portable-text-annotation-tool/