Elamur Research⌕
☰
home / papers / 2017 / TokTrack: A Complete Token Provenance and Change Tracking Dataset for the English Wikipedia

TokTrack: A Complete Token Provenance and Change Tracking Dataset for the English Wikipedia

Paper · 2017 · Fabian Flock, Kenan Erdogan and Maribel Acosta

Summary

Studying how Wikipedia text evolves requires knowing, for every word, which revision introduced it and when it was later removed or restored. Computing this is costly. Naive diffs between consecutive revisions also miss restorations of earlier content and text moved to another position. The authors adapted their Wikiwho algorithm, which matches paragraphs and sentences against all earlier revisions and falls back to a text diff. They ran it over the full English Wikipedia history dump of November 2016. The result is TokTrack: roughly 13.5 billion token instances. Each instance records its origin revision and every revision in which it was deleted or re-added. Using this data, the authors examine provenance, content survival, conflict and editor interaction at whole-Wikipedia scale. They show that partial reverts far outnumber full ones. They also show that the common identity-revert heuristic often mislabels which revision was reverted and whether the revert was full or partial. Researchers no longer have to run the expensive text comparison themselves.

Key points

  • Source and scale: the dump covers January 2001 to October 2016, about 12 TB of uncompressed XML. From it come 5,275,388 non-redirect articles, 451,350,901 revisions and 13,545,349,787 token instances.
  • Algorithm:
    • Wikiwho was chosen because earlier evaluation gave it about 95% correct authorship attribution, versus around 50% for Wikitrust on a small sample.
    • Tokens include wiki markup and special characters, and capitalization is ignored.
    • Processing took about 25 days on a 20-core VM with 122 GB RAM.
  • Output format: one token per line, with page ID, revision IDs, an internal token ID, and lists of deletion and reinsertion revisions. The "current content" type is one of several per-article outputs.
  • Revision statistics:
    • 61.51% of revisions removed or restored some content.
    • 14.62% were self-corrections.
    • 14.84% fully undid another editor's revision, and 50.65% did so partially.
  • Comparison with the identity-revert method:
    • That method flags only 8.87% of revisions as full reverts.
    • Only 73.45% of its reverting–reverted pairs are confirmed as full reverts by TokTrack, and 12.98% turn out to be partial.
    • In the 1000 most-edited articles, full reverts rise to 25.28% of revisions.

Interesting ideas

  • I1. Give every token a stable identity separate from its string value, with an origin plus lists of removal and revival events, so a word that disappears and returns keeps its history.
  • I2. Match new content against all previous versions, not only the immediately preceding one, so restorations and long-distance moves are recognized rather than counted as deletion plus fresh insertion.
  • I3. Match coarse units first (paragraphs, then sentences) and fall back to a fine-grained diff only when nothing matches; this was both faster and more accurate than the alternatives.
  • I4. Check a cheap, widely trusted heuristic against a fine-grained method at full scale. The identity-revert method agreed with TokTrack on only about three quarters of its pairs, a gap invisible without such a comparison.
  • I5. Measure reverts as a graded ratio of undone actions instead of a yes/no label, and inspect odd peaks by hand. The spike at 50% turned out to be chains of incremental replacements.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (world): Entities and states in the world of a text could carry the same birth/removal/restoration log, letting the order of events and "it came back" situations be explained instead of being treated as new objects.
  • I2 (learn): When silver data or repair rules are regenerated, comparing each analysis against all earlier runs would expose oscillating decisions and genuine regressions.
  • I3 (tokens): When a document is re-processed after small edits, unchanged sentences could reuse cached tokens, tags and lemmas, and only changed spans would go through the full pipeline.
  • I4 (learn): Shortcut evaluators, such as surface-match scoring, should be audited against a deep reference on the whole corpus, with disagreement rates reported as a negative control.
  • I5 (back): In the text → graph → text compressor, the fraction of source tokens that survive the round trip could be scored per sentence, with anomalous ratios checked manually.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Karlsruhe Institute of TechnologyKarlsruhe · Germany6 works · 790 citations

Also on the byline: GESIS - Leibniz Institute for the Social Sciences.

📚 Shelves 7 Computer Science · 9,086 works Artificial Intelligence · 8,889 works Arts and Humanities · 9,296 works Language and Linguistics · 9,206 works Social Sciences · 744 works Literature and Literary Theory · 340 works Communication · 44 works

🧭 Topics 1 English · 1,373 works

Cite

Cite as Fabian Flock, Kenan Erdogan and Maribel Acosta. 2017. TokTrack: A Complete Token Provenance and Change Tracking Dataset for the English Wikipedia. arXiv:1703.08244.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1703.08244 · arXiv 1703.08244

@misc{flock2017toktrack,
  title = {{TokTrack: A Complete Token Provenance and Change Tracking Dataset for the English Wikipedia}},
  author = {Fabian Flock, Kenan Erdogan and Maribel Acosta},
  year = {2017},
  doi = {10.48550/arxiv.1703.08244},
  eprint = {1703.08244},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1703.08244}
}

References

33 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. T. Adler and L. De Alfaro. A content-driven reputation system for the Wikipedia. In Proc. WWW, pages 261–270, 2007. Number of reverting revisions (log scale) 1e+08 1e+05 1e+02 1.00 0e+00 1e+05 2e+05 3e+05 Number of undone actions TokTrack: A Complete Token Provenance and Change Tracking Dataset find
  2. T. Adler, L. de Alfaro, I. Pye, and V. Raman. Measuring author contributions to the Wikipedia. In Proc. WikiSym, pages 15:1–15:10, 2008. find
  3. E. Borra, D. Laniado, E. Weltevrede, M. Mauri, G. Magni, T. Venturini, P. Ciuccarelli, R. Rogers, and A. Kaltenbrunner. A platform for visually exploring the development of wikipedia articles. In Proc. ICWSM, 2015. find
  4. U. Brandes, P. Kenis, J. Lerner, and D. van Raaij. Network analysis of collaboration structure in Wikipedia. In Proc. WWW, 2009. find
  5. S. Bykau, F. Korn, D. Srivastava, and Y. Velegrakis. Fine-grained controversy detection in Wikipedia. In Proc. ICDE, 2015. find
  6. L. De Alfaro and M. Shavlovsky. Attributing authorship of revisioned content. In Proc. WWW, pages 343–354, 2013. find
  7. S. DeDeo. Conflict and computation on wikipedia: a finite-state machine analysis of editor interactions. CoRR, abs/1512.04177, 2015. ↗
  8. F. Flöck and M. Acosta. WikiWho: Precise and efficient attribution of authorship of revisioned content. In Proc. WWW, pages 843–854, 2014. find
  9. F. Flöck and M. Acosta. whoVIS: visualizing editor interactions and dynamics in collaborative writing over time. In Proc. WWW - Companion Volume, pages 191–194, 2015. find
  10. F. Flöck, K. Erdogan, and M. Acosta. TokTrack: A complete token provenance and change tracking dataset for the english Wikipedia. Zenodo data sharing service, DOI: https://doi.org/10.5281/zenodo.345571, Feb 2017. ↗ our card
  11. F. Flöck, D. Laniado, F. Stadthaus, and M. Acosta. Towards better visual tools for exploring wikipedia article development–the use case of ”gamergate controversy”. In Wikipedia, a Social Pedia - Workshop at the Ninth International AAAI Conference on Web and Social Media, 2015. find
  12. F. Flöck and A. Rodchenko. Whose article is it anyway?–Detecting authorship distribution in Wikipedia articles over time with WIKIGINI. In Online proceedings of the Wikipedia Academy 2012. Wikimedia, 2012. find
  13. F. Flöck, D. Vrandečić, and E. Simperl. Revisiting reverts: Accurate revert detection in wikipedia. In Proceedings of the 23rd ACM Conference on Hypertext and Social Media, HT ’12, pages 3–12, New York, NY, USA, 2012. ACM. find
  14. A. Halfaker, R. S. Geiger, J. T. Morgan, and J. Riedl. The rise and decline of an open collaboration system: How Wikipedia’s reaction to popularity is causing its decline. American Behavioral Scientist, 2012. ↗
  15. A. Halfaker, A. Kittur, R. Kraut, and J. Riedl. A jury of your peers: quality, experience and ownership in Wikipedia. In D. Riehle and A. Bruckman, editors, Int. Sym. Wikis, 2009. find
  16. A. Halfaker, A. Kittur, and J. Riedl. Don’t bite the newbies: How reverts affect the quantity and quality of Wikipedia work. In Proc. WikiSym, pages 163–172, 2011. find
  17. T. Iba, K. Nemoto, B. Peters, and P. A. Gloor. Analyzing the creative editing behavior of Wikipedia editors: Through dynamic social network analysis. Procedia-Social and Behavioral Sciences, 2(4):6441–6456, 2010. find
  18. S. Javanmardi, C. Lopes, and P. Baldi. Modeling user reputation in wikis. Statistical Analysis and Data Mining, 3(2):126–139, 2010. ↗
  19. B. C. Keegan, S. Lev, and O. Arazy. Analyzing organizational routines in online knowledge collaborations: A case for sequence analysis in CSCW. In Proc. CSCW, 2016. ↗
  20. A. Kittur, B. Suh, B. A. Pendleton, and E. H. Chi. He says, she says: Conflict and coordination in Wikipedia. In Proc. CHI, pages 453–462, New York, NY, USA, 2007. find
  21. S. K. Lam, A. Uduwage, Z. Dong, S. Sen, D. R. Musicant, L. Terveen, and J. Riedl. WP:Clubhouse? an exploration of Wikipedia’s gender imbalance. In WikiSym, 10/2011 2011. find
  22. J. Lerner and A. Lomi. Dominance, deference, and hierarchy formation in Wikipedia edit-networks. In Proc. COMPLEX NETWORKS, volume 693, page 95. Springer, 2016. Fabian Flöck et al. find
  23. S. Maniu, B. Cautis, and T. Abdessalem. Building a signed network from interactions in Wikipedia. In Databases and Social Networks, pages 19–24, 2011. find
  24. R. Priedhorsky, J. Chen, S. T. K. Lam, K. Panciera, L. Terveen, and J. Riedl. Creating, destroying, and restoring value in Wikipedia. In ACM GROUP, pages 259–268, 2007. find
  25. H. Sepehri Rad, A. Makazhanov, D. Rafiei, and D. Barbosa. Leveraging editor collaboration patterns in Wikipedia. In ACM Hypertext, pages 13–22, 2012. find
  26. B. Suh, E. H. Chi, B. A. Pendleton, and A. Kittur. Us vs. them: Understanding social dynamics in Wikipedia with revert graph visualizations. In IEEE VAST, pages 163–170, 2007. find
  27. B. Suh, G. Convertino, E. H. Chi, and P. Pirolli. The singularity is not near: slowing growth of Wikipedia. In Proc. WikiSym, pages 8:1–8:10, 2009. find
  28. R. Sumi, T. Yasseri, A. Rung, A. Kornai, and J. Kertész. Edit wars in Wikipedia. arXiv preprint arXiv:1107.3689, 2011. ↗
  29. M. Tsvetkova, R. Garcı́a-Gavilanes, and T. Yasseri. Dynamics of disagreement: Large-scale temporal network analysis reveals negative interactions in online collaboration. Scientific reports, 6, 2016. ↗
  30. F. B. Viégas, M. Wattenberg, and K. Dave. Studying cooperation and conflict between authors with history flow visualizations. In Proc. CHI, pages 575–582, New York, NY, USA, 2004. find
  31. B. Vuong, E. Lim, A. Sun, M. Le, and H. W. Lauw. On ranking controversies in wikipedia: models and evaluation. In Proc. WSDM, pages 171–182, 2008. find
  32. T. Yasseri, A. Spoerri, M. Graham, and J. Kertész. The most controversial topics in Wikipedia: A multilingual and geographical analysis. Global Wikipedia: International and Cross-Cultural Issues in Online Collaboration., 2014. ↗
  33. T. Yasseri, R. Sumi, A. Rung, A. Kornai, J. Kertész, and A. Szolnoki. Dynamics of conflicts in Wikipedia. PLoS ONE, 7(6), 2012. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/flock-2017-toktrack-complete-token-provenance-change/