Elamur Research⌕
☰
home / papers / 2018 / Improving Crowdsourcing-Based Annotation of Japanese Discourse Relations

Improving Crowdsourcing-Based Annotation of Japanese Discourse Relations

Summary

Discourse parsing needs annotated corpora, but such resources are rare outside English and Chinese, and building them by hand is slow and costly. Kawahara et al. (2014) used crowdsourcing to build a Japanese discourse corpus of 10,000 web documents, but they never measured how reliable those labels were. This paper fills that gap. The authors had trained linguists annotate a gold subset and compared the crowd labels against it. The crowd-produced "Old Annotation" reached only 49.5% micro-averaged F-measure. Error analysis found two main problems:

  • Workers frequently overlooked explicit discourse connectives.
  • The task instructions did not make the labeling criteria clear.

To address this, the authors revised the instructions to include descriptions of connective phrases. These descriptions push workers to apply positive language tests, such as checking whether "if" can be inserted between two clauses. Re-running the crowdsourcing produced a "New Annotation" that improved F-measure by about 3 points, with a larger gain on explicit relations. The work shows that small changes to task design can raise crowd annotation quality. It also identifies implicit relations as the main remaining weakness.

Key points

  • Base resource: Each document is the first three sentences of a page from the Kyoto University Web Leads Corpus.
    • Clauses were segmented with the KNP parser.
    • Labels use a two-level tagset: CONTINGENCY covers Cause/Reason, Purpose, Condition and Ground; COMPARISON covers Contrast and Concession; OTHER is a separate label.
  • Original pipeline: Annotation ran in two stages on Yahoo! Crowdsourcing, with 10 workers per item and answers aggregated using GLAD.
    • The first stage filtered pairs labeled OTHER; the second stage classified the remaining relations.
    • The full annotation took under eight hours.
  • Gold data: Three professional annotators labeled 500 documents. Of these, 313 were sampled for evaluation.
  • Main error source: 40.2% (384/955) of explicit clause pairs were wrongly filtered out as OTHER. Roughly 47% of those pairs contained Condition-type connectives.
  • Results: Micro F-measure rose from 0.495 to 0.528, and recall rose by about 7.5 points. On explicit relations, F-measure improved from 0.513 to 0.605.
    • The authors plan a cloze-style connective-insertion test, modeled on the PDTB procedure, to handle implicit relations.
    • They also plan to release the re-annotated data publicly.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Discourse analysis

Cited by

11 papers of our library cite it (of the 140673 whose bibliographies we read); 5 of them have a page here, 6 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{kishimoto2018improving,
  title = {{Improving Crowdsourcing-Based Annotation of Japanese Discourse Relations}},
  author = {Kishimoto, Yudai and Sawada, Shinnosuke and Murawaki, Yugo and Kawahara, Daisuke and Kurohashi, Sadao},
  booktitle = {Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018)},
  year = {2018},
  doi = {10.63317/28vbj6p6xyoq},
  url = {https://doi.org/10.63317/28vbj6p6xyoq}
}

References

8 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Asher, N. and Lascarides, A. (2003). Logics of Conversation. find
  2. Cambridge University Press. Carlson, L., Marcu, D., and Okurowski, M. E. (2001). Building a discourse-tagged corpus in the framework of (iii) rhetorical structure theory. In Proceedings of the Second [(and) They made delicious wine.] SIGdial Workshop on Discourse and Dialogue - Volume The discourse… find
  3. [Tuchiya, who is a sales manager of Odoya, requested (a brand new product) for winery in Ya-magata,] Dublin City University and Association for Computational Linguistics. Kurohashi, S. and Nagao, M. (1994). A syntactic analysis method of long Japanese sentences based on the detection of conjunctive… find
  4. Prasad, R., Dinesh, N., Lee, A., Miltsakaki, E., Robaldo, L., Joshi, A., and Webber, B. (2008). The Penn discourse treebank 2.0. In Proceedings of the 6th International Conference on Language Resources and Evaluation (LREC2008), pages 2961–2968. find our card
  5. Prasad, R., Webber, B., and Joshi, A. (2014). Reflections on the penn discourse treebank, comparable corpora, and complementary annotation. Computational Linguistics, 40(4):921–950. find our card
  6. Snow, R., O’Connor, B., Jurafsky, D., and Ng, A. (2008). Cheap and fast – but is it good? evaluating non-expert annotations for natural language tasks. In Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing, pages 254–263. Association for Computational Linguistics. find
  7. Whitehill, J., Wu, T., Bergsma, J., Movellan, J. R., and Ru-volo, P. L. (2009). Whose vote should count more: Optimal integration of labels from labelers of unknown expertise. In Y. Bengio, et al., editors, Advances in Neural Information Processing Systems 22, pages 2035–2043. Curran Associates, Inc… find
  8. Zhou, Y. and Xue, N. (2012). PDTB-style discourse annotation of Chinese text. In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 69–77. Association for Computational Linguistics. find

↑ back to top

This page in Markdown →