Elamur Research⌕
☰
home / papers / 2012 / Evaluation of Discourse Relation Annotation in the Hindi Discourse Relation Bank

Evaluation of Discourse Relation Annotation in the Hindi Discourse Relation Bank

Summary

The paper tests whether changes made to the Penn Discourse Treebank (PDTB) scheme while adapting it to Hindi actually improve annotation quality. These changes came from earlier work on the Hindi Discourse Relation Bank (HDRB) and were meant to make sense classification more uniform and conceptually cleaner. One example is naming arguments by their semantic role rather than by linear order. The authors applied the HDRB guidelines to a subset of a 400K-word Hindi news corpus and ran an inter-annotator agreement study. It covered explicit connectives, implicit connectives, AltLex and EntRel. Annotators worked over continuous raw text with the PDTB annotation tool.

Agreement on relation senses was high, which supports the hierarchical sense scheme. Agreement was weak in other areas:

  • annotators disagreed on which tokens count as connectives at all;
  • argument spans matched poorly;
  • annotators seemed to prefer linear-order argument labels over semantic ones.

The authors conclude that some modifications made the task harder. They propose splitting the workflow into inter- and intra-sentential passes, returning to syntactic argument naming, and allowing implicit relations between non-adjacent sentences. The study is a practical lesson in annotation task design for discourse resources.

Key points

  • Corpus: HDRB is built on about 400K words of multi-domain news from the Hindi daily AMAR UJALA. The same corpus also carries dependency, predicate-argument and phrase-structure layers.
  • What was annotated: The scheme marks explicit connectives, implicit connectives, AltLex and EntRel/NoRel, with Arg1/Arg2 spans following the minimality principle. Unlike PDTB, it also annotates implicit relations across paragraph boundaries.
  • Connective counts and spans: Annotators produced different numbers of connective tokens, and the line between connectives and AltLex was especially hard to draw. Argument-span agreement was low for both explicit and implicit relations.
  • Senses: Sense agreement was high for both explicit and implicit connectives. However, annotators were allowed to back off to coarser type-level labels.
  • Proposed revisions: a two-step workflow (inter-sentential relations first, then intra-sentential), a return to PDTB-style linear-order argument naming, and annotation of implicit connectives between non-adjacent sentences.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Discourse analysis

Cited by

8 papers of our library cite it (of the 140673 whose bibliographies we read); 1 of them have a page here, 7 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{kolachina2012evaluation,
  title = {{Evaluation of Discourse Relation Annotation in the Hindi Discourse Relation Bank}},
  author = {Kolachina, Sudheer and Prasad, Rashmi and Sharma, Dipti Misra and Joshi, Aravind},
  booktitle = {Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC'12)},
  year = {2012},
  doi = {10.63317/4vqsnirkgvgm},
  url = {https://doi.org/10.63317/4vqsnirkgvgm}
}

References

15 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. A. Al-Saif and K. Markert. 2010. The Leeds Arabic Discourse Treebank: Annotating Discourse Connectives for Arabic. In Proceedings of the 7th International Conference on Language Resources and Evaluation (LREC-2010), Valletta, Malta. find our card
  2. R. Begum, S. Husain, A. Dhwaj, D.M. Sharma, L. Bai, and R. Sangal. 2008. Dependency Annotation Scheme for Indian Languages. Proceedings of IJCNLP-2008. find
  3. R. Bhatt, B. Narasimhan, M. Palmer, O. Rambow, D.M. Sharma, and F. Xia. 2009. A Multi-Representational and Multi-Layered Treebank for Hindi/Urdu. In Proceedings of the Third Linguistic Annotation Workshop, pages 186–189. Association for Computational Linguistics. find
  4. E. Miltsakaki, R. Prasad, A. Joshi, and B. Webber. 2004. Annotating Discourse Connectives And Their Arguments. In Proceedings of the HLT/NAACL Workshop on Frontiers in Corpus Annotation, pages 9–16. find our card
  5. L. Mladová, Š. Zikánová, and E. Hajičová. 2008. From Sentence to Discourse: Building an Annotation Scheme for Discourse Based on Prague Dependency Treebank. In Proceedings of the 6th International Conference on Language Resources and Evaluation (LREC 2008). find
  6. U. Oza, R. Prasad, S. Kolachina, S. Meena, D.M. Sharma, and A. Joshi. 2009a. Experiments with Annotating Discourse Relations in the Hindi Discourse Relation Bank. In Proceedings of the 7th International Conference on Natural Language Processing (ICON-2009), Hyderabad, India. find our card
  7. U. Oza, R. Prasad, S. Kolachina, D.M. Sharma, and A. Joshi. 2009b. The Hindi Discourse Relation Bank. In Proceedings of the Third Linguistic Annotation Workshop, pages 158–161. Association for Computational Linguistics. find our card
  8. M. Palmer, R. Bhatt, B. Narasimhan, O. Rambow, D.M. Sharma, and F. Xia. 2009. Hindi Syntax: Annotating Dependency, Lexical Predicate-Argument Structure, and Phrase Structure. In The 7th International Conference on Natural Language Processing, pages 14–17. find
  9. R. Prasad, N. Dinesh, A. Lee, A. Joshi, and B. Webber. 2007. Attribution and its annotation in the Penn Discourse TreeBank. Traitement Automatique des Langues, Special Issue on Computational Approaches to Document and Discourse, 47(2). find
  10. R. Prasad, N. Dinesh, A. Lee, E. Miltsakaki, L. Robaldo, A. Joshi, and B. Webber. 2008. The Penn Discourse Treebank 2.0. In Proceedings of the 6th International Conference on Language Resources and Evaluation (LREC 2008), pages 2961– 2968. Citeseer. find our card
  11. R. T. Rachakonda and D. M. Sharma. 2011. Creating an Annotated Tamil Corpus as a Discourse Resource. In Proceedings of the 5th Linguistic Annotation Workshop, pages 119–123, Portland, OR. find
  12. S. Tonelli, G. Riccardi, R. Prasad, and A. Joshi. 2010. Annotation of Discourse Relations for Conversational Spoken Dialogs. In Proceedings of the Seventh International Conference on Language Resources and Evaluation (LREC 2010), Valletta, Malta. find our card
  13. F. Xia, O. Rambow, R. Bhatt, M. Palmer, and D.M. Sharma. 2008. Towards a Multi-Representational Treebank. Proceedings of the 7th International Workshop on Treebanks and Linguistic Theories (TLT-7), pages 207–238. find
  14. D. Zeyrek and B. Webber. 2008. A Discourse Resource for Turkish: Annotating Discourse Connectives in the METU Corpus. Proceedings of IJCNLP-2008. Hyderabad, India. find our card
  15. Y. Zhou and N. Xue. 2012. PDTB-style Discourse Annotation of Chinese Text. In Proc. 50 th Annual Meeting of the ACL, Jeju Island, Korea. find

↑ back to top

This page in Markdown →