Crowdsourcing Discourse Relation Annotations by a Two-Step Connective Insertion Task

Summary

Labelling implicit coherence relations usually requires annotators trained in frameworks such as PDTB or RST, so crowd workers without training cannot assign sense labels reliably. Earlier crowdsourcing work avoided this by having workers insert a connective from a short fixed list. That approach only works when few relation types are possible, and the offered options may steer how workers read the text.

The authors propose a two-stage task. First, workers freely type a connective (or nothing) into the gap between two segments. If their choice is ambiguous, they then pick from a dynamically built list of unambiguous alternatives. The relation label comes from intersecting the senses of both insertions, which are looked up in a newly compiled connective bank.

The method was tested on WSJ passages that already carry PDTB and RST-DT labels. It proved about as reliable as the earlier restricted approach and reproduced label distributions across conditions. Its labels combined traits of both treebanks. This makes scalable, distribution-based discourse annotation feasible, including for other languages.

Key points

The released data cover crowdsourced labels for 447 implicit discourse relations, plus a connective bank of 800 phrases. The bank includes standard connectives, modified variants, combinations and alternative lexicalizations; typos are kept as variants but not counted.
The sense inventory follows PDTB 3.0 Level-3 senses, minus the four speech-act relations, and adds PRESENTATIONAL and BACKGROUND.
In step two, workers see at most 10 disambiguating connectives, chosen with the help of Knott's connective hierarchy. If the first insertion is unrecognized or empty, workers instead see a default list of ten connectives, such as "in other words" and "accordingly".
On a more representative random sample, majority crowd labels agreed with PDTB3 slightly better than RST-DT labels did (overall .44 vs .40). Workers, however, under-labelled INSTANTIATION and over-labelled CAUSE.
Open limitations include relations that cannot be marked by a connective (e.g. ENTITY RELATIONS) and segment pairs that are unrelated.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Discourse analysis

Cited by

22 papers of our library cite it (of the 140673 whose bibliographies we read); 7 of them have a page here, 15 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/w19-4003

@inproceedings{yung2019crowdsourcing,
  title = {{Crowdsourcing Discourse Relation Annotations by a Two-Step Connective Insertion Task}},
  author = {Yung, Frances and Demberg, Vera and Scholman, Merel},
  booktitle = {Proceedings of the 13th Linguistic Annotation Workshop},
  year = {2019},
  doi = {10.18653/v1/w19-4003},
  url = {https://doi.org/10.18653/v1/w19-4003}
}

References

8 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

Lynn Carlson, Daniel Marcu, and Mary Ellen Okurowski. 2003. Building a discourse-tagged corpus in the framework of Rhetorical Structure Theory. In Current and new directions in discourse and dialogue, pages 85–112. Springer. find our card
Alexander Philip Dawid and Allan M Skene. 1979. Maximum likelihood estimation of observer error-rates using the em algorithm. Journal of the Royal Statistical Society: Series C (Applied Statistics), 28(1):20–28. find
Vera Demberg, Fatemeh Torabi Asr, and Merel C.J. Scholman. 2017. How compatible are our discourse annotations? Insights from mapping RST-Lastly, the current method assumes that all dis-DT and PDTB annotations. arXiv:1704.08893. — [arXiv:1704.08893](https://arxiv.org/abs/1704.08893) find
Merel C.J. Scholman. 2019. Coherence relations in discourse and cognition: comparing approaches, annotations and interpretations. Ph.D. thesis, University of Saarland. arXiv preprint Daisuke Kawahara, Yuichiro Machida, Tomohide Shibata, Sadao Kurohashi, Hayato Kobayashi, and Manabu Sassano. 2014. Ra… find
Merel C.J. Scholman and Vera Demberg. 2017a. Crowdsourcing discourse interpretations: On the influence of context and the reliability of a connective insertion task. In Proceedings of the 11th Linguistic Annotation Workshop, pages 24–33. find
Merel C.J. Scholman and Vera Demberg. 2017b. Examples and specifications that prove a point: Identifying elaborative and argumentative discourse relations. Dialogue & Discourse, 8(2):56–83. find our card
Yudai Kishimoto, Shinnosuke Sawada, Yugo Murawaki, Daisuke Kawahara, and Sadao Kurohashi. 2018. Improving crowdsourcing-based annotation of japanese discourse relations. In Proceedings of the 11th Language Resources and Evaluation Conference. find our card
A Alastair Knott. 1996. A Data-Driven Methodology for Motivating a Set of Coherence Relations. Ph.D. thesis, University of Edinburgh. find

↑ back to top

📄 Crowdsourcing Discourse Relation Annotations by a Two-Step Connective Insertion Task