Elamur Research⌕
☰
home / papers / 2014 / Summarizing Online Forum Discussions – Can Dialog Acts of Individual Messages Help?

Summarizing Online Forum Discussions – Can Dialog Acts of Individual Messages Help?

Summary

Long threads on web forums often run across many pages, with several participants contributing competing opinions and fixes. A newcomer gains much more from a short digest than from reading every post. The authors address this with extractive summarization at the level of whole posts rather than sentences, because forum messages are short and splitting them would strip away context. Summarization is treated as binary supervised classification: each post is either in the summary or not. The classifier uses five textual features (cosine similarity to the thread title, length in unique words, normalized position, similarity to the thread centroid, and mean similarity to the other posts). It also uses binary indicators for the post's dialog act, such as Question, Solution or Feedback. Experiments use Ubuntu and TripAdvisor (NYC) threads with human-written reference summaries. Adding dialog act features consistently raised precision and F-1 over the textual features alone, and both setups clearly beat a rule-based baseline. The work shows that knowing the role a message plays in a discussion helps decide whether it belongs in a summary.

Key points

  • Data: 100 threads from ubuntuforums.org (556 posts) and 100 from tripadvisor.com NYC (916 posts), taken from Bhatia et al. (2012) with existing dialog act labels.
  • Dialog acts: eight categories are used: Question, Repeat Question, Clarification, Further Details, Solution, Positive Feedback, Negative Feedback and Junk.
  • Gold summaries:
    • Two annotators (H1, H2) each wrote free-text summaries of every thread.
    • Posts were ranked by cosine similarity to each summary, following Rambow et al. (2004).
    • The top 20% of posts formed the gold set.
    • The data is released for research use.
  • Setup: Weka Naive Bayes trained on a 75/25 train/test split. The baseline labels every Question and Solution post as summary-worthy, which gives low precision (0.31–0.39) but moderate F-1 (0.45–0.53).
  • Results:
    • For H1 on Ubuntu, dialog acts raised precision from 0.578 to 0.620 and F-1 from 0.536 to 0.608. On NYC, F-1 rose from 0.607 to 0.655.
    • Reported relative gains across all settings range from about 2.8% to 20.5%.
    • Parts of the H2 rows appear garbled in the PDF extraction, so their absolute values should be checked against the original paper.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Speech act theory

Cited by

18 papers of our library cite it (of the 140673 whose bibliographies we read); 2 of them have a page here, 16 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.3115/v1/d14-1226

@inproceedings{bhatia2014summarizing,
  title = {{Summarizing Online Forum Discussions – Can Dialog Acts of Individual Messages Help?}},
  author = {Bhatia, Sumit and Biyani, Prakhar and Mitra, Prasenjit},
  booktitle = {Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP)},
  year = {2014},
  doi = {10.3115/v1/d14-1226},
  url = {https://doi.org/10.3115/v1/d14-1226}
}

References

15 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Sumit Bhatia, Prakhar Biyani, and Prasenjit Mitra. 2012. Classifying user messages for managing web forum data. In Proceedings of the 15th International Workshop on the Web and Databases 2012, WebDB 2012, Scottsdale, AZ, USA, May 20, 2012, pages 13– 18. find
  2. Prakhar Biyani, Sumit Bhatia, Cornelia Caragea, and Prasenjit Mitra. 2012. Thread specific features are helpful for identifying subjectivity orientation of online forum threads. In COLING 2012, 24th International Conference on Computational Linguistics, Proceedings of the Conference: Technical Papers, 8-15 December 2012, Mumbai, India, pages 295–310. find
  3. Prakhar Biyani, Sumit Bhatia, Cornelia Caragea, and Prasenjit Mitra. 2014. Using non-lexical features for identifying factual and opinionative threads in online forums. ↗
  4. Knowledge-Based Systems, In Press, doi = http://dx.doi.org/10.1016/j.knosys.2014.04.048. ↗
  5. Antonella Carbonaro. 2010. Towards an automatic forum summarization to support tutoring. In Mil-tiadisD. Lytras, Patricia Ordonez De Pablos, David Avison, Janice Sipior, Qun Jin, Walter Leal, Lorna Uden, Michael Thomas, Sara Cervai, and David Horner, editors, Technology Enhanced Learning. Quality of Teaching and Educational Reform, volume 73 of Communications in Computer and Information Science, pages 141–147. Springer Berlin Heidelberg. find
  6. Giuseppe Carenini and Jackie Chi Kit Cheung. 2008. Extractive vs. nlg-based abstractive summarization of evaluative text: The effect of corpus controversiality. In Proceedings of the Fifth International Natural Language Generation Conference, pages 33–41. Association for Computational Linguistics. find
  7. William W. Cohen, Vitor R. Carvalho, and Tom M. Mitchell. 2004. Learning to Classify Email into “Speech Acts”. In EMNLP, pages 309–316. ACL. find our card
  8. Udo Hahn and Inderjeet Mani. 2000. The challenges of automatic summarization. Computer, 33(11):29– 36. ↗
  9. Mark Hall, Eibe Frank, Geoffrey Holmes, Bernhard Pfahringer, Peter Reutemann, and Ian H. Witten. 2009. The WEKA Data Mining Software: An Update. SIGKDD Explorations, 11(1). ↗
  10. Minwoo Jeong, Chin-Yew Lin, and Gary Geunbae Lee. 2009. Semi-supervised speech act recognition in emails and forums. In Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing: Volume 3 - Volume 3, EMNLP ’09, pages 1250–1259. find
  11. Shafiq R. Joty, Giuseppe Carenini, and Chin-Yew Lin. 2011. Unsupervised modeling of dialog acts in asynchronous conversations. In IJCAI, pages 1807– 1813. IJCAI/AAAI. find
  12. Fei Liu and Yang Liu. 2009. From extractive to abstractive meeting summaries: can it be done by sentence compression? In Proceedings of the ACL-IJCNLP 2009 Conference Short Papers, pages 261– 264. Association for Computational Linguistics. find
  13. O. Rambow, L. Shrestha, J. Chen, and C. Laurdisen. 2004. Summarizing email threads. Proceedings of HLT-NAACL 2004: Short Papers. find
  14. Zhaochun Ren, Jun Ma, Shuaiqiang Wang, and Yang Liu. 2011. Summarizing web forum threads based on a latent topic propagation process. In Proceedings of the 20th ACM International Conference on Information and Knowledge Management, CIKM ’11, pages 879–884, New York, NY, USA. ACM. find
  15. Liang Zhou and Eduard Hovy. 2005. Digesting virtual ”geek” culture: The summarization of technical internet relay chats. In Proceedings of the 43rd Annual Meeting on Association for Computational Linguistics, ACL ’05, pages 298–305, Stroudsburg, PA, USA. Association for Computational Linguistics. find

↑ back to top