Elamur Research⌕
☰
home / papers / 2017 / TrentoTeam at SemEval-2017 Task 3: An application of Grice Maxims in Ranking Community Question Answers

TrentoTeam at SemEval-2017 Task 3: An application of Grice Maxims in Ranking Community Question Answers

Paper · 2017 · Mohammed R. H. Qwaider, Abed Alhakim Freihat and Fausto Giunchiglia

Summary

The paper describes the Grice Ranking system, entered in SemEval-2017 Task 3 subtask A. The task is to reorder a list of community forum answers (ten per question, from Qatar Living) by how relevant each is to the question. The authors do not train an end-to-end ranker. Instead they turn three of Grice's conversational maxims (Quantity, Relation, Manner) into countable surface signals in each answer. Quality is left out.

  • Quantity counts informative content: named entities, URLs, emails and phone numbers, currency, and multi-digit numbers.
  • Relation combines question–answer similarity, imperatives and polite suggestion phrases, yes/no particles for factoid questions, and domain-specific terms.
  • Manner rewards positive sentiment and penalizes frustrated, ironic or insulting wording.

The signal counts are summed into a hand-weighted score. Answers posted by the person who asked the question are pushed to the bottom. The system finished 10 points below the best team, 6 points above the IR baseline and 15 points above the lowest system. It shows that an interpretable, pragmatics-inspired scorer is a reasonable baseline for answer ranking.

Key points

  • The ranking score adds counts of similarity, named entities, references, currency/numbers, imperatives/politeness, domain terms and positive words. It then subtracts counts of negative, ironic/humbling and insulting/degrading words.
  • Off-the-shelf NER performed poorly on forum text, so the authors retrained an openNLP name finder on a self-annotated slice of the training data. The new model reached 87% F1, and both the model and the corpus were released.
  • Similarity experiments mixed TF-IDF, string similarity and Word2Vec, Brown and Clark cluster embeddings at fixed weights (0.5/0.5, 0.3/0.7). These were evaluated by MAP on the SemEval 2016 test set.
  • Imperatives and polite phrases ("I suggest", "I recommend") were detected with an OpenNLP POS-tagger. Sentiment used public positive/negative word lists, which the authors extended with ironic and insulting expressions found in the training data.
  • Grammatical correctness was deliberately not used as a signal, because many forum users are not native English speakers.

Interesting ideas

  • I1. Each conversational maxim becomes a set of checkable surface features rather than an inferred intention, so every point an answer gains or loses can be traced to a cause.
  • I2. Polite forms such as "I would" or "I suggest" are treated as stand-ins for imperatives, so softened advice still counts as an instruction to act.
  • I3. In a factoid yes/no question, a "yes" or "no" particle in the reply signals that the reply actually engages with the question.
  • I4. Who wrote an utterance is used structurally: the asker's own follow-ups are ranked last as non-answers.
  • I5. When general NER failed on the domain, the authors retrained it on a small self-annotated in-domain sample and released both the data and the model.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (frames): The prag crate could tag answer-like utterances with explicit Quantity/Relation/Manner flags, recording which feature fired so implicature judgements stay explainable.
  • I2 (frames): The speech-act detector could map these hedged constructions onto the same directive act as bare imperatives, using the UD tree rather than word lists.
  • I3 (answer): QA readers on fables and school tests could check that a candidate answer to a polar question contains a polarity commitment before trusting it.
  • I4 (world): The world model could keep speaker identity per turn so that an asker's own clarifications are not mistaken for answers when reading dialogues.
  • I5 (learn): Mova could follow the same pattern for weak domains: annotate a small seed set, retrain, and publish the seed as a reusable evaluation slice.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of TrentoTrento · Italy62 works · 14,868 citations🔬Fondazione Bruno KesslerTrento · Italy25 works · 43,498 citations

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 9,086 worksArtificial Intelligence · 8,889 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 worksInformation Systems · 523 works

🧭 Topics 1Research subjects and languages the work is about.Implicature · 72 works

Cite

Cite as Mohammed R. H. Qwaider, Abed Alhakim Freihat and Fausto Giunchiglia. 2017. TrentoTeam at SemEval-2017 Task 3: An application of Grice Maxims in Ranking Community Question Answers. Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/s17-2043

@inproceedings{qwaider2017trentoteam,
  title = {{TrentoTeam at SemEval-2017 Task 3: An application of Grice Maxims in Ranking Community Question Answers}},
  author = {Mohammed R. H. Qwaider, Abed Alhakim Freihat and Fausto Giunchiglia},
  booktitle = {Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017)},
  year = {2017},
  doi = {10.18653/v1/s17-2043},
  url = {https://doi.org/10.18653/v1/s17-2043}
}

References

8 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Rodrigo Agerri and German Rigau. 2016. Robust multilingual named entity recognition with shallow semi-supervised features. Artificial Intelligence 238:63 – 82. 2. For the rest of the answers, we compute the similarity between the question Q and the answer a i , where 0 ≤ SM qi ≤ n (n = |L|). ↗
  2. Simone Filice, Danilo Croce, Alessandro Moschitti, and Roberto Basili. 2016. Kelp at semeval-2016 task 3: Learning semantic relations between questions and answers. In Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval-2016). Association for Computational Linguistics, San Diego, California, pages 1116–1123. http://www.aclweb.org/anthology/S16-1172.3. Then, based on Grice maxims, the answers are rewarded or penalized as follows. a The answer a i is rewarded according to the number of entities, reference expressions, currency and numbers, imperatives, domain specific terms, and positive sentiment words. b On the other hand, a i is penalized according to the number of negative sentiment, ironic, and insulting words. ↗
  3. R. Frederking. 1996. Grice’s maxims: do the right thing. Proc. of AAAI SpringSymp. on Compl. Implicature: Computational Approaches to Interpreting and Generating Conversational Implicature . find
  4. H. P. Grice. 1975. Logic and conversation. In P. Cole and J. L. Morgan, editors, Syntax and Semantics: Vol. 3: Speech Acts, Academic Press, San Diego, CA, pages 41–58. 4. After rewarding and penalizing all answers, we then sort the list of answers according to their achieved scores in a descending order. Best answer is the first answer in the list and so on. ↗
  5. Preslav Nakov, Doris Hoogeveen, Lluı́s Màrquez, Alessandro Moschitti, Hamdy Mubarak, Timothy Baldwin, and Karin Verspoor. 2017. SemEval-2017 task 3: Community question answering. In Proceedings of the 11th International Workshop on Semantic Evaluation. Association for Computational Linguistics, Vancouver, Canada, SemEval ’17.
  6. Evaluating the algorithm on the same test set in the previous experiments, we get MAP=0.7151. The best system in SemEval 2016 (Filice et al., 2016) achieved MAP=79.19 as shown in Table 1. find
  7. System Baseline Experiment 3 Experiment 1 Experiment 2 SemEval-2016 Worst System Experiment 5 Experiment 4 Our System SemEval-2016 Best System Preslav Nakov, Lluı́s Màrquez, Alessandro Moschitti, Walid Magdy, Hamdy Mubarak, abed Alhakim Freihat, Jim Glass, and Bilal Randeree. 2016. Semeval-2016 task 3: Community question answering. In Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval-2016). Association for Computational Linguistics, San Diego, California, SemEval ’16, pages 525– 545. http://www.aclweb.org/anthology/S16-1083. ↗
  8. Joseph Turian, Lev-Arie Ratinov, and Yoshua Bengio. 2010. Word representations: A simple and general method for semi-supervised learning. In Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics, Uppsala, Sweden, pages 384–394. http://www.aclweb.org/anthology/P10-1040. *Table 1: Results of some community Question Answer Ranking approaches in SemEval 2016.* Results Our system obtained a rank 7 of 12 out of 13 participated systems and a MAP of 78.56. It beat http://alt.qcri.org/semeval2017/task3/ ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/qwaider-2017-trentoteam-semeval-2017-task-3/