🧑‍🔬 Jekaterina Novikova

Jekaterina Novikova works on natural language generation, especially systems that turn structured meaning representations into fluent text for dialogue applications. Their papers here ask what data such systems should learn from and how their output should be judged. The work relies on crowdsourced corpora in domains such as restaurants and hotels, and compares automatic measures of output quality with ratings collected from people. These concerns sit within the topics of Text generation and Evaluation & benchmarks.

The most cited of these papers, both from 2017, are the natural place to start. The E2E Dataset: New Challenges For End-to-End Generation presents a large crowdsourced corpus that pairs dialogue-act-style meaning representations with restaurant descriptions written by people. Earlier end-to-end generators had mostly been trained on small delexicalised collections that yielded rather formulaic sentences. To elicit more natural wording, crowd workers saw pictures rather than textual prompts, and the new corpus is contrasted with older ones such as BAGEL. The companion study, Why We Need New Evaluation Metrics for NLG, tests how closely word-overlap, semantic-similarity and grammar-based metrics such as BLEU track human judgements. It runs these tests on the output of several data-driven generators across restaurant and hotel corpora, with particular attention to systems that can produce ungrammatical text. Read together, the two papers examine both the training material for data-driven generation and the methods used to assess it.

Novikova is currently at Vanguard. Earlier bylines include Heriot-Watt University, where much of the work on generation and evaluation appeared, as well as the University of Bath and, more recently, Cohere. Co-authors on the papers here include Ondřej Dušek and Verena Rieser, with Amanda Cercas Curry also appearing.

Jekaterina Novikova has 2 papers in our library, in 2017, most of them on Text generation, Evaluation & benchmarks; often with Ondřej Dušek, Verena Rieser, Amanda Cercas Curry.