Elamur Research⌕
☰
home / papers / 2026 / DIRECT: Directional Relevance in Conversational Trajectories

DIRECT: Directional Relevance in Conversational Trajectories

Paper · 2026 · Anshuman Mourya, Rajdeep Mukherjee, Prerna Jolly et al.

Summary

Users of conversational assistants frequently struggle with opening a dialogue or deciding which question to pose next. This Amazon industry paper treats Next Question Suggestion (NQS) as a fast retrieval task rather than LLM generation. Generation is costly, slow, and can produce unsafe or unanswerable questions, so DIRECT instead selects candidates from a pre-built Question Bank.

The key difficulty is directionality. A good suggestion should be related to the query but not a paraphrase of it, and it should move the dialogue forward rather than ask about background information (a "prior"). DIRECT addresses this in three steps:

  • domain-adaptive MLM pre-training,
  • a dual-encoder retriever fine-tuned hierarchically so that relevant pairs fall between similar and irrelevant ones in similarity,
  • a cross-encoder re-ranker trained to tell follow-ups from priors.

The system is evaluated on a proprietary dataset and on FQ-Bank. An LLM judge estimates click-through rate. DIRECT beats an off-the-shelf BGE-Large retriever while staying well within a 1-second latency budget. This makes it a practical recipe for low-latency suggestion features in commercial assistants.

Key points

  • Query–suggestion pairs fall into four categories: similar, follow-up, prior, and irrelevant. Only follow-ups should be shown to users, and priors serve as hard negatives for the re-ranker.
  • Pre-training uses NER-guided masking: function words are filtered out, and 25% of the remaining domain-specific tokens are masked.
  • Hierarchical fine-tuning improves Precision@Recall on the 3-way task by up to 0.08 on the proprietary data and up to 0.16 on FQ-Bank. On follow-up vs. prior detection, the gains are up to 0.12 and 0.1.
  • In the Claude Sonnet 4.5 LLM-as-a-judge evaluation, predicted CTR rises 4% without the re-ranker and 6% with it, relative to vanilla BGE-Large. p90 latency is roughly 1.3 ms and 27 ms respectively; this pairing is inferred from Table 5, whose layout was garbled in extraction.
  • Limitations: the system has not yet been deployed to production, depends on annotated data, and was tested on English only.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Indian Institute of Technology HyderabadSangāreddi · India4 works · 108 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.Dialogue systems · 624 works

Cite

Cite as Anshuman Mourya, Rajdeep Mukherjee, Prerna Jolly, Vinayak S Puranik and Sivaramakrishnan R Kaveri. 2026. DIRECT: Directional Relevance in Conversational Trajectories. Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 5: Industry Track).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.eacl-industry.71

@inproceedings{mourya2026direct,
  title = {{DIRECT: Directional Relevance in Conversational Trajectories}},
  author = {Anshuman Mourya, Rajdeep Mukherjee, Prerna Jolly, Vinayak S Puranik and Sivaramakrishnan R Kaveri},
  booktitle = {Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 5: Industry Track)},
  year = {2026},
  doi = {10.18653/v1/2026.eacl-industry.71},
  url = {https://doi.org/10.18653/v1/2026.eacl-industry.71}
}

References

15 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Jianlyu Chen, Shitao Xiao, Peitian Zhang, Kun Luo, Defu Lian, and Zheng Liu. 2024. M3-embedding: Multi-linguality, multi-functionality, multi-granularity text embeddings through self-knowledge distillation. In Findings of the Association for Computational Linguistics: ACL 2024, pages 2318–2335, Bang…
  2. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of Yubin Ge, Ziang Xiao, Jana Diesner, Heng Ji, Karrie Karahalios, and Hari Sundaram. 2023. What should I ask: A knowledge-driven approach for follow-up questions generation in conversational surveys. In Proce… find
  3. Ehsan Kamalloo, Xinyu Zhang, Odunayo Ogundepo, Nandan Thakur, David Alfonso-Hermelo, Mehdi Rezagholizadeh, and Jimmy Lin. 2023. Evaluating embedding apis for information retrieval. Preprint, arXiv:2305.06300. find
  4. Chankyu Lee, Rajarshi Roy, Mengyao Xu, Jonathan Raiman, Mohammad Shoeybi, Bryan Catanzaro, and Wei Ping. 2025. Nv-embed: Improved techniques for training llms as generalist embedding models. Preprint, arXiv:2405.17428.
  5. Chaofan Li, MingHao Qin, Shitao Xiao, Jianlyu Chen, Kun Luo, Yingxia Shao, Defu Lian, and Zheng Liu. 2024a. Making text embedders few-shot learners. Preprint, arXiv:2409.15700. find
  6. Haitao Li, Qian Dong, Junjie Chen, Huixue Su, Yujia Zhou, Qingyao Ai, Ziyi Ye, and Yiqun Liu. 2024b. Llms-as-judges: A comprehensive survey on llm-based evaluation methods. Preprint, arXiv:2412.05579.
  7. Jianyu Liu, Yi Huang, Sheng Bi, Junlan Feng, and Guilin Qi. 2025. From superficial to deep: Integrating external knowledge for follow-up question generation using knowledge graph and LLM. In Proceedings of the 31st International Conference on Computational Linguistics, pages 828–840, Abu Dhabi, UAE.… find
  8. Yan Meng, Liangming Pan, Yixin Cao, and Min-Yen Kan. 2023. FollowupQG: Towards information-seeking follow-up question generation. In Proceedings of the 13th International Joint Conference on Natural Language Processing and the 3rd Conference of the Asia-Pacific Chapter of the Association for Computa… find
  9. Luke Merrick, Danmei Xu, Gaurav Nuti, and Daniel Campos. 2024. Arctic-embed: Scalable, efficient, and accurate text embedding models. arXiv preprint arXiv:2405.05374. — [arXiv:2405.05374](https://arxiv.org/abs/2405.05374) find
  10. Jianmo Ni, Gustavo Hernandez Abrego, Noah Constant, Ji Ma, Keith Hall, Daniel Cer, and Yinfei Yang. 2022. Sentence-t5: Scalable sentence encoders from pre-trained text-to-text models. In Findings of the Association for Computational Linguistics: ACL 2022, pages 1864–1874, Dublin, Ireland. Associatio…
  11. Nils Reimers and Iryna Gurevych. 2019. Sentence-BERT: Sentence embeddings using Siamese BERT-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3982–3992,…
  12. Christopher Richardson, Sudipta Kar, Anjishnu Kumar, Anand Ramachandran, Zeynab Raeesy, Omar Khan, and Abhinav Sethy. 2023. Learning to retrieve engaging follow-up queries. In Findings of the Association for Computational Linguistics: EACL 2023, pages 2009–2016, Dubrovnik, Croatia. Association for C… find
  13. Aivin V. Solatorio. 2024. Gistembed: Guided in-sample selection of training negatives for text embedding fine-tuning. Preprint, arXiv:2402.16829. — [arXiv:2402.16829](https://arxiv.org/abs/2402.16829) find
  14. Hongjin Su, Weijia Shi, Jungo Kasai, Yizhong Wang, Yushi Hu, Mari Ostendorf, Wen-tau Yih, Noah A. Smith, Luke Zettlemoyer, and Tao Yu. 2023. One embedder, any task: Instruction-finetuned text embeddings. In Findings of the Association for Computational Linguistics: ACL 2023, pages 1102–1121, Toronto…
  15. Zhichao Xu, Fengran Mo, Zhiqi Huang, Crystina Zhang, Puxuan Yu, Bei Wang, Jimmy Lin, and Vivek Srikumar. 2025. A survey of model architectures in information retrieval. Preprint, arXiv:2502.14822. — [arXiv:2502.14822](https://arxiv.org/abs/2502.14822) find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/mourya-2026-direct-directional-relevance-conversational-trajectories/