Elamur Research⌕
☰
home / papers / 2026 / Designing LLM Agents for User-Centered Language Service Selection

Designing LLM Agents for User-Centered Language Service Selection

Summary

Non-expert users who want to build language applications face a large, heterogeneous pool of services offered across many platforms. Choosing the right chain is hard because plain-language requests mix stated tasks with unstated quality wishes such as privacy, speed or reliability. Classic QoS-optimization methods expect explicit numeric constraints, so they cannot interpret these wishes. The authors build an LLM agent that turns a free-text request directly into a service composition. Without any fine-tuning, the agent follows a four-stage reasoning scheme guided by Few-Shot exemplars and Chain-of-Thought. It identifies the required functions, derives quality criteria from contextual hints, uses those criteria to filter candidate services, and settles on a final pipeline. Its decisions draw on a Service Quality Information Table compiled through LLM-based analysis of specifications, technical reports and papers. For evaluation, the team created a benchmark linking varied user requests to standardized service profiles with metadata and quality indicators. Compared with prompting-only baselines, the agent achieved consistently higher Precision, Recall and F1. This suggests that structured reasoning makes service selection more accurate and explainable for ordinary users.

Key points

  • The four reasoning steps are: extract functional requirements, induce non-functional evaluation axes, apply those axes as retrieval constraints, and choose the final composition.
  • The knowledge base has two parts: Few-Shot examples, which capture reasoning strategies that can be applied again across requests, and the Service Quality Information Table, which provides factual grounding about each service.
  • In the motivating example, a request about a "confidential" report image implies an OCR → translation → summarization pipeline plus a privacy constraint.
  • The work positions itself against LR infrastructures (META-SHARE, LAPPS Grid, Language Grid, European Language Grid, PANACEA, DKPro Core) and tool-use LLMs (ToolLLM, Gorilla). Unlike those tool-use LLMs, it needs no fine-tuning and adds reasoning about non-functional quality.
  • The excerpt provided does not include exact scores or the benchmark size. Planned future work covers robustness under ambiguity, real-time QoS monitoring, a larger benchmark, and the ordering of multi-service workflows.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.63317/59qnh9w46bv6

@inproceedings{ogawa2026designing,
  title = {{Designing LLM Agents for User-Centered Language Service Selection}},
  author = {Ogawa, Ryoichiro and Lin, Donghui and Uwano, Fumito},
  booktitle = {Proceedings of the Fifteenth Language Resources and Evaluation Conference},
  year = {2026},
  doi = {10.63317/59qnh9w46bv6},
  url = {https://doi.org/10.63317/59qnh9w46bv6}
}

References

22 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in Neural Information Processing Systems, 33:1877–1901. find
  2. Richard Eckart de Castilho and Iryna Gurevych. 2014. A broad-coverage collection of portable nlp components for building shareable analysis pipelines. In Proceedings of the Workshop on Open Infrastructures and Analysis Frameworks for HLT, pages 1–11, Reykjavik, Iceland. European Language Resources A… find
  3. Vahideh Hayyolalam and Ali Asghar Pourhaji Kazem. 2018. A systematic literature review on qos-aware service composition and selection in cloud environment. Journal of Network and Computer Applications, 110:52–74. find
  4. Ching-Lai Hwang and Kwangsun Yoon. 2012. Multiple attribute decision making: methods and applications a state-of-the-art survey. Springer Science & Business Media. find
  5. Nancy Ide, James Pustejovsky, Christopher Cieri, Eric Nyberg, Di Wang, Keith Suderman, Marc Verhagen, and Jonathan Wright. 2014. The language application grid. In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC’14), pages 22– 30, Reykjavik, Iceland. Europ… find
  6. Toru Ishida, Yohei Murakami, Donghui Lin, Takao Nakaguchi, and Masayuki Otani. 2018. Language service infrastructure on the web: The language grid. Computer, 51(6):72–81. find
  7. LangChain AI. 2022. Langchain. https://github.com/langchain-ai/langchain. Accessed: 2025-10-24. ↗
  8. Donghui Lin, Yohei Murakami, and Toru Ishida. 2018. A framework for multi-language service design with the language grid. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan. European Language Resources Association (ELRA). find
  9. Nelson F Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. 2024. Lost in the middle: How language models use long contexts. Transactions of the Association for Computational Linguistics, 12:157–173. find our card
  10. Shishir G. Patil, Tianjun Zhang, Xin Wang, and Joseph E. Gonzalez. 2023. Gorilla: Large language model connected with massive apis. arXiv preprint arXiv:2305.15334. find our card
  11. Stelios Piperidis. 2012. The meta-share language resources sharing infrastructure: Principles, challenges, solutions. In Proceedings of the 8th International Conference on Language Resources and Evaluation (LREC’12), pages 36–42, Istanbul, Turkey. European Language Resources Association (ELRA). find
  12. Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, et al. 2024. Toolllm: Facilitating large language models to master 16000+ real-world apis. In Proceedings of the Twelfth International Conference on Learning Representations (ICLR 2024). find our card
  13. Georg Rehm, Stelios Piperidis, Dimitris Galanis, Penny Labropoulou, Maria Giagkou, Miltos Deligiannis, Leon Voukoutis, Martin Courtois, Julian Moreno-Schneider, and Katrin Marheinecke. 2024. European language grid: One year after. In Proceedings of the 2024 Joint International Conference on Computat… find
  14. Significant Gravitas. 2023. Autogpt: Build, deploy, and run ai agents. https://github.com/Significant-Gravitas/AutoGPT. Accessed: 2025-10-24. ↗
  15. Antonio Toral, Pavel Pecina, Andy Way, and Marta Poch. 2011. Towards a user-friendly webservice architecture for statistical machine translation in the panacea project. In Proceedings of the 15th Conference of the European Association for Machine Translation (EAMT’11), pages 63–70, Leuven, Belgium. find
  16. Lei Wang, Chen Ma, Xueyang Feng, Zeyu Zhang, Hao Yang, Jingsen Zhang, Zhiyuan Chen, Jiakai Tang, Xu Chen, Yankai Lin, et al. 2024. A survey on large language model based autonomous agents. Frontiers of Computer Science, 18(6):186345. find our card
  17. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems, 35:24824–24837. find our card
  18. Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Beibin Li, Erkang Zhu, Li Jiang, Xiaoyun Zhang, Shaokun Zhang, Jiale Liu, Ahmed Hassan Awadallah, Ryen W. White, Doug Burger, and Chi Wang. 2023. Autogen: Enabling next-gen llm applications via multi-agent conversation. arXiv preprint arXiv:2308.08155… find our card
  19. Zheming Yang, Yuanhao Yang, Chang Zhao, Qi Guo, Wenkai He, and Wen Ji. 2024. Perllm: Personalized inference scheduling with edge-cloud collaboration for diverse llm services. arXiv preprint arXiv:2405.14636. — [arXiv:2405.14636](https://arxiv.org/abs/2405.14636) find
  20. Liangzhao Zeng, Boualem Benatallah, Marlon Dumas, Jayant Kalagnanam, and Anne H. H. Ngu. 2025. Qos-aware service composition: A retrospective. IEEE Transactions on Software Engineering, 51(03):836–841. find
  21. Liangzhao Zeng, Boualem Benatallah, Anne HH Ngu, Marlon Dumas, Jayant Kalagnanam, and Henry Chang. 2004. Qos-aware middleware for web services composition. IEEE Transactions on Software Engineering, 30(5):311–327. find
  22. Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, et al. 2023. A survey of large language models. arXiv preprint arXiv:2303.18223. find our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/ogawa-2026-designing-llm-agents-user-centered/