📄 AgentRouter: A Knowledge-Graph-Guided LLM Router for Collaborative Multi-Agent Question Answering

Summary

With so many LLMs and agent strategies available, it is hard to know which configuration will work best for a given QA task. Agents on the same backbone can differ widely in quality, and their rankings change across datasets. AgentRouter addresses this by treating multi-agent QA as a routing problem over a knowledge graph. Each QA instance becomes a heterogeneous graph that links the query, entities extracted from the context with spaCy, and a pool of 24 agents. A type-aware heterogeneous GNN, called RouterGNN, passes messages across these node types and outputs a weight distribution over agents. It is trained with KL divergence against each agent's measured performance rather than against a single "best agent" label. The final answer combines agent outputs by weighted voting, optionally limited to the top-K agents. Tested on 2Wiki, HotpotQA, NewsQA and TriviaQA, the router beats the best single agent and ensemble baselines, with the largest gains on multi-hop data. This matters because it shows that learned, context-aware routing can outperform heuristic voting or LLM-based judges.

Key points

• Motivating evidence: Six classic agent designs (raw, CoT, self-consistency, MAD, ReAct-Reflect, summary) were run on Llama-3-8B-Instruct and Mixtral-8×7B-Instruct over HotpotQA and NewsQA. Their F1 distributions were wide and their rankings unstable across tasks and backbones.
• Graph size: Each instance graph contains 1 query node and 24 agent nodes. On average it has about 111.6 entity nodes for HotpotQA and 46.0 for NewsQA, and HotpotQA graphs are much denser in entity–entity edges.
• Top-K ablation: On multi-hop datasets, keeping a moderate number of agents (roughly 5–10) works best. On single-hop QA, including more agents tends to give better and steadier results.
• Hyperparameters: Making the GNN deeper does not reliably raise F1. Larger hidden dimensions give more consistent improvements.
• Cross-dataset transfer is asymmetric:
  • Training on 2Wiki and testing on HotpotQA costs at most about 2% F1.
  • Training on TriviaQA and testing on 2Wiki loses roughly 29–38% F1.
  • This suggests the router needs training data from the target task.