ETOM: A Five-Level Benchmark for Evaluating Tool Orchestration within the MCP Ecosystem

Dong, Jia-Kai; Huang, I-Wei; Wu, Chun-Tin; Tsai, Yi-Tien

doi:10.18653/v1/2026.findings-eacl.75

ETOM: A Five-Level Benchmark for Evaluating Tool Orchestration within the MCP Ecosystem

Jia-Kai Dong^*, I-Wei Huang^*, Chun-Tin Wu, Yi-Tien Tsai

National Taiwan University

^*Equal contribution

Findings of EACL 2026

Paper arXiv Code Poster

Diagram comparing three evaluation paradigms. Traditional: a retriever over a flat API space judged by an LLM. Early MCP work: an LLM with a fixed retriever over an MCP ecosystem, judged by an LLM. ETOM: an end-to-end agent with its own architecture and retrieval strategy, working over tools grouped into MCP servers, scored with EM and F1 against a ground-truth dependency graph that uses equal function sets. — ETOM evaluates the whole agent, retriever and reasoner together, inside a hierarchical MCP ecosystem of 491 servers and 2,375 tools, and scores its plan with EM and F1 instead of an LLM judge.

Abstract

We introduce ETOM, a five-level benchmark for evaluating multi-hop, end-to-end tool orchestration by LLM agents within a hierarchical Model-Context Protocol (MCP) ecosystem. Existing benchmarks often assess tools in isolation, overlooking challenges such as functional overlap and cross-server orchestration, which can lead to overly optimistic evaluations. ETOM addresses these gaps by constructing ground truth through “equal function sets”, enabling objective metrics such as F1 score and reducing reliance on LLM-as-a-judge evaluation. Its five-level curriculum systematically tests agent capabilities, from single-tool orchestration to complex cross-server planning, as well as robustness to out-of-scope requests. Experiments reveal that rigid hierarchies can hinder performance without co-designed strategies, and even state-of-the-art agents exhibit systemic weaknesses in robustness. ETOM provides a diagnostic framework to expose these limitations and guide the development of more capable and efficient tool-using agents.

A Five-Level Curriculum

Five levels from low to high complexity. Single-turn: Level 1, explicit single-tool; Level 2, implicit tool selection. Multi-turn: Level 3, intra-server chains; Level 4, cross-server chains. Level 5, robustness to out-of-scope requests. — **Figure 1.** The curriculum moves from single-tool tasks (L1–L2) to sequential orchestration within one server (L3), compositional chaining across servers (L4), and rejection of requests that no tool can serve (L5).

Each level targets a distinct reasoning challenge. L1 queries name the target platform or tool; L2 queries describe the need in natural language, so the agent must pick among functionally equivalent tools. L3 asks for a chain of dependent calls inside one server, built from data-flow graphs over inferred output schemas. L4 samples two to four servers from distinct categories and asks for a coherent workflow across them. L5 holds 91 out-of-scope queries covering 31 capability gaps, such as physical-world interaction and real-time sensory processing, where the correct answer is to refuse.

Table 1. Tasks per level, with the average number of tools in the ground-truth plan and the average number of servers it spans.

Level	Key challenge	Tasks	Avg. plan length	Avg. servers	Metric
L1	Direct retrieval	781	1.00	1.00	Exact Match (EM)
L2	Context-aware retrieval	773	1.00	1.00	EM against set
L3	Intra-server chaining	327	2.87	1.00	Node set EM, F1
L4	Cross-server chaining	103	3.83	3.78	Node set EM, F1
L5	Robust rejection	91	0.00	0.00	Exact Rejection Match

Example tasks

L1: “On Semgrep, run semgrep_scan on the file ‘vulnerable_code.js’ with the configuration file ‘semgrep_config.yaml’ to detect vulnerabilities and return findings in JSON format.” Expected: semgrep_scan on the Semgrep MCP Server.
L2: “Convert this image to JPEG format with 90% quality.” Either convert_to_jpeg or image_format_converter is correct, because both belong to one equal function set.
L3: “Analyze the code in src/main.py for security vulnerabilities, generate a detailed report, and send it to the development team.” Expected: analyze_code → generate_report → send_notification, each step depending on the one before.
L4: “Deploy the latest version of my web application to production, run security scans, and notify the team about the deployment status.” Expected: deploy_app (Deployment Server) → run_security_scan (Security Server) → send_notification (Notification Server).
L5: “Turn on the lights in my living room and adjust the temperature to 22°C.” Expected: {server: "no", tool: "no"}, since no digital MCP tool can act in the physical world.

Corpus and Equal Function Sets

ETOM’s tools come from the glama.ai MCP server registry. The authors scraped the top 50 servers in each category, then removed servers that do not add a genuine external capability: trivial tools that a plain LLM already covers (such as calculators), meta-tools that augment the agent’s own memory or reasoning, and template servers that exist only as developer examples. The filtered corpus has 491 servers exposing 2,375 tools, each with a name, description, and input schema.

Bar chart of the number of servers in each of 30 categories. Most categories hold between 21 and 29 servers; python has 13 and remote has 9. — **Figure 2.** Servers per category across the 30 categories of the corpus; a server can carry more than one category label.

Handling functional overlap

Real ecosystems contain many tools that do the same job, so a benchmark that accepts only one “correct” tool penalises valid answers, and one that avoids overlap is unrealistic. ETOM groups equivalent tools into equal function sets (EFS) and accepts any member of a set. The sets come from a round-trip process. Bottom-up, each tool retrieves semantically similar tools (Qwen3-Embedding-0.6B, similarity above 0.8), an LLM verifies each pair, and verified pairs are merged with Union–Find into 145 candidate sets. Top-down, each Level 2 query retrieves its top-10 tools, an LLM selects every tool that can satisfy it, and these choices are cross-checked against the bottom-up sets; human validators then settle borderline and platform-specific cases. The result is 95 high-confidence sets that serve as ground truth for L2 and as interchangeable nodes in L4 plans, so plans can be scored with deterministic EM and F1 rather than an LLM judge.

Two-phase diagram. Phase 1, bottom-up: an LLM groups tools from the MCP tool pool that it judges functionally equal, for example HackMD list_note with Notion list_note. Phase 2, top-down: for the query 'What are the notes I have in HackMD?', an LLM picks feasible tools, HackMD list_note and HackMD list_document. Human verification removes Notion list_note, leaving the two HackMD tools as ground truth. — **Figure 3.** Building an equal function set for “What are the notes I have in HackMD?”: the bottom-up phase groups similar tools across servers, the top-down phase finds every tool that can answer the query, and human verification removes the Notion tool, which cannot read HackMD notes (from the poster).

Results

Four orchestration architectures were evaluated: ReAct (a generative baseline with no retriever), ToolShed (flat retrieval with dense search, query expansion, and reranking), MCP-Zero (hierarchical retrieval, first over servers and then over tools), and a Hybrid that applies MCP-Zero filtering before ToolShed search. Each was paired with several foundation models. Retrieval clearly helps: averaged over the five levels, ToolShed with Qwen3-4B scores 60.35 against 34.81 for ReAct with the same model. On L4, the best configuration, ToolShed with GPT-4.1, reaches 55.06 F1.

Table 2. Performance and normalized latency (N-Lat.) for every architecture and foundation model. L1, L2, and L5 report Exact Match (%); L3 and L4 report F1 (%). Latency is relative to MCP-Zero with Meta-Llama-3-8B-Instruct; it is omitted (–) for GPT-4.1, whose API latency is not comparable with locally run models.

Arch.	Foundation model	L1 EM	N-Lat.	L2 EM	N-Lat.	L3 F1	N-Lat.	L4 F1	N-Lat.	L5 EM	N-Lat.
MCP-Zero	Qwen3-4B-Instruct-2507	69.01	0.89	43.02	0.59	46.85	5.64	30.42	6.50	74.72	0.22
MCP-Zero	Qwen3-8B	65.68	3.14	31.56	3.60	48.16	3.87	33.95	3.73	62.63	3.86
MCP-Zero	Ministral-8B-Instruct-2410	46.99	1.29	23.67	1.24	39.65	1.43	25.99	1.71	51.60	0.70
MCP-Zero	GPT-4.1	63.38	–	31.82	–	53.12	–	34.20	–	80.29	–
MCP-Zero	Gemma-3-12B-IT	70.55	1.73	33.50	1.56	49.70	1.71	32.43	1.99	65.93	0.75
MCP-Zero	Microsoft Phi-4	38.79	3.54	19.01	3.37	53.97	3.31	33.98	3.42	69.23	2.58
MCP-Zero	Meta-Llama-3-8B-Instruct	48.08	1.00	19.25	1.00	38.04	1.00	25.70	1.00	39.56	1.00
ToolShed	Qwen3-4B-Instruct-2507	72.72	3.31	51.87	3.58	69.41	5.38	37.45	5.26	70.32	2.22
ToolShed	Qwen3-8B	80.65	11.63	53.29	11.33	77.45	15.74	44.96	15.31	76.92	11.25
ToolShed	Ministral-8B-Instruct-2410	64.66	7.86	43.46	8.24	82.36	10.15	50.98	8.68	45.05	4.70
ToolShed	GPT-4.1	69.91	–	49.02	–	82.55	–	55.06	–	81.31	–
ToolShed	Gemma-3-12B-IT	80.66	7.89	56.79	7.87	73.96	11.80	47.29	12.19	60.43	6.07
ToolShed	Microsoft Phi-4	38.66	15.12	28.71	14.54	81.85	14.52	49.67	14.81	67.03	11.93
ToolShed	Meta-Llama-3-8B-Instruct	57.03	5.22	42.12	6.02	74.34	5.22	50.77	5.68	27.40	6.65
ReAct	Qwen3-4B-Instruct-2507	41.43	8.80	19.76	8.65	33.61	6.75	6.74	10.54	72.50	12.18
Hybrid	Qwen3-4B-Instruct-2507	68.62	3.61	60.20	3.68	56.91	3.63	26.78	2.37	74.72	1.58
Hybrid	Meta-Llama-3-8B-Instruct	48.08	4.74	39.79	4.39	50.89	6.07	21.61	5.48	14.28	4.94

Scatter plot of average score against normalized latency on a log scale. MCP-Zero configurations, drawn as crosses, sit at low latency between about 1x and 3.5x with scores from about 34 to 53. ToolShed configurations, drawn as circles, sit at about 4x to 14x latency with scores from about 50 to 67. GPT-4.1 is drawn as a square. — **Figure 4.** Performance against efficiency for each model and architecture. Crosses are MCP-Zero, circles ToolShed, squares GPT-4.1 through the API; colour marks the model family. The x-axis is normalized latency on a log scale.

The configurations split into two clusters. MCP-Zero occupies the low-latency region (about 1.0–3.0× the baseline), while ToolShed occupies the high-performance, high-latency region (about 5.0–15.0×): hierarchical filtering buys efficiency by giving up comprehensiveness, and flat retrieval buys accuracy at a steep computational cost.

How much to retrieve

Line chart of ToolShed performance on Levels 1 to 4 as the number of retrieved tools rises from 5 to 25. Level 3 F1 rises slowly from about 74 to 77. Level 1 EM stays near 56 to 58. Level 4 F1 peaks at 20 tools near 52, then drops to about 46 at 25. Level 2 EM falls from about 43 at 15 tools to about 34 at 25. — **Figure 5.** ToolShed with Meta-Llama-3-8B-Instruct as the number of retrieved tools (tool_top_k) varies. L1 and L2 show EM, L3 and L4 show F1.

An ablation of ToolShed shows that retrieval breadth matters differently at each level. More candidates steadily help intra-server planning (L3), and cross-server planning (L4) peaks at 20 retrieved tools (52.11 F1), but the same breadth hurts L2, where extra, similar-looking tools distract the model and EM falls to 33.72 at 25 tools.

What ETOM Reveals

Hierarchy isn’t free

Hierarchical retrieval is often assumed to be better. MCP-Zero does deliver large efficiency gains (up to 5.76×), but it pays a significant, uneven accuracy cost on multi-step orchestration (L3/L4), where its rigid server-then-tool pruning appears to limit the model. Simply combining the two paradigms is not enough either: the Hybrid architecture is competitive with Qwen3-4B (57.45 average) but drops to 34.93 with Llama-3-8B, against 50.33 for ToolShed with the same model.

No universal best model or architecture

Performance emerges from the pairing of model and architecture. Qwen models are strongest at direct, single-step retrieval (L1/L2), which suits MCP-Zero’s efficient pruning; Llama’s multi-step reasoning shows only when ToolShed’s broad, high-recall retrieval gives it a large enough candidate set. Model selection and orchestration design therefore need to be optimised together.

Failure modes

Contextual drift. The most common Level 4 failure: when moving across server boundaries, agents lose intermediate tool outputs or explicit user constraints, and later tool choices drift away from the user’s intent.
Premature decomposition. Agents split simple, single-step queries into unnecessary sub-tasks, adding hallucinated dependencies and extra points of failure.
Complexity decay. On Level 3, ToolShed with Qwen3-4B falls from 70.2% F1 on 2-step tasks to 43.1% F1 on 7-step tasks, and the pattern holds across foundation models.
A safety gap at Level 5. None of the evaluated orchestrators has an explicit check for requests the tool set cannot satisfy; correct rejection comes from the backbone model’s own reasoning, so it varies widely, from 14.28% to 81.31% EM.

The authors point to hierarchy-aware reasoning, task decomposition that carries global context between steps, hybrid systems that switch between flat and hierarchical retrieval by query complexity, and dedicated out-of-scope detection as directions for future agents.

Acknowledgments

This work was primarily conducted during an internship at Delta Electronics, Inc. We gratefully acknowledge their support, which made this research possible. We would like to express our sincere gratitude to our mentors—Jimmy, TinTin, Alex, Leo, and Kun-Tai—for their invaluable guidance and insightful discussions. We also thank the anonymous reviewers for their constructive feedback and suggestions. Finally, our gratitude extends to the broader MCP community and the glama.ai platform, whose efforts in consolidating tool servers provided the essential foundation for our benchmark corpus.

BibTeX

@inproceedings{dong-etal-2026-etom,
    title = "{ETOM}: A Five-Level Benchmark for Evaluating Tool Orchestration within the {MCP} Ecosystem",
    author = "Dong, Jia-Kai  and
      Huang, I-Wei  and
      Wu, Chun-Tin  and
      Tsai, Yi-tien",
    editor = "Demberg, Vera  and
      Inui, Kentaro  and
      Marquez, Llu{\'i}s",
    booktitle = "Findings of the {A}ssociation for {C}omputational {L}inguistics: {EACL} 2026",
    month = mar,
    year = "2026",
    address = "Rabat, Morocco",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2026.findings-eacl.75/",
    doi = "10.18653/v1/2026.findings-eacl.75",
    pages = "1453--1488",
    ISBN = "979-8-89176-386-9",
    abstract = "We introduce ETOM, a five-level benchmark for evaluating multi-hop, end-to-end tool orchestration by LLM agents within a hierarchical Model-Context Protocol (MCP) ecosystem. Existing benchmarks often assess tools in isolation, overlooking challenges such as functional overlap and cross-server orchestration, which can lead to overly optimistic evaluations. ETOM addresses these gaps by constructing ground truth through ``equal function sets'', enabling objective metrics such as F1 score and reducing reliance on LLM-as-a-judge evaluation. Its five-level curriculum systematically tests agent capabilities, from single-tool orchestration to complex cross-server planning, as well as robustness to out-of-scope requests. Experiments reveal that rigid hierarchies can hinder performance without co-designed strategies, and even state-of-the-art agents exhibit systemic weaknesses in robustness. ETOM provides a diagnostic framework to expose these limitations and guide the development of more capable and efficient tool-using agents."
}