Your Agent Doesn't Need 138 Tools: Picking MCP Tools with Semantic Search and Jev

An agent connected to many MCP servers sends every tool definition with every model call. I tested sending only a small set per request, chosen by semantic search or by Jev, on MCP-Atlas tasks. Both more than halved the median cost per task. Semantic search was the faster and cheaper of the two, and Jev’s answers were on par with loading every tool in the context.

Connecting an agent to MCP servers is easy. Connecting it to many of them gets expensive quickly. Each server brings its own tools, and each tool’s name, description and parameter schema is sent to the model on every call, whether the request needs it or not. In this experiment, 20 MCP servers came to 138 tools and about 30,000 tokens of tool definitions per model call, before the model had even read a word of the user’s question.

So the problem is clear: the more capable you make the agent, the more each call costs and the more tools the model has to sort through.

At the same time, I wanted to try Jev from TypeSafeAI. It’s one of the new, popular things in AI right now, and most of the examples I found were vague. I didn’t want to build another generic demo. I wanted a real problem I could measure, where the result would be useful whichever way it came out.

Tool picking matched exactly what I was looking for. Deciding which tools can help with a request is a question about meaning, and Jev is built to answer typed questions like that about a piece of context. There’s also a clear baseline to compare against: embeddings plus keyword search, usually referred to as “semantic search”, which is what most people build first.

The idea came from Claude’s built-in tool search. In the Claude API (directly or on Amazon Bedrock) you can mark tools as deferred, and Claude searches for the ones it needs instead of seeing every definition upfront. The MCP client best practices recommend the same approach under the name “Progressive Tool Discovery”. I wanted to see how well that idea works when the selection is done via semantic search or via Jev.

To make the results mean something, I needed a benchmark. MCP-Atlas from Scale AI fits well:

• It runs real MCP servers in one Docker container: Wikipedia, arXiv, PubMed, clinical trials, OpenStreetMap, weather, filesystem, git, code runners and more.
• Its tasks are written by people and never name a tool. “Which food type in my local food consumption data…” doesn’t say “use filesystem_read_text_file”.
• Every task comes with answer facts and a reference solution, so answers can be graded against something fixed.

Atlas has 36 servers total, but only 20 run without API keys. These 20 servers provide 138 tools combined, which can solve 89 of the 500 public tasks. They need a median of 4 tools and 3 answer facts each.

That gave me one question to answer:

Can you give the model 12 tools instead of 138 without losing quality, and which picker does it best?

Why 12? MCP-Atlas’s own hand-picked tool lists hold a median of 12 of the available tools for these 89 tasks, so 12 is roughly what a person would choose. It also cuts the tool definitions from about 30,000 tokens to about 4,000. I fixed the number before any run and gave both pickers the same budget, so they compete on equal terms.

Three setups#

I compared three ways of giving the model its tools, from the most obvious to the newest.

1. All tools#

Connect every server and give the model all 138 tools on every call. This is what you get by default, and it’s the quality baseline: every tool is always in view, so the model never misses one because it wasn’t shown.

2. Semantic search (embeddings + BM25)#

Before each request, rank the tools by how well their text matches the request and load the top 12. The embeddings come from Amazon Nova 2 multimodal embeddings on Bedrock, with 1,024 dimensions; Nova beat Titan Text Embeddings V2 in my first tests. Embeddings alone miss exact words, so the ranking fuses them with BM25 keyword scores. This is the standard retrieval recipe, and it’s a strong baseline. Many retrieval setups add a third step: a reranker model that reads the request next to each tool and reorders the list. Jev could also fill that role, rescoring only semantic search’s top results instead of all 138 tools. At this stage, both were unnecessary: I wanted to compare the simplest version of each picker.

3. Jev#

Same budget of 12 tools, but Jev scores them. Jev takes a state (the user’s request) and answers typed questions about it with calibrated probabilities. I pinned the version (jev-1.13.0) so results stay comparable.

Jev can ask this in two ways:

• Noul asks one yes/no question per tool: “Would calling tool help an assistant answer request?” The criteria spell out what yes and no mean: yes if the tool “looks up or returns information that request asks for, or is a step toward getting it”, no if it’s “for a different service, resource, or task”. Each tool gets its own probability, so several tools can score high at the same time.
• Choice asks one pick-one question over all 138 tools: “Which tool should an assistant call first to answer request?” The probabilities add up to 1, so tools compete for a single spot.

What each picker sends and gets back#

The two pickers look similar from the outside: a request goes in, 12 tools come out. They move very different data to get there.

Embeddings: the tools are processed once, and each request is a single small call#

The tools are embedded ahead of time, not per request. Each tool becomes one short text: its name, the first 600 characters of its description, and its parameter names. Nova takes one text per call, so indexing is 138 Bedrock calls (8 in parallel), each returning a vector of 1,024 numbers. The vectors are cached on disk and only recomputed when a tool’s text changes. The BM25 keyword index is built locally from the full descriptions.

For each request, only the request needs to be embedded: one Bedrock call of about 80 tokens, which returns one vector. Everything else happens locally:

1. Compare the request’s vector with each of the 138 tool vectors (a dot product).
2. Score the request’s words against each tool with BM25.
3. Fuse the two rankings with reciprocal rank fusion: each tool gets 1 / (60 + its position) from each list, and the sums decide the order.
4. Load the top 12.

The request and the tools are never read together. Each is turned into a vector separately, and the picker only compares vectors. However, it also means a tool can rank high because its description shares words with the request even when it can’t help, and the right tool can rank low because it shares none.

Jev: no index, and every pick is one request that carries everything#

With Noul, each pick is one call to Jev’s API that contains the state and all 138 questions, one per tool, keyed by the tool’s name. Each question includes that tool’s name, the first 600 characters of its description, its parameter names, the question and the yes/no criteria. In simplified form:

state = {"request": "Which food type in my local food consumption data…"}
questions = {
    tool.name: Noul(
        instructions={
            "tool": {"name": tool.name, "description": tool.description[:600], "parameters": tool.parameter_names},
            "question": "Would calling `tool` help an assistant answer `request`?",
        },
        criteria=NoulCriteria(true="…", false="…"),
    )
    for tool in tools  # all 138 in the same request
}
response = client.system_one(state=state, questions=questions, model="jev-1.13.0")
scores = {name: answer.noul for name, answer in response.nouls.items()}
top_12 = sorted(scores, key=scores.get, reverse=True)[:12]

That’s about 23,300 input tokens per pick, within Jev’s documented limit of 64k tokens per request. For a larger set of tools, the code splits the questions into several requests and sends them in parallel. Jev returns one probability per tool, and the picker keeps the 12 highest.

Choice takes a different approach. Instead of 138 separate yes/no questions, it sends the same state with a single multiple-choice question, “Which tool should an assistant call first to answer request?”, with the 138 tools as the options. Jev returns one probability distribution across all of them that adds up to 1, and the picker keeps the 12 most likely. Descriptions are shortened only if all the options wouldn’t fit; with these 138 tools they didn’t need to be.

Comparing the two approaches#

Both return a probability for every tool, so either list can be sorted to take the top 12. What differs is what those probabilities mean. A Choice question has exactly one right answer, and its probabilities add up to 1. That’s why its question asks for the first tool: “which tools would help?” has several right answers, and a one-answer question can’t express that. As a result, Choice’s probabilities measure “is this the first step?”, not “is this useful?”. If one tool is the obvious first step, it takes most of the probability, and tools needed later, such as running code after reading a file, get small shares even when the task can’t be done without them. Noul asks about each tool on its own, so four useful tools can all score close to 1. That matches what the top 12 should contain, so the real run used Noul.

So where embeddings compare two vectors that were made separately, Jev reads the request next to each tool and judges them as a pair. That costs about 290 times as many input tokens per pick and takes longer, but it means a tool is judged by whether it would help with the request, not by how similar its wording is. Because Jev’s scores are probabilities, a fixed cutoff such as “load every tool above 0.5” means the same thing on every request. That lets the agent load a different number of tools for each request, for example 3 for a simple question and 10 for a complex one. Embedding scores don’t allow that: a similarity of 0.6 can be a good match for one request and a poor one for another, so one cutoff doesn’t work across requests. I still used a fixed 12 tools so both pickers got the same budget.

The find_tools fallback#

Both pickers also load a find_tools tool. If the model needs something it can’t see, it describes the capability (“read a CSV file”) and the same picker adds matching tools. So a bad first pick can be fixed, but each fix costs an extra round trip.

Model and run settings#

• Agent: Strands Agents on Amazon Bedrock with Claude Sonnet 4.6 at temperature 0, max_tokens 32,000, and a limit of 100 tool calls per answer. I chose Sonnet 4.6 because it still accepts a temperature setting; newer models reject sampling parameters, and temperature 0 keeps the three setups as comparable as possible.
• Judge: also Sonnet 4.6 at temperature 0, using Atlas’s own per-fact rubric. Each answer scores 1 (fulfilled), 0.5 (partly) or 0 (missed). Coverage is the mean, and a task passes at 75% coverage. Every answer is graded the same way, including the all-tools ones.
• Fairness: each answer uses a fresh one-shot agent. The sandbox caches tool results, so the same tool call returns the same data in all three setups. Bedrock’s prompt cache only helps a request that starts with exactly the same tool definitions as an earlier one. The all-tools setup sends 138 tools and the pickers send 12, so the pickers can never reuse its cache. The two pickers can pick the same 12 tools for a task, so the run waits 5.5 minutes before the second picker, long enough for semantic search’s cache entries to expire.

All 89 tasks were answered three times, once per setup.

Quality#

Jev performed similarly to loading every tool: 60 passes against 61, and mean coverage 2.6 percentage points lower. Coverage is the share of a task’s answer facts that the answer got right, with half credit for partly right ones, averaged over all 89 tasks.

Semantic search clearly fell behind: 52 passes and coverage 10.4 points lower.

The first pick explains most of the difference. On average, Jev’s first 12 tools included 59% of each task’s reference tools; semantic search’s included 32%. It also shows in the fallback: the semantic-search agent reached for find_tools on 36 tasks, the Jev agent on 12.

Is the gap real, or luck in which tasks were tested?#

89 tasks is a small sample. A different set of 89 tasks would give different gaps. To estimate how much they could move, I used a bootstrap:

1. For each of the 89 tasks, subtract all tools’ coverage from the picker’s coverage. That gives 89 differences.
2. Draw 89 of those differences at random, allowing the same task to be drawn more than once, and average them. This mimics testing on a different set of 89 tasks.
3. Repeat step 2 a total of 10,000 times, which gives 10,000 averages.
4. Sort the averages and drop the lowest 2.5% and the highest 2.5%. The middle 95% is the range the gap would likely fall in on a different set of tasks.

A negative number means the picker scored lower than all tools; a positive number means it scored higher.

• Jev: −7.9 to +2.5 points. The range includes zero, so these 89 tasks can’t show whether Jev is worse than loading every tool, equal to it, or better. Jev could plausibly be up to 7.9 points worse or up to 2.5 points better.
• Semantic search: −17.0 to −4.4 points. The whole range is below zero, and 9,999 of the 10,000 averages were below zero, so semantic search’s lower score is not luck.

When the right tools share no words with the request#

The gap isn’t spread evenly. On the other 39 tasks, mostly about public information (museums, web search, Wikipedia, books, maps and so on), all three setups scored 72–75% coverage. On the 50 tasks about the user’s local files, all tools scored 85%, Jev 83%, and semantic search 69%.

One task illustrates it best. It asks: “Which food type in my local food consumption data has the most satisfaction ratings of 5, and which beverage types accompany it when the rating is 5?” The reference solution lists the allowed directories, lists a directory, reads a text file and runs code. Apart from words like “the” and “and”, none of those four tools shares a word with the request. Semantic search’s 12 tools held none of the four. Instead they included map, weather and meeting-point tools whose descriptions share words like “type” and “data”, and that answer covered none of the answer facts. Jev’s 12 held three of the four, including the file-reading and code tools, and that answer covered all of them. On seven tasks, the semantic-search agent answered without calling any tool at all.

Embeddings and keywords measure how similar the words are. Jev’s question asks whether the tool would help. When the words point one way and the meaning another, that difference decides the result.

Cost#

Both pickers cut tool tokens per model call by 85–88% and the median cost per task by 56% ($0.20 → $0.09).

The total bill fell less: 30% for semantic search and 38% for Jev. A few answers drive much of the cost: the 10 most expensive make up 38% of the bill with all tools, 54% with semantic search and 68% with Jev. They read whole directory trees, open many files at once, or loop through 50–95 model calls. Most of their tokens are file contents and long conversations rather than tool definitions, so picking tools saves proportionally less on them. Savings per call don’t turn directly into savings on the bill.

Picking tools costs very little either way. Across all 89 tasks, Jev’s picks cost $0.096 and the embedding picks $0.0012. In total Jev’s picks cost about 80 times more, but that’s still around 0.3% of the bill, and Jev’s total came out $4.36 lower than semantic search.

Speed#

Speed was the least interesting result: fewer tools didn’t make the agent faster. Median model time was 20.6–20.7 seconds in all three setups. Most of an answer’s time is the model’s own time, and the tool definitions are served from the prompt cache after the first call, so a shorter tool list likely changes little.

The difference is in the pick itself. Jev takes 2.2 seconds at the median (6.1 s at the 90th percentile), against 0.5 seconds for embeddings. For a chat agent answering in 25 seconds that doesn’t matter much. For something latency-sensitive, it does.

Caveats#

• Sample size. 89 tasks is small; treat differences of a few tasks as noise. Of the 30 tasks also answered in an earlier run, pass counts barely moved (all tools 23 → 23, semantic search 19 → 19, Jev 23 → 24), but 2–6 individual tasks per setup flipped between pass and fail.
• LLM grading. Answers are graded by Claude against fixed facts, not by exact matching.
• Stale facts. Some answer facts were written against 2025 data (repository stars, domain records, live listings) and may have changed since.
• Failed runs. Each picker had 2 answers that hit the output token limit; all tools had none. Those answers count as failed with 0% coverage, but their cost isn’t in the cost totals. One Jev request failed with a server error, and that turn fell back to loading every tool.
• A leftover example. The description of find_tools’s parameter still carried an example from an earlier version of the agent (“list CloudWatch alarms”). The pickers’ agents saw it and the all-tools agent didn’t. It has since been removed.
• Duplicate tools. The sandbox’s Wikipedia server lists its 11 tools twice under two names. Reference solutions use the first names.
• One budget, one model. I tested only 12 tools on Sonnet 4.6. Other budgets, models or tool collections could come out differently.
• Not Claude’s tool search. find_tools is my own, built on each picker. I didn’t benchmark Anthropic’s built-in tool search tool.

Conclusion#

Back to the question: yes, you can give the model 12 tools instead of 138. With a good picker you lose little or no quality and cut the bill by more than a third. The picker matters a lot, though.

What this means for your agent#

1. Stop sending every tool. With the right picker, 12 tools did the work of 138. Quality held, and the median task cost less than half as much.
2. Users don’t speak in your tools’ words. Someone asking about “my local food consumption data” needs a file reader and a code runner, but never says “file” or “code”. Your picker has to understand what the request needs, not just match its words.
3. Your token count isn’t your bill. An 88% cut in tool tokens became a 38% cut in cost, because a few long answers make up much of what you pay. Measure the whole bill.

What I’d test next: larger collections of tools, other budgets than 12, and Claude’s built-in tool search as a fourth setup.

By default, an agent sends its whole toolbox with every call: here, 138 tools and 30,000 tokens, for a task that typically needed 4. Your agent doesn’t need 138 tools. It needs the four that answer the question, and a picker smart enough to find them.