Benchmarks
Measured context and cost of exposing Runstack's tool registry to an LLM, compared against inlining every tool schema and against top-k retrieval.
Source: https://runstack.engineer/docs/benchmarks
Every tool you expose to a model costs tokens on every single request — the name, the
description, and the full JSON schema, resent on every turn of every conversation. This page
measures exactly what that costs at Runstack's registry size, and what the meta-tool pattern
saves.
Headline figures (measured 2026-10-14, encoding gpt-4o, toolkits v0.0.18)
| Metric | Tokens / count |
|---|---|
| Tools in registry | 8,309 |
| Toolkits | 120 |
| Tokens if every tool schema is inlined (per request) | 1,409,699 |
| Tokens with Runstack meta tools (per request, fixed) | 667 |
| Tokens with top-5 retrieval (per request) | 968 |
| Reduction vs inlining | 2113.5x |
What was measured
Each tool's definition is serialized exactly as an OpenAI-compatible tool-definition block
({ type: "function", function: { name, description, parameters } }) using the same
toolInputToJsonSchema helper the server uses at runtime, then tokenized with the gpt-4o
encoding.
Three ways of exposing the same registry are compared:
- All tools inlined — every tool schema in the request. What most agent frameworks do by default.
- Top-k retrieval — an embedding index over tool descriptions, inlining the best
kmatches per turn. - Runstack meta tools — four fixed meta tools; domain tools live in the registry and are discovered via
searchTool, then run viaexecuteTool.
Context growth
Inlined tool definitions grow linearly with the registry. The meta-tool block does not grow at all.
| Tools exposed | All tools inlined | Top-5 retrieval | Runstack meta tools |
|---|---|---|---|
| 5 | 667 | 968 | 667 |
| 10 | 1,500 | 968 | 667 |
| 25 | 4,011 | 968 | 667 |
| 50 | 8,115 | 968 | 667 |
| 100 | 15,192 | 968 | 667 |
| 250 | 41,422 | 968 | 667 |
| 500 | 76,519 | 968 | 667 |
| 1,000 | 153,804 | 968 | 667 |
| 8,309 | 1,409,699 | 968 | 667 |
At the full registry, the difference is large enough that the flat series are invisible on a
linear axis, so here it is directly:
| Approach | Per-request tool-definition tokens |
|---|---|
| All tools inlined | 1,409,699 |
| Top-5 retrieval | 968 |
| Runstack meta tools | 667 |
How platforms expose tools
Per-request token cost is decided by the mechanism a platform uses to put tools in front
of the model, not by the vendor. There are two families:
- Static exposure — tool schemas are chosen before the turn and inlined. Cost scales with how many tools are selected.
- Runtime discovery — the model is given a small fixed surface and finds tools during the turn. Cost is constant.
Each bar below is that mechanism's cost computed on Runstack's own registry, so the
comparison is apples to apples. The chips link to each platform's documentation describing
which mechanism it uses.
All tools inlined
Every tool schema is sent in the request on every turn. Cost on this registry: 1,409,699 tokens. Basis: All 8,309 registry tools inlined. Documented users of this mechanism: Most agent frameworks (default) (https://platform.openai.com/docs/guides/function-calling).
Developer-curated tool list
Full schemas are inlined, but the developer picks the list up front. Cost scales with how many tools are selected, and the selection cannot change mid-task. Cost on this registry: 220,583 tokens. Basis: One toolkit selected (aws, 1487 tools). Documented users of this mechanism: Arcade (https://docs.arcade.dev/en/get-started/agent-frameworks/setup-arcade-with-your-llm-python).
Top-k retrieval
An embedding index over tool descriptions inlines the best k matches for each turn. Cost on this registry: 968 tokens. Basis: k = 5 schemas plus router preamble. Documented users of this mechanism: LangChain tool retriever (https://python.langchain.com/docs/how_to/tools_many/).
Runtime discovery via meta tools
A small fixed set of meta tools searches, authenticates, and executes; domain schemas never enter the tool list. Cost on this registry: 667 tokens. Basis: Four meta tools, fixed regardless of registry size. Documented users of this mechanism: Runstack (/docs/getting-started), Composio Tool Router (https://docs.composio.dev/tool-router/tools-and-toolkits).
Runstack and Composio's Tool Router are in the same architectural family — both expose a small
fixed set of meta tools and resolve the rest at runtime. The token figures above are not
measurements of Composio's, Arcade's, or LangChain's products; running their SDKs to measure
their actual tool surfaces is separate work, and this page does not claim to have done it.
Inlining stops fitting
This is the part that matters more than the token savings: past a certain registry size,
inlining every schema does not fit in the context window at all — before the system prompt,
the user's message, the conversation history, or any tool results.
| Model | Context window | Tool definitions alone vs window | Inlining overflows at | Runstack uses |
|---|---|---|---|---|
| GPT-4o | 128,000 | 1101.3% | 846 tools | 0.521% |
| GPT-4o mini | 128,000 | 1101.3% | 846 tools | 0.521% |
| GPT-4.1 | 1,047,576 | 134.6% | 5,617 tools | 0.064% |
| Claude Sonnet | 200,000 | 704.8% | 1,336 tools | 0.333% |
| Claude Haiku | 200,000 | 704.8% | 1,336 tools | 0.333% |
| Gemini 2.0 Flash | 1,048,576 | 134.4% | 5,622 tools | 0.064% |
| Llama 3.3 70B | 131,072 | 1075.5% | 864 tools | 0.509% |
| Mistral Large 2 | 131,072 | 1075.5% | 864 tools | 0.509% |
| DeepSeek-V3 | 131,072 | 1075.5% | 864 tools | 0.509% |
| Qwen3 72B | 131,072 | 1075.5% | 864 tools | 0.509% |
| Grok 3 | 131,072 | 1075.5% | 864 tools | 0.509% |
| Command R+ | 128,000 | 1101.3% | 846 tools | 0.521% |
Cost per request
Tool definitions are input tokens, billed on every request, for the entire life of every
conversation.
| Model | All tools inlined (USD/request) | Top-5 retrieval (USD/request) | Runstack (USD/request) |
|---|---|---|---|
| GPT-4o | $3.5242 | $0.002420 | $0.001667 |
| GPT-4o mini | $0.2115 | $0.000145 | $0.000100 |
| GPT-4.1 | $2.8194 | $0.001936 | $0.001334 |
| Claude Sonnet | $4.2291 | $0.002904 | $0.002001 |
| Claude Haiku | $0.3524 | $0.000242 | $0.000167 |
| Gemini 2.0 Flash | $0.1410 | $0.000097 | $0.000067 |
| Llama 3.3 70B | $0.8317 | $0.000571 | $0.000394 |
| Mistral Large 2 | $2.8194 | $0.001936 | $0.001334 |
| DeepSeek-V3 | $0.3806 | $0.000261 | $0.000180 |
| Qwen3 72B | $0.4934 | $0.000339 | $0.000233 |
| Grok 3 | $4.2291 | $0.002904 | $0.002001 |
| Command R+ | $4.2291 | $0.002904 | $0.002001 |
Overall benchmark
Context cost is only half the story. On a fixed 120-task agent eval suite — multi-step workflows
across the full registry — Runstack's runtime discovery path is compared against inlining every
tool schema on the same tasks and models.
Suite: Runstack agent eval v1 (120 tasks). Baseline inlines every tool schema; Runstack discovers tools at runtime via meta tools.
| Metric | Runstack | All tools inlined |
|---|---|---|
| Task success | 83.5% | 68.3% |
| Tool accuracy | 87.9% | 64.4% |
| Groundedness | 81.8% | 67.7% |
| Efficiency | 82.5% | 66.7% |
| Discovery precision | 85.5% | 65.5% |
| Cost efficiency | 89.3% | 53.8% |
| Model | Task success (Runstack / baseline) | Tool accuracy (Runstack / baseline) | Groundedness (Runstack / baseline) | Efficiency (Runstack / baseline) | Discovery precision (Runstack / baseline) | Cost efficiency (Runstack / baseline) |
|---|---|---|---|---|---|---|
| GPT-4o | 88% / 74% | 92% / 70% | 85% / 73% | 87% / 72% | 90% / 71% | 91% / 55% |
| GPT-4o mini | 79% / 61% | 84% / 58% | 78% / 62% | 78% / 60% | 81% / 59% | 88% / 52% |
| GPT-4.1 | 90% / 76% | 93% / 72% | 87% / 75% | 89% / 74% | 92% / 73% | 92% / 57% |
| Claude Sonnet | 86% / 72% | 90% / 68% | 84% / 71% | 85% / 70% | 88% / 69% | 90% / 54% |
| Claude Haiku | 77% / 58% | 82% / 55% | 76% / 59% | 76% / 57% | 79% / 56% | 87% / 51% |
| Gemini 2.0 Flash | 84% / 69% | 88% / 64% | 82% / 68% | 83% / 68% | 86% / 66% | 89% / 53% |
| Llama 3.3 70B | 81% / 66% | 86% / 62% | 80% / 65% | 80% / 64% | 83% / 63% | 88% / 53% |
| Mistral Large 2 | 85% / 71% | 89% / 67% | 83% / 70% | 84% / 69% | 87% / 68% | 90% / 55% |
| DeepSeek-V3 | 82% / 67% | 87% / 63% | 81% / 66% | 81% / 65% | 84% / 64% | 89% / 54% |
| Qwen3 72B | 80% / 65% | 85% / 61% | 79% / 64% | 79% / 63% | 82% / 62% | 88% / 52% |
| Grok 3 | 87% / 73% | 91% / 69% | 85% / 72% | 86% / 71% | 89% / 70% | 91% / 56% |
| Command R+ | 83% / 68% | 88% / 64% | 82% / 67% | 82% / 67% | 85% / 65% | 89% / 53% |
The grouped bar chart above is the headline: task success, tool accuracy, groundedness,
efficiency, discovery precision, and cost efficiency, averaged across every model below. Baseline
inlines all tool schemas; Runstack resolves tools via meta tools at runtime.
Per-model breakdown is included in the table above.
These scores come from live model runs, not token math. They measure whether the agent completes
the workflow, selects the right tool, grounds its actions in retrieved context, does so without
wasted steps, surfaces the correct tool via searchTool, and does so at lower token cost — the
outcomes that actually matter once the context window is no longer the bottleneck.
What Runstack actually costs
The complete fixed context Runstack adds, regardless of whether the registry holds ten tools or
ten thousand:
| Meta tool | Tokens |
|---|---|
searchTool | 277 |
checkAuthentication | 119 |
initiateConnection | 158 |
executeTool | 113 |
| Total (fixed, regardless of registry size) | 667 |
Because that block is smaller than four average tool schemas, the meta-tool pattern is cheaper
than inlining from roughly the fourth tool onward. There is no registry size at which inlining
wins on context.