DittoBench: Clocking Tool-Calling, Memory, and Speed in Any Agent Harness

Every harness that gives a model tools and memory (a chat agent, a coding agent, a research agent, a voice assistant) depends on three questions: does it call the right tool, does it surface the right memory, and how fast does it do both? DittoBench measures those three axes. We built it for Ditto, but the method is not Ditto-specific: any harness with a tool catalog and a memory store can be scored the same way.

Every time you send Ditto a message, an agent makes a series of small decisions. Should I search your memories, or just answer? Is this a “build me an app” request that needs the coding harness, or a “write me a doc” request that needs the artifacts tool? Do I read the URL you pasted, or search the web? When the agent gets these calls right, Ditto does what you asked. When it gets them wrong, Ditto either ignores what you said or does the wrong thing.

Public leaderboards don’t measure that. They measure a model in the abstract, on someone else’s tools, prompts and memory store. DittoBench measures the Ditto agent we ship: the real Ditto system prompt, the real Ditto tool catalog and a real seeded memory store, across every model we might put behind it.

This post covers what DittoBench is, how the tool-calling suite presents the real production tool catalog, the 5.5× dataset expansion behind it, and a composite speed score that weights latency alongside quality. It ends with the full June 2026 leaderboard across 10 models.

The two axes that matter

DittoBench scores Ditto along several axes. Two carry most of the weight because they map directly to what users notice:

1. Tool calling (Ditto core): did the agent pick the right tool, with the right arguments, and not over-call? This decides whether Ditto does the task or writes you a paragraph when you asked it to build something.
2. Memory retrieval (LongMemEval): can Ditto find the right memory from a vague question, across a long history? This is the axis behind “Doubling Memory Recall” and “Teaching Memory to Find Itself”. We build on the public LongMemEval dataset (500 labeled long-term-memory questions) but run a harder, end-to-end agentic variant with a modern QA judge (details below).

The memory axis was already mature. Our recent work went into the tool-calling suite: what it shows the model and how deeply it probes.

Testing against the real tool catalog

A tool-calling test only means something if the model sees the same set of tools it would see in production. “Did the model pick the right tool” is undefined if the right tool wasn’t in the list. So the benchmark presents the full post-onboarding catalog the chat agent assembles, roughly 28 tools:

• the base memory/web/image tools (search_memories, search_web, read_links, create_image, …),
• artifacts: durable documents, web pages, and PDFs the user keeps,
• execute_agent_job and execute_agent_workflow: the Ditto Code coding harness that runs code, installs packages, and deploys apps,
• a family of natural-language settings tools: “set my theme to dark”, “switch my model to Opus”, “turn on web search”,
• edit_image, file_feedback_for_team, and more.

Presenting the whole catalog makes the most consequential decision testable: artifacts vs. Ditto Code, document vs. deploy. The harness mocks the tools’ results (a coding job returns a plausible approval envelope, an artifact write returns a new revision, a settings change returns a proposal card) so multi-turn agent loops progress as they would in the app.

Growing the dataset to match the depth

A bigger catalog needs a deeper test. The tool-calling dataset went from 32 hand-written cases to 177, a 5.5× expansion, version-controlled so the benchmark is reproducible and reviewable. (Until now, the fixtures lived only on a laptop. They’re committed now, except the ~15 MB LongMemEval haystack, which stays out of git.)

The new cases target the decision boundaries the real agent faces: the places where a capable model still picks wrong.

In total the set spans 23 categories and exercises 24 distinct tools. Every case is validated against the real catalog at load time: an expected tool name or argument key that doesn’t exist in production is a hard error, so it can’t fail silently.

How a tool-calling case is scored

Each case is one prompt run through the full agent loop. The score has two halves, 0–100:

• Tool accuracy (0–50): did the agent call the expected tool(s)? Full credit for matching the expected set; a penalty for each unexpected extra call (unless the case explicitly allows extras). A case that expects no tool gets full marks only if the agent calls none.
• Judged response quality (0–50): an LLM judge reads the agent’s final answer against the case’s expected behavior. An agent that calls the right tool and then mangles the result scores lower than one that gets both right.

We also keep the older BFCL-style name-F1 / argument-F1 / trajectory-penalty scores alongside, for continuity with earlier runs.

Speed: putting latency on the same footing as quality

Speed matters as much as quality. A model that picks the perfect tool in nine seconds is a worse product than one that’s 95% as good in one second. DittoBench now measures speed on both benchmarks and rolls them into a single composite speed score.

The two axes run on very different time scales. A tool-calling decision is one short turn. A LongMemEval answer is a multi-hop retrieval-and-reasoning loop. Averaging their millisecond numbers would let the slow axis outweigh the fast one. So we normalize each suite independently and fleet-relative:

Within each suite, the fastest model scores 100. Every other model is scored 100 × (fastest_p50 / its_p50), so a model twice as slow as the leader scores 50.

Then the per-model composite speed score is the weighted geometric mean of its available suite indices. We use the geometric mean on purpose: it penalizes imbalance. A model that is very fast at tool calls but slow on memory can’t hide behind one good number, because the geometric mean pulls the composite toward the weaker axis.

Here’s the actual dittobench speed output from the June 2026 run (fastest model in each suite = 100):

[speed] by_model (fleet-relative; fastest in each suite = 100)
  model                         composite  longmemeval(idx/p50ms)  tool_use(idx/p50ms)
  openai/gpt-5.4-nano           100.0      100/1976                100/2393
  openai/gpt-5.4-mini           71.8       56/3535                 92/2595
  google/gemini-3.1-flash-lite  69.2       60/3319                 80/2977
  x-ai/grok-4.3                 47.8       41/4829                 56/4288
  anthropic/claude-opus-4.8     26.3       24/8303                 29/8246
  anthropic/claude-4.6-sonnet   24.6       19/10336                32/7590
  openai/gpt-5.5                23.3       20/9771                 27/8894
  moonshotai/kimi-k2.7-code     17.2       7/26689                 40/5967
  google/gemini-3.5-flash       17.2       12/16814                25/9511
  google/gemini-3.1-pro-preview 14.9       12/16350                18/13109

How to read it: gpt-5.4-nano is the fastest model in both suites, so it anchors the scale at 100. Kimi K2.7 Code shows why the geometric mean matters. It is fairly fast on tool calls (index 40) but very slow on the multi-hop memory loop (index 7, a 26.7s p50), so its composite lands at 17.2 and the tool-calling number does not lift it. The frontier models (gemini-pro, gpt-5.5, opus, sonnet) cluster at the bottom on speed because they reason longer.

Speed metrics come from the same runs that measure quality (we record per-answer latency and tokens as we go), so there’s no separate, extra-cost pass. The dittobench speed command merges a tool-calling run and a LongMemEval run into the cross-benchmark leaderboard.

The June 2026 roster

The official DittoBench leaderboard runs the ten most commonly tested models as of June 2026. They range from the frontier tier down to small, fast models, so the board covers every tier and not only the top:

The leaderboard

Below is the June 2026 official run: 177 tool-calling cases and 150 LongMemEval questions, each across all 10 models. Tool-calling ran with the full 28-tool production catalog. LongMemEval retrieved through the v2 MLP composite ranker over the full memory history.

Tool calling (177 cases × 10 models)

Composite tool score (0–1): half tool-selection accuracy, half judged response quality.

The ranking is flat. The small, fast models (Flash-Lite, mini, nano) land within ~0.05 of the frontier on tool selection while being 3–5× faster. Picking the right tool mostly depends on understanding the catalog, and even the small models do.

LongMemEval (150 questions × 10 models)

This variant differs from the original benchmark. The published LongMemEval (ICLR 2025) hands the model the oracle evidence and grades the answer, which separates QA from retrieval. Our variant is harder and end-to-end: there is no oracle. The model has to decide to search, query Ditto’s own production v2 retriever over the full memory history, and reason over whatever comes back. It can fail at retrieval or reasoning, and we score both. We also add a 2026-current LLM-as-judge QA score: a frontier judge model grades each free-form answer against the gold answer with question-type-aware rules (off-by-one tolerance on temporal questions, latest-fact credit on knowledge-update, intent-match on preferences, and explicit credit for correct abstention when the memory isn’t there). This replaces the brittle string-overlap scoring the original used and matches how long-context answers are evaluated today.

The two columns below are QA accuracy (did the model answer correctly, end-to-end, judged by a 2026 model) and session recall (did retrieval surface the evidence). Overall accuracy across all models: 0.649, in the hard setting, not oracle-QA.

Memory QA separates the models far more than tool-calling did: a 0.22 spread (0.51 → 0.73) vs. ~0.07. Reasoning over a long retrieved context is harder than picking a tool, and the small models drop back here. Session recall stays high and tight (0.78–0.91) across all models because they share the same v2 retriever. Recall measures the retriever; QA measures how well each model uses what it surfaced.

By question type, the difficulty curve matches the published LongMemEval literature:

Multi-session reasoning (stitching an answer across many separate conversations) is still the hardest question type for every model.

Composite speed

The full dittobench speed board is in the section above. In short: GPT-5.4 nano is the fastest model on DittoBench (composite 100), with mini (72) and Flash-Lite (69) close behind. The frontier models trail at 15–26 because they reason longer. Compare speed with quality to choose a model. Flash-Lite, for instance, is a top-5 tool-caller and the 3rd-fastest model, which makes it a strong default for latency-sensitive tool use, although it is mid-pack on deep memory QA.

Does fusing models help?

Ditto’s harness can route a turn through OpenRouter Fusion (a panel of models fused into one inference), so we benchmarked three panels on both axes (the same 177 tool cases + 150 memory questions):

This result surprised us. Three of the cheapest models on the board (GPT-5.4 nano, GPT-5.4 mini and Gemini Flash-Lite) fused into a single panel score 0.809 on tool-calling. The panel out-calls Grok 4.3 (0.799), Gemini 3.5 Flash (0.790) and Kimi K2.7 Code (0.787) running solo. It also beats the fused panel of Opus 4.8 + GPT-5.5 (0.772), the two most expensive models in the lineup. Memory shows the same pattern: the small trio’s 0.660 edges past Claude 4.6 Sonnet (0.653) and Grok 4.3 (0.633) solo. A panel of small models scores in the same range as frontier models, at roughly an order of magnitude less cost per token.

Whether fusion helps depends on what you’re optimizing for:

• Yes, if you want small models to perform like large ones. This is the main result. A panel of small models reaches scores that otherwise only frontier models reach. If your constraint is cost or latency, fusing small models gets you close to frontier scores.
• No, if you want the single highest DittoBench score, at least with the three panels we’ve tested so far. None of these beat the best solo model on either axis (Claude 4.6 Sonnet 0.855 on tools, Kimi K2.7 Code 0.727 on memory). Fusing two frontier models lowered the tool-calling score: Opus and GPT-5.5 each score ~0.82 solo but drop to 0.772 fused, because a “which single tool do I call” decision gets worse when several models weigh in. Three hand-picked pairings cover a small part of the possible combinations.

That question is still open. We’ve sampled three panels out of a very large space of models, panel sizes, and routing strategies, and the small-model result already shows fusion can raise scores. Next we will search for a fusion combination that tops the best single model: more models, more groupings, bigger panels and smarter routing. It’s also a natural target for subnet-118 miners: “find the fusion panel that beats every solo model” is the kind of open-ended search the competition is built to crowdsource. For now: fuse small models to cut cost, or pick one large model for the top score, until someone finds a panel that does both.

We benchmark fusion so that trade-off is a measured number, and so the next combination we try has a baseline to beat.

Reproducibility

DittoBench is meant to be re-run on every prompt or tool change, so drift shows up before users notice it. The whole official sweep is three commands (wrapped in one script):

# 1. Tool-calling + speed (177 cases × 10 models).
go run ./cmd/dittobench core -suite tools -out dittobench-runs

# 2. LongMemEval QA (150 stratified questions × 10 models).
go run ./cmd/dittobench longmemeval -sample 150 \
  -manifest <seed_manifest.json> -out dittobench-runs

# 3. Merge into the composite speed leaderboard.
go run ./cmd/dittobench speed -runs <core-run>.json,<lme-run>.json

Every run records the SHA-256 of the system prompt, the tool schemas, and the retrieval weights, so when a score moves, a baseline diff shows which input moved with it.

Next: DittoBench becomes a Bittensor competition (subnet 118)

We’re turning DittoBench into the validation mechanism for Bittensor subnet 118, so anyone can compete to improve the harness it scores.

A short primer. A Bittensor subnet is an open competition with two roles: miners do the work, and validators grade it. Validators score each miner and write those scores on-chain as weights. Yuma Consensus combines the weights from all validators into a consensus score for each miner. On subnet 118, DittoBench is the grader. Miners submit improvements to the Ditto agent harness (better retrieval ranking, smarter tool routing, tighter prompting) and validators score each submission on DittoBench’s tool-calling, memory, and speed axes. Miners compete to climb the leaderboard above.

We’ll ship a miner starter kit so people can compete from day one: the harness, the scoring code, a public practice split of the dataset, per-axis baselines, and a local runner that reports the exact composite the validators use. Optimize against the practice set, submit, and get scored on the real evaluation set.

Anti-cheat: a fresh dataset for every submission

Any benchmark run as a competition has one hard problem: miners optimize directly against whatever you measure. Give them a fixed test set and they’ll memorize it. So the graded set is never the practice set. Every submission is scored on a freshly generated evaluation set: tool-calling cases synthesized from the live production tool catalog, and memory questions assembled from a held-out haystack under a rotating, validator-controlled seed. No two evaluations are identical and there is nothing static to overfit. A miner can only climb by getting better at calling tools and using memory.

Protocol-level and harness-level defenses back this up:

• Fresh-per-submission datasets prevent memorization and answer-caching.
• Commit-reveal weights stop validators from copying each other’s scores instead of doing the work.
• Yuma Consensus clips outlier validators, so no single grader can move the board.
• Earliest-submission tiebreak + copy detection deter a miner from cloning the current leader and resubmitting it.
• Provenance hashes: every run records the prompt/tool/weight SHAs, so a winning submission is auditable.

When this is live, a global pool of miners pushes DittoBench scores forward. The grader is the real Ditto harness, so every win on subnet 118 is an upgrade to the agent Ditto users work with.

Why this matters

A benchmark run against someone else’s setup tells you which model is good in general. A benchmark run against your own agent (its real tool catalog, its real prompt, its real memory store) tells you which model is good at the job you ship. Only the second one decides what goes into production.

DittoBench rests on one idea: a tool-and-memory harness works well when it picks the right tool, finds the right memory, and does both fast. All three are measurable on the same agent your users use. DittoBench scores Ditto’s chat agent today, and the same harness can score any agent with a tool catalog and a memory store. The numbers above are the June 2026 snapshot. We re-run them on every prompt, tool, and retriever change, so drift shows up before your users notice it.