Open-Sourcing the Ditto Harness and a DittoBench Miner Starter Kit

A while back we introduced DittoBench: a benchmark for any agentic harness that uses tools and memory. It scores whether the harness calls the right tool, surfaces the right memory, and how fast it does both. DittoBench is also the scoring core of an upcoming Bittensor subnet (SN118) where miners compete to build the best agent harness.

Today we’re shipping the two things we promised the community:

1. ditto-harness: Ditto’s agent + memory harness, rewritten in Rust and dual-licensed: open source under AGPL-3.0, with a commercial license for partners.
2. dittobench-starter-kit: a self-contained kit to run that harness, talk to the agent, practice memory retrieval, and prepare a mining submission.

A note for readers, up front. Ditto’s production backend does not run this Rust harness yet. It still runs the original Go implementation. We’re migrating to the Rust harness by the end of the month, around the same time the on-chain mining mechanism goes live. We’re open-sourcing it now so the community can start building and practicing against the real code while we finish that migration.

Why a harness, and why Rust

A “harness” is everything around the model: it stores and retrieves memories, exposes tools, runs the multi-turn agent loop, and assembles the prompt. Most of what users see as an assistant’s intelligence comes from the harness. The model inside it can be swapped.

The original harness was extracted from our Go backend (Postgres + pgvector). The rewrite is Rust, backed by an embedded Turso/SQLite database with native vector search. It needs no external database and no services to stand up. A miner clones one repo, runs one binary, and has the whole agent + memory stack running locally. The same harness can also compile to native bindings (the repo ships NAPI Node bindings too).

The crate is intentionally small and importable: a chat::Harness (prepare → agent loop → save), a memory::Store (ingest, vector + composite search, a subject graph), the retrieval pipeline, and pluggable model/embedder clients via rig-core (Ollama, OpenRouter, vLLM). It deliberately omits Ditto’s closed-source application features and billing.

How the pieces fit

You only ever touch the starter kit. It depends on the harness as a git crate, and the validator scores it: off-chain today, on-chain soon.

The kit is the part you optimize: one file, baseline.rs, marked with EXTENSION POINTs. Everything a miner tunes (model choice, system prompt, retrieval, tools) lives there. You score yourself locally against a fixed benchmark while you iterate. The hosted validator (coming soon) will rotate a fresh, randomized dataset every submission, so there’s nothing to overfit. It scores tool selection and memory recall, the same loop the on-chain validator uses.

Memory retrieval, mirrored 1:1 from production

This is the part we most wanted to release. The kit doesn’t approximate Ditto’s memory system. It ships the same retrieval pipeline and trained ranking models production uses (with one change for the local embedder, below).

The pipeline runs in three stages:

1. Candidate pool: a vector search over the embedded store pulls the top ~50 memories by cosine similarity.
2. Composite scoring (V2): seven signals (semantic similarity, linear + exponential recency, subject frequency, subject semantic match, session continuity, neighbor density) are fused into one score. The fusion weights aren’t hand-tuned. A small weight-predictor MLP (model.bin, ~217K parameters) predicts them per query from the query embedding plus 17 auxiliary features.
3. Cross-encoder rerank: a TinyBERT-L2 cross-encoder (model.onnx, run locally via ONNX Runtime) scores each (query, memory) pair and fuses its ranking with the composite ranking via Reciprocal Rank Fusion.

Both models ship in the kit as weights. To match production, we added a clean Reranker hook to the harness itself, so a second-stage reranker can slot in between the candidate pool and the final ordering. Production has the same shape.

One difference: production embeds with Vertex text-embedding-005, but the kit embeds locally and for free with Ollama’s embeddinggemma, which uses a different vector space. The cross-encoder is unaffected because it scores raw text. The MLP is affected, since it’s calibrated to the embedding space it was trained on. So we retrained the MLP on embeddinggemma through the same production training pipeline (on LongMemEval) and shipped that. It’s calibrated to the embedder the kit uses, and on the bundled seed user it lifts retrieval from hit@10 0.90 to 0.96. To run the exact production stack, swap in Vertex + the production weights; everything else is identical.

One dummy user to experiment with

A retrieval pipeline needs realistic memories to test against, so the kit bundles a self-contained seed user: a coherent, type-balanced slice of LongMemEval with 477 conversation pairs, 1,049 subjects already run through subject-sync, 1,710 subject links, and 50 questions. One command bulk-loads it into the local vector store and embeds everything, and you have a realistic user to query.

Then mem-eval runs the bundled questions through the full pipeline and reports retrieval recall@k. It makes no LLM calls, so it’s free and fast, and it measures retrieval quality separately from the model:

cargo run -- seed-user        # load the dummy user (embeds pairs + subjects)
cargo run -- mem-eval --k 10  # recall@k over the full pipeline, per question type

Talk to the agent

Benchmark scores are abstract, so the kit also lets you talk to the agent. It includes a playground: a single-file web UI wired to a 1:1 production-Ditto chat agent, with the real v2 system prompt and persona, the production default model, the full tool catalog, and real memory retrieval over the seed user.

cp .env.example .env          # paste your OpenRouter key
cargo run -- seed-user        # one-time
cargo run -- playground       # open http://127.0.0.1:8088

Action tools (web search, image generation, agent jobs, settings) return fake-but-plausible results so you can exercise tool-calling without wiring up real integrations. The memory tools are real and query the seed user. The UI shows every tool’s definition and, after each turn, a live trace of the tool calls and the memories that were retrieved. Ask “how many postcards have I collected?” and watch it answer from memory, with citations; ask it to “search the web for…” and watch the tool fire.

The playground also has a Score tab: run the fixed benchmark in the browser and watch it score live. It shows a progress bar, each case as it comes in with its score, latency, and called-vs-expected detail, then a composite breakdown with a “how to read this” legend. It’s the same evaluation as the CLI, shown step by step.

Practice the mining flow

The kit gives you a fixed local benchmark to iterate against: the same static seed user and the same questions every run, so you can tell whether a change helped:

cargo run -- evaluate   # static seed user + the same questions, every run

Because the inputs are fixed, your score is comparable run-to-run (the model itself is still stochastic). To check for overfitting, cargo run -- practice rotates a fresh random dataset instead.

The real subnet must rotate a fresh dataset every submission so nothing can be overfit. The hosted DittoBench validator (coming soon) does that: the same loop, with a freshly randomized dataset per submission and (eventually) a Docker sandbox that builds and runs each submission in isolation, the way the on-chain validator will execute untrusted miner code.

What’s next

• By end of month: Ditto’s production backend migrates onto this Rust harness, and the on-chain mining mechanism (Bittensor SN118) goes live, with validators scoring signed submissions.
• Going fully public: the harness is dual-licensed (open under AGPL-3.0, with a commercial license for partners). We’re finishing the last housekeeping (a stable tagged release) so external miners can depend on it without friction.

Get started

• Starter kit: github.com/ditto-assistant/dittobench-starter-kit. Start with the SETUP.md.
• The harness: github.com/ditto-assistant/ditto-harness (AGPL-3.0 + commercial).
• The benchmark itself: DittoBench: how we measure Ditto’s agent.

A hosted validator that rotates a fresh dataset per submission is coming soon. Until then, practice locally with the kit’s evaluate.

If you build something on it, or you’re planning to mine SN118, we’d love to hear from you.