Docs

Benchmark

This page tells you how we measured Pensieve against grep, what the results are, and what we did not measure.

These four numbers are from the latest batch (2026-10-13), with clean Claude Code settings. Pensieve solves as many tasks as grep, with about half the tool calls. An earlier batch, with Pensieve 0.5.6, solved 91% against 85% with grep. The two later clean batches did not repeat this gain. Both results are below.

Method

The script is bench/agent_search.py. It does these steps:

1. It makes tasks from indexed code. Each task describes one piece of code. The answer is the file that holds it.
2. It writes each task in one of these styles: a plain description, a name that the code defines, an error string from the code, or a mix of a name and words.
3. It splits the tasks into a dev set, which we used to tune Pensieve, and a test set, which we did not use to tune.
4. It runs Claude Sonnet headless through Claude Code (claude -p) on each test task, once in each setup.
5. It records whether the answer names the right file, the tool calls, the tokens, the time, and the cost that Claude Code reports.

Cost depends on the state of Claude's prompt cache. The same tokens can cost 4 times more with a cold cache. Thus we compare cost only between setups in one batch. Calls, tokens and time can be compared between batches.

Results: first batch (Pensieve 0.5.6)

In 82 of the 144 guided runs, the agent answered with one tool call or none. Pensieve solved as many tasks or more, and cost less, in every repo:

The two medium repos have 1,783 and 2,383 files.

Results: later batches with clean settings

The first batch loaded the Claude Code settings of the computer that ran it. Later, the script loads only the settings of the repo (--setting-sources project,local). These are the last of those batches, run on 2026-10-13 with the same 72 tasks:

• Accuracy is level with grep. The 6-point gain of the first batch did not come back with clean settings. This task set suits grep: most descriptions use words that are in the code.
• Cost and time are lower. Guided was 23 to 25% cheaper and 25 to 29% faster than grep in the two clean batches.
• 6 tasks fail in every setup. Their description fits a near-copy of the answer as well as the answer.

Search alone, with no agent

A second set measures one search, with no agent. It has 2,377 queries in the same four repos. Each query has one right file. “@1” is the share of queries where the right file is first. The grep row is one case-insensitive grep, with files ranked by the number of matches.

In the top 8 results, which is what the search tool shows, Pensieve has the right file for 91.6% of queries. Grep has it for 53.1%. On the held-out queries only, @1 is 66.7% against 35.2%. A search took 73 ms at the median, on an Apple M4 Max. Grep is level with Pensieve on exact strings. STRATEGIES.md lists every ranking method that we tried, with its numbers.

Repo overviews

We asked “Give me a high-level overview of this repository” in five repos, three times each. A second Claude run checked each claim against the code. Only confirmed errors count.

If a repo has good notes for the agent, grep needs no help. The overview tool helps most when a repo has no notes. The test used the overview tool. Today, repo gives the same overview.

Shared threads

This test asks whether a shared thread helps a teammate's agent. The agent worked in a fresh clone, with no access to the owner's computer. It answered questions about 4 real pieces of work, 3 times each. A judge counted the facts from a list that the answer got right, and the wrong claims.

The script is bench/workflows.py. This test is small: 4 pieces of work and 12 runs in each setup.

What we did not measure

• Other models and agents. Every agent run used Claude Sonnet in Claude Code. We did not measure Codex, Cursor, Gemini CLI, Qwen Code, Windsurf, Claude Desktop or VS Code.
• Other repos. The tasks come from four repos on one developer's computer. We do not publish their names.
• Full coding tasks. Each task asks the agent to find one file. We did not measure whether Pensieve helps an agent fix a bug or ship a feature.
• Generated tasks. A model wrote the tasks from the code. Real questions from real sessions are noisier. We have only about 18 of them, too few to report.
• Independent runs. Nobody outside the project has run the benchmark yet.
• Linux and Intel. All runs were on Apple Silicon.
• The prompt hook. The optional claude-prompt-search hook cut tool calls by 33 to 46% in three batches. Its accuracy was level with guided, and its cost was the same. The results above do not use it.

Data and scripts

The scripts are in bench/. The run files hold paths and code from the four repos, so they are not public. To get your own numbers, run the benchmark on your repos:

uv run python bench/agent_search.py gen --repo ~/code/my-app -n 40 --tasks tasks-v2.jsonl
uv run python bench/exact_tasks.py      # adds name, error-string and mixed tasks
uv run python bench/agent_search.py run --repo ~/code/my-app --tasks tasks-v2.jsonl --arms grep,guided

Something wrong or missing? Tell us on GitHub.