Stop Letting Claude Code Grep the Whole Repo Itself: ColGrep-Agent, a 2B Local Model, Takes Over Code Search — 34% Fewer Tokens, 40% Lower Cost

A boundary to draw first. This site's "small models" series has already published three pieces this month: ant-ling-3-1-flash going open source, JetBrains Mellum 2.1, the open coding-agent model, and Agent37's low-cost agent sandbox from YC F26 — plus a Groundcover piece on agent-to-agent observability. Those were all about "the model itself": parameter counts, licenses, benchmarks. This piece is not about a model. It is about how to carve up a frontier model's job and hand the dirty work to a small one: a 2-billion-parameter local model that takes over "digging through the repo" for Claude Code, cutting API tokens by a third in measured testing. For model background, see the related reading at the end; this article covers exactly one division-of-labor experiment.

On October 12, 2026, the French AI company LightOn published ColGrep-Agent-2B on the official Hugging Face blog: an open 2-billion-parameter code-search agent that runs locally and sits between your coding agent and the repository. The flow: you ask it a natural-language question, it searches with ColGrep's multi-vector semantic search, reads the promising files, follows reference chains for a few more rounds, and hands back file paths, line ranges, and the code itself — so the big model can reason from the locations instead of burning turns figuring out where to look.

The post opens with a terminology distinction worth pinning down before anything else, because everything after it gets confusing otherwise. ColGrep is the multi-vector semantic code-search engine (under the hood, a 17-million-parameter retrieval model called LateOn-Code-edge that keeps one vector per token instead of one per function, so a query about "custom attributes" can surface code that calls the thing fieldDefinition). ColGrep-Agent-2B is the model itself (booted from MiniCPM5-2B, first SFT cold-started, then reinforcement-learned with GRPO). colgrep --agent is the command that runs the model. Engine, model, command — three different things.

How to read this article: the eval design first (that is where the numbers' credibility comes from), then the numbers and the savings mechanism, then the setup commands, then the "small-model division-of-labor economics" argument, and finally the verdicts and the unsaid parts. If you just want to copy the homework, jump straight to the "Getting started" section — two commands and you are running.

How the eval was designed: where the credibility comes from

The short version: this is a vendor-run benchmark, but an unusually honest one — honest enough to pre-empt its own cheating vectors. The test harness is Claude Code running Sonnet 5.5, on 52 code-search questions spread across 13 repositories, each question run three times, across four configurations, for 624 sessions total.

The anti-contamination design has two layers. First, all 13 repositories are public repos created in 2026, spanning Rust, Go, TypeScript, and Python, ranging from 17k to 1.8M lines of code — new repos lower the odds the models saw them during training. Second, the authors ran a sanity check: they asked Sonnet 5.5, with no repository access, to name 10 source files from each repo. It could not name a single one. The post admits, in its own words, that this does not prove the repos were absent from training data — but it does show the model has no usable memory of them. That is an "honest but limited" claim, and it reads as more credible than the usual one-liner of "we used new data so there is no leakage."

The 52 questions come in two flavors: 39 "cross-folder" questions, built by first finding a real reference chain (a symbol defined in one place, used in one or two others) and then having Claude Opus 5.5 write a debugging question that requires finding every location; and 13 "single-location" questions that describe a behavior in words different from the code, testing semantic understanding rather than keyword matching. The four setups: plain built-in grep (the control), ColGrep (the big model calling semantic search directly, no small agent), ColGrep-Agent (search delegated to the local 2B agent), and Balanced (names and patterns go through plain ColGrep, behavior questions go through the agent). Grading is blind: Opus 5.5 scores answers correct, partial, or incorrect against reference locations without knowing which setup produced them, and a separate judge ranks full traces head-to-head.

The token accounting is spelled out too: only Claude Code's API input and output tokens count. Tokens generated by the local 2B model are excluded — because they are not billed, and never were. Confidence intervals are 95% bootstrap, averaging each question's three runs first, then computing intervals over questions.

The numbers: 34% fewer tokens, 40% lower cost, no significant accuracy change

Straight to the results table (all relative to built-in grep):

• API tokens per question: 142k with grep, 93k with ColGrep-Agent — 34.3% fewer, 95% CI [28, 40]. 88% of questions used fewer tokens. Balanced saved 21.2%, CI [13, 29].
• Cache-aware API cost per question: 7.3 cents with grep, 4.4 cents with ColGrep-Agent — 40% lower; Balanced came in at 5.1 cents.
• Correct answers: 83.3% with grep, 80.8% with ColGrep-Agent — a 2.6-point gap with a 95% interval of [-10.9, +5.1], i.e. no statistically significant difference. Balanced matched grep's observed 83.3% exactly.
• Turns and latency: average turns fell from 5.5 to 3.9 — but wall-clock time per question rose from 13.0s to 21.6s. What you save is money and tokens, not time.

624 sessions measured: tokens, cost, and accuracy vs built-in grep. Data: LightOn's HF blog post (2026-10-12), Table 2.

Why do tokens drop? The post gives the mechanism: the savings come mostly from cutting the big model's search turns. Every new turn resends the accumulated context — about 23k tokens per turn in these runs. With grep, Claude averaged 5.5 turns, 5.1 grep/find/ls commands, and 2.0 file reads per question; with the agent delegated, that became 3.9 turns, 2.0 greps, and 0.9 file reads. Fewer turns means less context resent — that is where the third of tokens goes. The trace judge preferred the agent's trace 75% of the time in head-to-head comparisons with grep, with the common pattern being "one delegated search instead of several rounds of guessing symbol names."

One distribution matters a lot for vibe developers: the bigger the repo, the bigger the savings. Under 50k lines, 19%; 50k–150k lines, 29%; 150k–600k lines, 40%. All 13 repositories saved tokens, with no exceptions. The post walks through one concrete example from a CRM repo: "if I add a new property to the custom attributes admins define on records, where is the shape of the returned list declared, and where does the auto-generated typed API layer bring that shape in?" The trap is vocabulary — the code calls these things field (fieldDefinition, fieldListOutput), not "custom attributes." The grep route took 6 turns and 160.6k tokens, guessing keywords wrong several times; the agent route landed in the fields module on the first semantic search and finished in 3 turns and 79.2k tokens, both answers correct. That question saved 51% — better than average, but the mechanism is typical: semantic search skips the "guess the name" step, which is the most token-expensive part of the whole dance.

How good is the 2B model on its own: punching far above 2B

Section 4 of the post changes the exam: forget the token savings, how accurate is the 2B model at finding code by itself? The testbeds are adapted SWE-bench Lite and Multi-SWE-bench Flash — but reframed as code localization, not patch generation: given an issue, point at the relevant files and lines. ColGrep-Agent-2B posted the highest line-level F1 (19.92) of every model tested on SWE-bench Lite, against opponents more than ten times its size, including Gemma-4-31B, Qwen3.5-27B, and DeepSeek-V4-Flash. Its file-level F1 (76.18) trades blows with the 30B class too.

The other table is arguably more illuminating. On the Semble code-search benchmark (1,251 questions, 63 repos, 19 languages), the baselines are serious retrieval methods — ripgrep, BM25, CodeRankEmbed. ColGrep-Agent-2B leads the field on NDCG@10 at 0.868, and its Precision@10 hits 0.744 against BM25's 0.296. The telling column is the last one, "files returned": the agent returns 1.7 files on average, versus 4.6 for BM25 and 9.2 for CodeRankEmbed. Fewer files, far more of them relevant — which is the micro-mechanism of the token savings: the big model no longer has to read a pile of "maybe relevant" files to rule them out.

How was the 2B trained? Starting from MiniCPM5-2B, the team generated non-reasoning trajectories with Qwen3.5-27B as the teacher for an SFT cold start, then ran GRPO reinforcement learning with a reward built on the average of file F1 and line F1, plus a turn penalty kicking in after the second assistant turn — "don't dawdle" written directly into the reward function. The harness exposes three tools (semantic search, read-only terminal, submit), a 10-turn budget, and the exact same setup ships in colgrep --agent. One nice detail: during training, invoking grep incurred a small penalty — "don't use grep" was learned, not hardcoded.

Back to the delegation experiment, one more set of numbers deserves to be pulled out on its own: plain ColGrep semantic search (no small agent, the big model calling it directly) already saved 24.6% of tokens; adding the agent's multi-round "search, read, follow references" loop took it to 34.3%. In other words, semantic search accounts for roughly three quarters of the win, and the agent's autonomous exploration squeezes out another ~10 points. That decomposition says something: embedding retrieval is the foundation, the agent harness is the house on top — semantic search alone saves plenty, but letting the small model decide "search again or submit" has extra juice in it.

Getting started: two commands, three integrations

Installation is two commands: brew install lightonai/tap/colgrep && colgrep --install-agent. The second downloads the default colgrep-agent-2B-Q8_0.gguf model (about 2.5GB), verifies it loads, and prepares the prompt cache. No Python environment, no external inference server, no API key. Inference uses the GPU when available (Metal on Apple Silicon, Vulkan on NVIDIA/AMD/Intel), falling back to CPU; on an M3 Pro a question takes around 20 seconds with 3–4GB of peak memory.

On the repo side, run colgrep init once to build the index; afterwards it detects changed files and updates incrementally. Wiring it into a coding agent is one command each: colgrep --install-claude-code, --install-codex, --install-opencode. The Claude Code integration works through hooks that suggest a search tool in the agent's context — they do not force it. The suggestion strength is adjustable: colgrep settings --hook-mode agent is the maximum-savings agent mode, hybrid is the default Balanced setup, colgrep suggests plain semantic search only. "Suggest, don't force" is a design choice worth noticing — it concedes that the big model has its own judgment, and the tool's job is just to put the option in front of it.

Why it matters: the economics of small-model division of labor

The closing section of the post contains a line worth treating as quotable: not every step in a coding agent's workflow needs to run through a frontier model. Finding the right files is important work, but much of it can be handed to a smaller model trained specifically for repository exploration. This is the first piece to put hard numbers on the "small-model division of labor" thesis.

The accounting works on three levels. First, what you save is the bill, not the compute. The local 2B model's inference is never API-billed, so it is excluded from the token count outright. The 34% token drop converts almost one-to-one into a 40% cost drop (on the cache-aware accounting, cost falls even further than tokens), because what gets cut is the most expensive part of the big model's work: multi-round trial and error carrying tens of thousands of context tokens. For pay-as-you-go vibe developers, this is a savings plan you can use today — no model swap, no prompt changes, two install commands.

The division: the big model keeps reasoning and verification; search drops down to the local 2B model. Local inference is never API-billed.

Second, the split is drawn at "search," not "reasoning." The small model only does localization: search, read, follow references, hand back positions. Judging correctness and continuing the reasoning stays with the big model. In the post's example, the agent returned fields.service.ts with its code, and Claude itself re-grepped to confirm fields.contracts.ts:101 before answering — the big model kept final verification rights. This "outsource the dirty work, keep the decision rights" split is far more robust than "let the small model write your code": a wrong search costs you one more search round, a wrong inference is a real mistake.

Third, the scale economics favor big repos. 40% savings in the 150k–600k-line band, 37–46% on the two repos over a million lines. Vibe-coded projects tend to start small and bloat fast — at a few thousand lines the 19% savings barely register, but as the project grows, that 2.5GB local model gets more valuable every month. It runs opposite to the "buy a bigger model" instinct: instead of stacking every step onto a pricier model, you carve out the most expensive step and hand it to nearly-free local compute.

Our take: facts and judgments, labeled separately

[Fact] Every number above comes from LightOn's October 12, 2026 official blog post — a vendor-run eval: 52 questions × 13 repos created in 2026 × 3 runs × 4 setups = 624 sessions, Claude Code + Sonnet 5.5, blind grading. 34.3% fewer tokens (95% CI [28, 40]), 40% lower cache-aware cost, 80.8% vs 83.3% accuracy with no statistically significant difference, Balanced saving 21.2% with accuracy flat, latency 13.0s → 21.6s.

[Judgment 1] This is the first reusable ledger for "small-model division of labor." Until now the small-model story was always "the model itself is cheap." This story is "carve out the most expensive step of the big model's workflow." The significance of 34% is not its size — it is that this is the first time "which step to carve out, how much money it saves, how much accuracy it costs" got measured together. Every future "frontier model + local small model" split will have to file its accounts in this format: tokens saved, accuracy confidence interval, latency added.

[Judgment 2] For vibe developers, the conclusion is "install it" — with expectations set correctly. Two commands, 2.5GB, no API key, no Python environment: the bar is too low not to try. But it saves tokens and money, not time — 13 seconds per question becomes 21.6, ~20 seconds a question on an M3 Pro. If you are in "ship the demo" mode, Balanced (21% saved, accuracy flat, 17.7s) is the more sensible default; if you are in "let the agent run overnight" mode, the agent mode's 34% is where the money is. Saving money and saving time point in opposite directions here — pick by scenario.

[Judgment 3] Hooks that "suggest rather than force" are the integration posture agent tools should learn. Plenty of tool integrations try to weld themselves into every step of the agent; ColGrep does the opposite: it only suggests, the big model may decline, and there are three strength settings. Designs that leave the choice with the big model will outlive forced integrations — because the big model keeps getting more capable, and a step that needs forcing today is one it will do on its own tomorrow.

[Judgment 4] Open + local is what makes this division of labor actually work. If the 2B model were a cloud API, the token money you saved would come back in another form, with less controllable latency. It is precisely because this is GGUF weights running on llama.cpp with the index on your disk that "division of labor" equals "money saved." LightOn open-sourced the model, the weights, the ColGrep engine (the next-plaid repo), and the eval harness — the 624-session exam room can be re-run by anyone. That "open the ledger" style of release deserves more encouragement than merely open-sourcing a model: it leaves the burden of proof with everyone, not just the vendor's marketing.

What remains unsaid

First, discount the phrase "no statistically significant change in accuracy" appropriately. The post says it itself: the interval is [-10.9, +5.1], wide enough that a meaningful drop cannot be ruled out. With 52 questions, a 2.6-point dip is indeed not significant — but neither does it prove "no effect." The Balanced mode's flat 83.3% is the more solid result: if you care about not losing a single point of accuracy, pick Balanced, not pure agent mode.

Second, latency is a real cost. A local query took 7 seconds at the median and 22 seconds at the 90th percentile on the vendor's GPU server; end-to-end per-question time rose from 13.0s to 21.6s. The 34% token saving is bought with time, and that trade does not always pencil out in interactive "human waiting on the result" scenarios. The post's choice of Balanced as the default hook mode amounts to recommending the compromise — the right product call.

Third, what was measured is "code-search questions," not "end-to-end coding tasks." Finding files is only the first step of fixing a bug; the tokens for editing code and running tests are untouched. The 34% is savings on the search leg — it dilutes when spread across a full task. Do not read it as "your Claude Code bill gets a 34% haircut."

Fourth, the inherent limits of a vendor-run eval. The 13 repos and 52 questions are LightOn's own picks, and the sanity-check wording is admirably honest ("does not prove"). Until an independent third party re-runs it, treat 34% as "the score on LightOn's home exam." The good news: the repos, questions, and harness are all public, so anyone can re-run the whole thing.

Fifth, the index itself never entered the ledger. Every repo needs one colgrep init to build the multi-vector index, then incremental updates. Build time and disk for the index are unreported. Negligible for small repos; for a 1.8M-line monster, the first index build is an upfront payment — amortized quickly, since every question after it saves, but still worth knowing. On the flip side there is a privacy dividend the post does mention: the index lives on your machine, and neither retrieval nor agent inference needs code to leave it. The "digging through the repo" step touches no third-party API at all — the big-model step still goes through your own API key, but the search itself is fully local, which is friendlier for proprietary code.

Related reading

The small-models series (the "the model itself" pieces): ant-ling-3-1-flash goes open source, JetBrains Mellum 2.1: the open coding-agent model, Agent37: YC F26's low-cost agent sandbox; the agent-collaboration angle: Groundcover on agent-to-agent observability. This is the first piece in the series about division of labor — there are plenty of models already; what was missing was the ledger for splitting the work.

In one sentence: ColGrep-Agent-2B proves with 624 sessions that the "dig through the whole repo" step of a coding-agent workflow is not worth routing through a frontier model every time — outsourcing it to a local 2B model cuts API tokens 34% and cost 40% with no significant accuracy change, at the price of a few extra seconds per query. It is the first hard ledger of small-model division-of-labor economics.