Microsoft Open-Sources Scope: Stop Asking If the Agent "Works" — Give It an Exam

Lede: stop asking whether the agent "works" — give it an exam

On October 19, 2026 (16:00 UTC), Microsoft's Developer Relations team open-sourced Scope, an agent evaluation platform it had been refining internally for over a year, under the MIT license. It answers a question every SDK, docs, and MCP server team has quietly worried about for two years without a standard answer: now that coding agents read docs, pick SDKs, and call APIs on developers' behalf, how do you know your product is "agent-friendly"?

Scope's answer is blunt: stop asking whether the agent "works" — give it an exam. Send the same set of representative tasks to multiple agents — GitHub Copilot, Claude Code, and others — score each against criteria you define, and let the evidence speak. The most valuable part of the exam isn't the score; it's the complete "answer sheet" the agent leaves behind: which step it got wrong, which doc line it misread, which error message it got stuck on.

Microsoft's own dogfooding is persuasive: the Azure Cosmos DB team used Scope to compare bare models against its Agent Kit across 8 end-to-end scenarios in 5 programming languages, fixing 20 Agent Kit issues and 27 documentation issues. After one targeted Agent Kit update, hierarchical partition-key adoption rose from 47% to 100%, while OFFSET/LIMIT misuse fell from 47% to 0%. Note: these numbers are self-reported by the Cosmos DB team as quoted in Microsoft's official blog, not independently verified — we'll unpack that in the caveats section.

What Scope is — and what it isn't

First, the most common misunderstanding: Scope is not a model leaderboard. It doesn't care which model is "smarter"; it cares whether your product works smoothly when an agent uses it. You — the SDK maintainer, CLI author, MCP server developer, docs team — write the exam. The agents are the test-takers. Your product is what's being graded.

Microsoft calls this DevAx (Developer Agent Experience): understanding how agents interact with your software and using evidence to improve the outcomes they achieve. Teams have long refined DevEx through user testing, interviews, and feedback — those still matter. But there's a new class of user now: the agent. It discovers APIs, chooses SDKs, follows documentation, and invokes tools. An experience that feels great to a human can still leave an agent picking the wrong API, missing a key instruction, or stalling mid-task — and you'd never know.

Scope is a self-hosted, open-source platform built on Kubernetes (a local stack starts with Docker Compose). Crucially, it is software you deploy — not a hosted service. Your code and run data stay in your hands, which matters for teams evaluating private workloads.

It drives existing coding agents — GitHub Copilot, Claude Code, and VS Code with Copilot are officially named — not a new Microsoft model. Scope doesn't sell you the test-taker; it sells the exam hall, the grading service, and the mistake notebook.

How it works: write the exam, run it, grade it, learn from it

• Write the exam: you define representative tasks — e.g., "add an integration to the sample app using our SDK" — optionally with a starting codebase. Configurable personas vary the simulated user's perspective and communication style.
• Run it: the same task goes to multiple agents in parallel. Every configuration is captured in a versioned, reusable profile: worker, model, agent version, skills, MCP servers, extensions. Keep the task and criteria fixed and change only the variable you want to evaluate — that's what makes experiments repeatable.
• Grade it: evaluation criteria are expressed as a directed acyclic graph (DAG): checks can stand alone or depend on prerequisites — a runtime check can depend on a successful build. The Judge service uses LLM-based evaluation to score outputs against those criteria.
• Learn from it: the strongest link. Scope streams activity as the agent works and retains interaction logs and workspace snapshots. You can trace tool calls backward from a failed check to see exactly what happened. A failure becomes a concrete improvement: which doc sentence to fix, which SDK default to change, how to rewrite an error message.

The official blog gives a telling example: an SDK team asks agents to integrate a sample app, then evaluates whether it builds, uses the intended API, and handles a documented error response. If missing-item handling keeps failing, the team gets a specific, investigable behavior — not a vague "the agent isn't great." Fix the docs or tooling, rerun, and watch the pass rate move. The same scenarios become your product's "agent regression tests", runnable from the Portal, CLI, or REST API — evaluation becomes a recurring source of product feedback, not a one-off check.

The three areas it targets: docs, SDKs, tooling

Microsoft groups agent-experience problems into three areas, each producing its own kind of "checkup report":

• Documentation: find instructions agents repeatedly miss and assumptions they get wrong, update the guidance, and rerun to see whether those failures get rarer. Often the agent isn't "dumb" — the key sentence is just buried too deep, or the sample code itself is wrong.
• SDK discovery and adoption: check whether agents pick the intended SDK, initialize it correctly, and use APIs as expected. Compare a baseline against a version with revised examples or an agent skill, and you can quantify exactly what "one more example" is worth.
• Tooling: examine where agents struggle with CLI tools or MCP servers, and use the evidence to prioritize: unclear tool descriptions, unactionable errors, or bugs in the integration itself.

Together they cover the full loop of a product being used by an agent: discover, understand, use correctly. The Cosmos DB case is a complete demonstration: the team compared bare models against its Agent Kit (a guided skill pack) on the same 8 scenarios in 5 languages, and the gap became the fix list — 20 Agent Kit issues, 27 doc issues. The elegant part of the experiment design: it doesn't ask "which model is better," but "did our guidance change agent behavior" — the fundamental difference between DevAx and traditional model benchmarking.

Why it matters: DevEx has to grow up into DevAx

For SDK, docs, and MCP teams: the agent is becoming your software's de facto new user, and traditional DevEx methods are blind to it. You can't interview an agent, survey it, or read its clicks in analytics. When an agent skips a doc sentence, picks the wrong API, or stalls on a vague error message, no traditional feedback channel tells you. Scope restores that missing feedback loop: diagnosis before prescription. Without repeatable evaluation, every effort to make docs "more agent-friendly" is guesswork.

For indie developers, the bar is lower than you'd think: it's open source, MIT-licensed, and starts locally with Docker Compose. If you maintain an MCP server, a skill pack, or API docs, you can write 3–5 "must-pass agent exam questions" yourself — every place the agent gets stuck is your highest-value fix list. That's far more reliable than rewriting docs on instinct. And the criteria are your own DAG — no need to wait for an official benchmark.

A third layer of meaning is in the open-sourcing itself. Evaluation methods need scrutiny too. Microsoft's blog says it plainly: when a result informs a docs update or SDK change, teams should be able to inspect the task, the evidence, and the judgment behind it. By open-sourcing the execution and evaluation code, Scope invites everyone to question its methods, investigate surprises, and improve the evaluation itself — exactly what black-box "AI evaluations" lack: a score with no visible grading rubric and no way to reproduce it.

There's also a strategic read: it turns "agent experience" from vibes into engineering. "Is our SDK agent-friendly?" stops being a subjective judgment and becomes a CI pipeline, a weekly number, a before/after experiment around each release. When "agent pass rate" sits next to "unit test pass rate" as a metric, product teams move from "supporting agents when convenient" to "being accountable for them." That Microsoft's DevRel needed over a year of internal refinement before open-sourcing suggests even Microsoft had to be schooled by hard-to-please agents first — the blog also teases a related talk at AI DevCon NYC in November, a sign they want DevAx to become an ongoing community discipline, not a one-off launch.

How it differs from classic model benchmarks: grade the product, not the model

• SWE-bench, HumanEval: grade the model — "can this model code?" The test-taker is the model, the exam board maintains the questions, scores rank models.
• ThinkingBox (covered by this site on Oct 7): grades the evaluation method — "are your exam questions cheat-proof?"
• Scope: grades your product — "do agents use my SDK/docs/MCP smoothly?" The test-takers are off-the-shelf agents, you write the exam, and the scores tell you what to fix.

This determines Scope's design tradeoffs: it doesn't care which agent scores highest. The blog stresses keeping task and criteria fixed while changing only the variable under evaluation — e.g., bare model vs. model plus your Agent Kit/skill pack. The delta is the true value of your guidance material. That "delta thinking" beats absolute scores: it directly answers "did last week's doc rewrite actually make agents smarter?"

Scope vs. ThinkingBox, for readers keeping track: ThinkingBox is a benchmark methodology (how to write cheat-proof questions — "don't trust the agent's report, check the database"); Scope is a self-hosted evaluation platform (the exam hall, proctoring, grading, and mistake notebook for continuous runs). Complementary, not overlapping — ThinkingBox-style questions could well serve as DAG criteria inside Scope.

Honest caveats: four buckets of cold water

One: the Cosmos DB numbers are vendor-reported. 8 scenarios × 5 languages, 20 Agent Kit issues, 27 doc issues, 47%→100%, 47%→0% — all quoted from the Cosmos DB team in Microsoft's blog, with no independent reproduction. Microsoft itself adds the qualifier: the numbers "describe the Cosmos DB scenarios evaluated, rather than a general ranking of agents." Treat them as evidence that the method produces action, not as a promise of X% gains for every product.

Two: Microsoft admits two hard limitations. Repeatable configurations don't guarantee identical behavior — agent outputs are inherently stochastic, so repeated runs are needed to separate flukes from systemic issues. And automated judges make mistakes — LLM-graded LLM outputs carry intrinsic bias. Microsoft's prescription: representative tasks + repeated runs + human review of the evidence, all three. Scope gives you the checkup report, not the diagnosis; humans still make the call.

Three: self-hosting has a price. Scope is not a turnkey SaaS — you manage the Kubernetes (or local) deployment, credentials, and access to experiment data; running agents needs provider access and the LLM Judge costs model calls. More importantly, designing good tasks and DAG criteria is skilled work: too-easy tasks reveal nothing, unrepresentative tasks distort conclusions. That "exam-writing cost" is widely underestimated.

Four: don't treat it as "integrate once, benefit forever." Agents, models, MCP servers, and plugins all move fast; today's conclusions may expire in three months. Scope's value is precisely in the "continuous" part — wire it into your workflow and rerun regularly, not once at launch.

What to do: start with one question

The blog's closing advice is practical and worth echoing: start with one task that matters to your product. Define what success looks like, run it repeatedly, investigate where agents struggle, use what you learn to improve the experience, then evaluate again. Concretely:

• If you maintain an SDK / MCP server / docs: write the single most important agent task first (e.g., "complete a full onboarding from the docs"), run it in Scope, and watch the failure logs and workspace snapshots like user-session recordings — where the agent stalls is the three doc sentences you most need to rewrite.
• Start local, not on Kubernetes: bring up the local stack with Docker Compose, get your first experiment running, confirm the method yields real signal for your product — then consider a production-grade deployment. The CLI and REST API were built for scripts and pipelines from day one.
• Make "agent pass rate" a routine metric: rerun the same scenarios after every docs or SDK update and watch the pass rate move. Even a manual monthly run beats gut feeling by an order of magnitude — for the first time, how smoothly agents use your MCP server has a quantifiable answer.

The one-line version: as agents become users of software, DevEx must grow into DevAx — and the first step of DevAx isn't optimization, it's the checkup. You can't write the prescription without a repeatable exam. Scope just open-sourced the exam hall; the questions are yours to write. The day your docs update or SDK release routinely includes a glance at "did the agent pass rate go up or down," DevAx will have truly landed — and that day may arrive sooner than many expect.

Primary source: Microsoft Open Source Blog, "Introducing Scope: Open-source evaluation for AI coding agents" (2026-10-19, read in full). Effectiveness figures are vendor self-reports quoted in the official blog, as flagged in the article.