Built for humans or for agents? Microsoft open-sources Scope to turn agent experience into repeatable experiments

A long-overdue question: is your product built for humans, or for agents?

On October 19, 2026, Microsoft's open source blog announced Scope: an open-source platform for evaluating AI coding agents, MIT licensed, built on Kubernetes, and developed by Microsoft's Developer Relations team for more than a year. Its starting point is a reality that is becoming impossible to ignore: coding agents have become users of your software. They discover APIs, choose SDKs, read documentation, and invoke tools on a developer's behalf. An experience that works beautifully for a human developer can still leave an agent choosing the wrong API, missing a critical instruction, or getting stuck halfway. Traditional developer-experience work — user interviews, usability testing, product feedback — answers "how do humans find this?" It cannot answer "how do agents find this?" Microsoft has a name for the second question: Developer Agent Experience, or DevAx. Scope is the tool for turning DevAx from a feeling into evidence.

Two boundaries first, so this piece does not collide with recent coverage on this site. First, this site's cycle100 covered Artificial Analysis's Coding Agent Index v2.0, a cross-model capability leaderboard: which model scores higher across 290 tasks. Scope is not a leaderboard. It does not rank models. It is a tool for running repeatable experiments on your own product: the same task, the same criteria, a changed document or a changed example, run again, and see whether agent behavior changed. Second, Scope is not itself a coding agent. It drives agents that already exist — GitHub Copilot, Claude Code, VS Code with Copilot — through tasks you define, then judges the results. Describing it as an agent is the most common misreading of the announcement.

How one Scope experiment runs

A Scope experiment has four parts: a representative task, an optional starting codebase, an agent configuration, and a set of success criteria. The task is not a toy like "print hello world." It is a slice of a real usage scenario for your product, for example "integrate our SDK into a sample application, make one API call, and handle the documented error response correctly." Success criteria are written as a directed acyclic graph (DAG): checks can stand alone or depend on prerequisites, so a runtime check can require a successful build first. As the agent works, Scope streams its activity and retains interaction traces and workspace snapshots. Afterwards, a Judge service built on LLM-based evaluation scores the run against the criteria, showing which requirements were met, which were not, and what the agent did before the failing step. Task, criteria, and agent configuration are all captured in versioned profiles — worker, model, agent version, skills, MCP servers, extensions — so when you change one variable and rerun, you know what you actually changed.

The most important design choice here is not the Judge but the requirement to run repeatedly. Microsoft's own blog post says so plainly: models, agent runtimes, skills, plugins, MCP servers, and custom instructions all influence behavior, results vary between attempts, and configurations keep changing. One successful demo or one passing CI test proves nothing. Only repeated runs of the same task separate recurring failures from one-off flukes. And the Judge itself can be wrong: the blog post explicitly requires human review of the evidence for important conclusions. A team that treats "the Judge said pass" as a release gate is using Scope backwards.

The Cosmos DB case: great numbers, but read their scope correctly

The blog post's Azure Cosmos DB case is the most quotable passage in the announcement and the easiest to misquote. The Cosmos DB team compared bare models with its Agent Kit across eight end-to-end scenarios in five languages. The evaluations led to 20 Agent Kit issues and 27 documentation issues being resolved. After one targeted Agent Kit update, hierarchical partition-key adoption rose from 47% to 100%, while OFFSET/LIMIT misuse fell from 47% to 0%. All of these figures come from the Cosmos DB team's own account as quoted in Microsoft's blog post, and their scope is strictly limited to those eight evaluated scenarios. They do not prove that "Scope improves every product." They prove that a targeted change can move agent-behavior metrics to full marks inside the scenarios tested. Quoting them without that limitation is hype. Read the other way, the case's real value is the workflow itself: evaluate, find the specific documentation vagueness or tooling friction in the evidence, change the documentation or the tool, and evaluate again. Without that final "evaluate again," everything before it is just a feeling.

How it differs from traditional testing: it tests whether the agent can get through, not whether the code is correct

The blog post lists three areas of improvement Scope helps investigate, and each is worth holding against your own product. First, documentation that guides agents to successful outcomes: look for instructions the agent repeatedly misses or assumptions it repeatedly gets wrong, update the guidance, and rerun to see whether failures become less frequent. Second, SDK discovery and correct adoption: does the agent pick the SDK you intend, initialize it correctly, and use its APIs as expected? Compare a baseline against revised examples or a new agent skill to see what helps. Third, tooling that supports task completion: are CLI tools and MCP servers described clearly enough, are error messages actionable enough for the agent to self-correct, and should the next change be to a description, an error message, or the integration itself? None of these three questions is "does the code have a bug?" They all ask: "can a capable agent, following what you gave it, reach the finish line?" That is precisely what traditional tests do not cover. A fully green unit-test suite does not mean an agent can understand your README.

Two deployment facts matter as well. First, Scope is software you deploy, not a hosted service bundled with the release: you can start a local stack with Docker Compose or run the full platform on Kubernetes, and you choose the connected services and manage access to experiment data and credentials. Running agents and model-based evaluation requires your own access to the corresponding providers; Microsoft does not pay that bill. Second, it is MIT licensed, and the repository includes the application services, agent workers, evaluation service, and documentation, welcoming new agent integrations, reusable criteria, and analysis improvements. Together, those facts mean Scope's barrier is not price. It is whether you are willing to build a continuously running experiment for agent experience — machines, time, and human review are all real costs.

Using it in a vibe project: make your own product the test subject first

If you are building a vibe project with an SDK, a CLI, an MCP server, or developer documentation, you do not need to wait for a bigger team to use Scope. The workflow shrinks to three steps. First, pick the one task your users walk most often and agents most often get wrong — "follow the README to set up login," or "initialize a project with the CLI and run the first command" — and write it as a Scope task, with success criteria split along the DAG into "installs, initializes correctly, calls the API, handles the error." Second, run the baseline: change nothing, run three to five times, and record which criteria fail consistently and which fail intermittently. Consistent failures are documentation or tooling problems; intermittent ones are agent variance. Fix the former first. Third, change one thing and run again. The highest-yield first changes are usually not documentation rewrites: moving the instruction the agent always misses to the top, rewriting an error message into an executable form ("change X, not Y"), or adding a minimal agent skill for the SDK. Change only one thing per round, or you will not know which change worked.

There is also an overlooked use: Scope as a documentation reviewer. Much vibe-project documentation is written for humans — elegant paragraphs, vivid examples — but agents do not read documentation by feel. They read it by structure: what to do first, which command to use, what success looks like, where to look when it fails. One Scope run shows you directly which passage of your documentation does not exist in the agent's eyes, because the failure trace ends exactly there. That is cheaper and more honest than asking ten people to review the docs.

Its limits, stated as this article's limits too

Finally, what Scope cannot do, stated plainly so readers do not deploy it with inflated expectations. First, it does not rank agents and is not a general benchmark; using it to argue "Copilot beats Claude Code" misuses the tool, and the blog post says explicitly that the goal is not one universal leaderboard or definition of success. Second, the Judge makes mistakes and single runs vary; without human review and repeated runs, conclusions are not trustworthy. Third, it evaluates how smoothly agents use your product, not how strong a model is or how good code is; a bug in your product logic will show up as the point where the agent trips, not as a fix. Fourth, self-hosting means data and credential responsibility sits with you: private code and secret configuration used in experiments must be managed to your own security standard.

The summary in one sentence: documentation and SDKs used to assume a human reader; with tools like Scope, the readership formally gains an agent. It will not intuit your omissions and implications the way a person might. It will follow what you wrote, step by step, and wherever it gets stuck is exactly where your product needs changing. Adding that perspective to the development process will help a vibe project more, over the long run, than chasing one more model leaderboard.

From a failure trace to a documentation PR: what a full improvement loop looks like

The earlier sections covered Scope's components and how to write a task. Here is one complete improvement loop, step by step, so that "evaluation" lands on concrete actions. Suppose your SDK documentation has a section on configuring the client, and the evaluation task is "configure the client following the documentation and make one successful query." The first round of runs shows, in the failure traces, that across three separate attempts the agent skipped a precondition buried mid-section (an environment variable that must be set first), went straight to the API call, and got stuck on an authentication error. Do not change code first. Look at where that precondition sits in the documentation: is it hidden mid-paragraph, missing an example, or out of order with the surrounding steps? Usually at least one of those is true. The fix is equally specific: move the precondition to the front of the steps, add one copy-pasteable command example, and add a line to the authentication error message saying "check whether the environment variable is set." After the fix, do not declare victory on a single run: following the official blog's own requirement, run the same task repeatedly, confirm the failure rate genuinely fell rather than one run getting lucky, and check for new failure modes (for example, the agent now setting an environment variable it does not actually need every time). Once confirmed, submit the documentation change as a pull request, and in the PR description attach the evaluation task name, the failure rate before and after, and the number of repeat runs — so reviewers see not "I felt the documentation was unclear" but "evaluation showed three in ten attempts stuck here, and the change brought that to zero."

One step in this loop is easy to skip: archiving failure patterns. Every failure type an evaluation surfaces — missed preconditions, wrong SDK choices, misread errors, wrong parameters — is worth recording in a standing list of "mistakes agents commonly make," to check against when writing new documentation or examples: does this passage bury the same trap? The longer that list grows, the better new documentation gets, because most failure patterns recur. That is where Scope's value becomes complete: it is not a one-off testing tool but a machine that continuously produces evidence about where a product is hard to use. Accumulated, that evidence turns documentation and tooling improvements from personal hunches into auditable records.

Caption: A real photograph of a developer reviewing metrics on a screen, standing in for evidence-driven product improvement — a generic workplace photo, not a screenshot of Scope's official interface.