MirroS Open-Sources AgentGarten: Code-Built Worlds Where Agents Learn by Playing

Lead: The Agent Arms Race Just Changed Tracks

On October 8, 2026, the Chinese team MirroS published the research AgentGarten: Code Worlds for Evolving Agents on its official blog, releasing the paper (arXiv:2610.12374, with authors including Tsinghua University's Mingsheng Long) and open-sourcing the code repository (Apache-2.0). This is not another "we trained a bigger model" paper — it is an infrastructure manifesto: the next round of the agent arms race will not be fought over parameters, but over worlds. Whoever can build believable, physically credible, endlessly retryable environments at low cost gets to mass-produce experience.

That claim deserves to be taken seriously. For two years, agent progress has been explained almost entirely by "stronger models": longer reasoning, steadier tool use, longer context. But where does experience itself come from? The real world is too expensive, too slow, too dangerous; simulators are too fake. MirroS's answer is "code-built worlds": physics goes to code, lighting goes to a neural renderer. Code worlds export geometry conditions like depth and surface normals, and the renderer generates the agent's first-person view in real time — over 30fps at 480p on a single GPU. Environments that are queryable, intervenable, and rule-enforceable, with visuals realistic enough to act on. For the first time, both.

The most striking evidence is a historical reenactment: in OpenAI's famous 2019 hide-and-seek experiment, self-play RL needed roughly 25 million episodes to learn "carry panels to build a shelter" and roughly 100 million episodes to learn "use a ramp to climb walls". In AgentGarten, pretrained agents running an "act — review — write a playbook" loop learned to carry panels by round 4 and to use ramps by round 10. The accumulation of experience changed form: from "write gradients into weights" to "write lessons into files" — which is exactly the academic version of the Skills-accumulation loop in vibe coding. For vibe coders who live on prompts and skill files, this paper's methodology section may be more useful than its renderer.

An Old Problem, Solved Anew: Code Handles Physics, Diffusion Handles Light

First, the field's dilemma. Code-built environments (simulators, game engines) have one giant advantage: state is explicit — you can query an object's attributes, change relationships between objects, and declare "what counts as cheating". But their visuals depend on whatever art assets and rendering pipelines you can afford; procedurally assembled scenes tend toward simplified geometry, repetitive materials, and lighting that looks like a game from last century. The other route is video-generation world models: beautiful visuals, controllable cameras and actions — but the world's information lives entirely in generated history and internal memory, with no explicit, queryable, editable world state. Want to check a coordinate, change a rule, enforce a referee? A purely generative system can't give you that programmable interface.

AgentGarten welds the two routes together: code determines how the world changes; the renderer learns what those changes should look like. Unlike approaches that generate a full video window from conditions supplied in advance, its renderer generates incrementally in short chunks — the agent can take a brief action, see the consequences with its own eyes, and decide what to do next before the next chunk is even generated. Future observations depend on the agent's response to what it actually saw. That is a genuine interaction loop, not "film the movie first, then let the agent watch".

The technical details are solid: the renderer starts from a pretrained video model (NVIDIA Cosmos3-Nano) and conditions on geometry — depth and normals. Geometry is a clever choice: simulators export depth and normals directly at training time, real videos can be processed with off-the-shelf estimators, and at inference any code world with a 3D scene can render these conditions cheaply. Training uses a method called Adversarial Forcing: block-causal attention for autoregression, exact replay so gradients reach the history encoding, and a real-data adversarial signal to keep long rollouts from drifting. On the inference side, hand-written Triton fused kernels plus CUDA graph capture push it to real time.

My judgment: the real value of this architecture isn't "pretty pictures" — it's that the environment library can now scale. Previously, every new environment meant redoing art assets and rendering tuning; now every environment shares one learned renderer, and building new worlds is just writing code — layouts, objects, dynamics, interaction rules, and task goals can all be generated and modified systematically. The marginal cost of world-building drops from "art engineering" to "writing programs". That is what gives the "arms race is about worlds" claim its teeth.

Revisiting 2019 Hide-and-Seek: From "100 Million Episodes" to "10 Rounds"

The paper's most persuasive section is a head-on reenactment of OpenAI's 2019 hide-and-seek study — the classic demonstration of emergent tool use. Two self-playing agents: hiders learned to carry panels and build shelters (after roughly 25 million episodes), then seekers learned to drag ramps and climb walls (after roughly 100 million cumulative episodes). Repeated competition forced out a sequence of increasingly sophisticated strategies.

MirroS moved the same game into a code world, but swapped in a completely different learning mechanism: a pretrained agent plays 5 games per round, seeing only the neural-rendered first-person view — no coordinates, no hidden state, no opponent's private observations, not even the score until the round ends. After each round, the hider and the seeker each review their action programs and the visual evidence they were permitted to see, and write down what they learned as a playbook: one lesson per short skill file. Next round, the agents inherit the accumulated playbook and decide which old experience applies to the current scene. Both successful and failed attempts feed into the next revision.

The result: by round 4, hiders were carrying long panels into corners to build cover; by round 10, seekers were dragging ramps to walls and climbing over. The most telling moment is the "move the ramp closer and try again" clip: the first climb falls just short, the seeker nudges the ramp toward the wall, and the second attempt succeeds — not a scripted behavior, but evidence that a reviewed lesson made it into the playbook and got called up a round later.

Side by side, the difference is a paradigm shift:

	OpenAI 2019 · Self-play RL	AgentGarten · Pretrained Agents
What the agent sees	Object coordinates, velocities, lidar readings	Neural-rendered first-person frames
How it acts	One discrete action per step	Short Python action programs
Where experience lives	Policy network weights	Playbook: a library of short skill files
Shelter building emerges	≈25M episodes	Round 4 (5 games per round)
Ramp use emerges	≈100M episodes	Round 10

A necessary cold shower: this comparison isn't entirely fair — 2019 trained policies from scratch; AgentGarten starts from a pretrained model that has already absorbed humanity's knowledge of how the world works. Different starting lines entirely. But that is precisely the point: pretrained models front-load "common sense", so what world models must solve is no longer "learn to walk from zero" but "accumulate task-specific experience and keep it". The playbook mechanism answers exactly that: how experience becomes explicit, inheritable, and auditable. That matters more than the "how many rounds" number itself.

Five Code Worlds, One Practice Loop

Hide-and-seek is just the appetizer. The same "read task — play in parallel — write playbook — archive" loop was ported unchanged to four very different worlds, four rounds each:

• Companion dog: keep a dog comfortable and willingly engaged for a 60-second session. Offer a hand to sniff, stroke its chest or head, play with a ball, or draw back. From the camera view alone, the agent must figure out what this particular dog responds to — a relaxed dog takes small steps closer; an uneasy one turns aside.
• One-lane bridge: two cars start on opposite banks of a bridge that fits only one. Each is driven by its own agent from the windshield view; both must reach the other side as fast as possible within 120 simulated seconds. Pullouts on the banks let one car wait while the other passes.
• Herding: two dogs, each seeing only from its own eye height, get 180 seconds to guide four sheep into a fenced pen and keep them inside for five seconds. Sheep move away from nearby dogs and scatter when pressed too closely.
• Quarry loader: a wheel loader has 360 seconds for three jobs in order — push two rocks onto staging pads, move one of them around a wall into a bunker, and park in its bay.

Note the span of that list: two-player adversarial play, human–pet interaction, multi-agent cooperative driving, herd control, long-horizon tool operation — five worlds sharing one renderer and one practice loop, with only the code changing. That is the scalability promise of code-built worlds: new tasks need new programs, not new pipelines. Each world keeps its own scoreboard (engagement score for the dog, crossing time for the bridge, 100-point scales for herding and the loader), and scores climbed across the four rounds — playbook accumulation is not a hide-and-seek fluke.

Why Vibe Coders Should Care: The Playbook Is the Academic Version of Skills Accumulation

Now, the part our readers should actually take home. Translate AgentGarten's practice loop into vibe-coding language: review at the end of each round → write experience into short files → archive → start the next round with the old files loaded. Isn't that what we do every day? Claude Skills, cursor rules, the AGENTS.md in a project — all of it is "write the pitfalls down in files so the next round of you (or the next agent) doesn't step in them".

MirroS's contribution is to formalize and mechanize this folk practice: the review consumes "action programs plus permitted visual evidence", the unit of accumulation is "one skill per file", and inheritance means "new agents start from the task file and the old playbooks". Three details are worth copying: first, constrain the review's inputs — only what the agent was legitimately allowed to see, so no god's-eye-view cheating leaks into the lessons; second, keep skill files short and single-purpose, one lesson per file, so the next round can retrieve and apply them; third, freeze and archive playbooks with versions instead of appending blindly to one file, so you know which round added which lesson and how it performed.

This must be read together with the RRSI piece we published in cycle37. That article covered Google and collaborators' proposal on "regularizing self-improvement" — puncturing the hype bubble of agent self-evolution, arguing that letting a model grade itself and award itself points will eventually collapse. AgentGarten is the other side of the same coin: RRSI is the critique (don't train like that); AgentGarten is the infrastructure (train like this) — experience goes into files instead of weights, evaluation comes from referee rules declared in code rather than self-congratulation, and every step is auditable and replayable. One says "this road is closed", the other says "the road is here"; read together, they're the complete picture of "how agents evolve". We recommend starting with the RRSI article's methodological critique, then coming back to this piece for the engineering answer.

Honesty Corner: What's Open, What's Not Yet

"Open-sourced" in the headline needs precision, lest readers clone the repo and complain. MirroS-Lab/AgentGarten is Apache-2.0, created October 8, 2026, and what it ships today is the neural renderer: training code (bidirectional, autoregressive, and Adversarial Forcing stages), streaming inference code, and checkpoints on Hugging Face. Stars climbed to 161 within days — the community is clearly watching.

But the README's TODO list is explicit: the code worlds and the agent practice loop (the play–review–playbooks machinery) are still "to be released", along with the real-time engine's streaming server and interactive frontend. So today you can run the renderer; the five worlds and the playbook loop are still on the roadmap. The right reading of this story is "direction validated, core component open, full picture pending" — worth starring, worth tracking, but don't expect to herd sheep today. "Paper first, code in batches" is academia's normal rhythm, and MirroS at least shipped the hardest part — the renderer — first. That's more honest than papers with no code at all.

Closing: The Scarce Resource Is Shifting from Models to Worlds

Zoom out: Alibaba's Qwen-AgentWorld is a contemporaneous world-simulator effort — both lines of work are answering the same question: what kind of "world manual" do agents need? One side bets on video-generative unified world models; the other on the hybrid code-plus-neural-rendering route. My call is that the hybrid wins in the near term, because what agent training lacks most is not "looking real" but "being intervenable, referee-able, and reproducible" — and those three happen to be code worlds' native instincts.

Three action items for vibe coders: first, star the repo and save the renderer's training recipe — the video-generation-plus-geometry-conditioning combo will come in handy for embodied AI and game AI work later; second, copy the playbook's "review — short files — freeze and archive" process into your own agent workflows — usable today; third, remember the thesis — over the next three years, what limits agent progress may not be models, but how fast we can build worlds. Whoever builds the world first gets the experience first.