Qwen Open-Sources BlenderWorld: A 223-Task Benchmark for Visual Agents Doing Real 3D Creation in Blender

On October 14, Alibaba's Qwen team open-sourced BlenderWorld on GitHub under the MIT license. The full title says it all: "Benchmarking Visual Agents for 3D Creation from Real-World References." Note the wording — not "image understanding," not "text-to-3D," but 3D creation from real-world references. An agent is handed a reference from the real world — a line of text, a set of images with captions, a short video — and has to build the scene by hand inside Blender. What it submits is not a render, but an editable .blend project file: something you can open, modify, and keep producing from.

The benchmark ships 223 tasks (bw_0001 through bw_0223) with 2,651 binary rubric criteria. Inputs come in three flavors: 78 text-only prompts, 130 image-plus-text, and 15 video-plus-text. 59 of the tasks also carry spec files — Markdown, YAML, or JSON spelling out dimensions, materials, and camera parameters, essentially the production brief from a client that the agent must follow instead of improvising. The scenarios span storyboarding, product launches, game assets, e-commerce visuals, real-estate marketing, scientific illustration, renovation proposals, classroom teaching, safety training, exhibition planning, construction reporting, and more — nearly all of them jobs someone actually pays for.

The deliverable: an editable .blend, not a pretty render

This is the fundamental break from almost every visual benchmark that came before. Traditional vision benchmarks test looking and answering; generative benchmarks test prompt-to-image quality. BlenderWorld tests whether an agent can deliver inside real production software: open Blender, model, light, shade, frame the camera, and hand over a .blend file, with renders, videos, or model exports attached as needed.

The design hits a genuine pain point: today's generative AI can produce images, but it can't produce anything that survives a production pipeline. However beautiful a 4K render is, a 3D artist can't do anything with it — the topology is dead, the materials are fake, the lighting is uneditable, and changing a single detail means starting over. A .blend file is alive: meshes, modifiers, shader nodes, light rigs, all there for the next artist to pick up and continue. By writing "editable" into the submission requirement, BlenderWorld moves the finish line from "looks right" to "usable." That matters, because "looks right" is exactly where generative models are best at cheating.

The deliverable definition is carefully weighted, too: the .blend project is the core; renders, videos, and model exports are attached "as needed." Renders are evidence, the project file is the deliverable. Once that hierarchy is established, most shortcuts die — you can't bluff your way through with one polished render, because the first judging pass (Agent-as-Judge) opens your .blend scene and inspects it directly. Topology, hierarchy, parameters: nothing hides.

How the exam runs: one bw CLI from prep to verdict

The official bw command-line toolchain wraps the whole pipeline: bw download fetches the dataset, bw verify checks it, bw list browses tasks, bw prepare sets up the environment, bw pack-submission bundles the entry, and bw evaluate runs the judging. The full chain is reproducible — and for a benchmark that isn't a nice-to-have, it's the lifeline. If the runtime, render settings, or judge models differ by a hair, nobody can compare scores.

The evaluation pipeline is deliberately unforgiving. Each submitted .blend scene gets rendered from 8 adaptive viewpoints in Cycles at 1024×1024 with 64 samples, allowing at most 2 corrective re-renders per view. Judging runs three ways: Agent-as-Judge inspects the Blender scene itself (are the objects there, is the hierarchy right, do parameters match the spec); a VLM scores the visuals against the 2,651 binary criteria; and 8 holistic visual-quality ratings round it out. The final score is 0.5 × rule score + 0.5 × dimension score — rules and overall impression split evenly, so neither "detail-stuffed but broken overall" nor "pretty but spec-wrong" entries can sneak a high score.

The weighting reveals the authors' attitude: each criterion carries a weight of 1, 2, or 3, and weight-3 criteria make up 83.3% of all 2,651 — weight 2 is 14.9%, weight 1 a mere 1.8%. Translation: miss the hard requirements and your score collapses; no amount of bonus-point polish saves you. That "majors over minors" judging is exactly how production works — no client forgives wrong product dimensions because the mood lighting was nice.

One detail worth chewing on: the paper uses GPT-5.6-terra for all three judge roles (scene inspection, visual criteria, quality ratings). That makes a full evaluation run expensive, and reproducing someone else's score requires the same judge model. It's the endemic tradeoff of agent-as-judge evaluation — BlenderWorld is no exception. The smarter the judging, the pricier the reproducibility.

What the 2,651 criteria look like

The official dataset statistics figure lays the benchmark's anatomy bare, and it's worth a close look:

The left panel shows task counts across 11 application scenarios: storyboarding leads with 35, followed by product launches and game assets at 21 each, e-commerce visuals at 20, real-estate marketing at 19, scientific illustration and renovation proposals at 18 each, classroom teaching at 16, and safety training, exhibition planning, and construction reports at 15 each. Storyboarding and product visuals dominating is no surprise — those are the two trades where "reference-to-3D-scene" demand is hottest and easiest to standardize.

The middle panel breaks down references and attachments: 130 image references (58.3%), 15 video references (6.7%), plus spec documents — 28 in Markdown, 20 in YAML, 11 in JSON, exactly the 59 tasks that ship with a spec sheet. Note the ratio: over half the tasks are image-plus-text inputs. Text-only is too open-ended, image-only lacks constraints — image-plus-text is what a real client brief looks like.

The right panel shows rubric categories and weights: of the 2,651 criteria, Function leads with 660, Geometry with 446, Composition with 310, Material with 291, Lighting with 203, followed by spatial layout, holistic, layout, physics, scale, presence, count, and color. Function plus geometry alone exceed 1,100 criteria — over 40% of the total. The authors are voting with numbers: get it right first, get it pretty second.

The exam environment: Blender 4.2.10, with an official coding-agent image

The runtime is pinned to Blender 4.2.10. The team also bundles a ready-to-use coding-agent environment — Codex CLI 0.147.0, Claude Code 2.1.220, Node.js 22.17.0 — that can drive Blender through MCP tools. In other words, contestants don't have to figure out "how do I get an LLM to operate Blender" themselves: the wiring between the "hands" (MCP tools) and the "brain" (coding agent) is already connected. The contest is about solving the tasks correctly, not about plumbing.

Optional Harbor integration is supported as well. The dataset, code, and evaluation tooling are released simultaneously on GitHub, Hugging Face, and ModelScope — the ModelScope mirror makes fetching the data painless for developers in China. The MIT license means it's commercially friendly: fork it, adapt it, run internal evaluations, no strings attached.

Opinion: the evaluation winds have shifted — from chat scoring to production deliverables

The number 223 isn't the most interesting thing about BlenderWorld. The paradigm shift is. For the past two years, the mainstream of agent evaluation has been "chat and score": hand the model a conversation and some images, check whether its answers are right and well-phrased. But since SWE-bench, code-agent evaluation has moved to real tasks — did it actually fix the bug, did the tests actually turn green. BlenderWorld ports that logic to visual creation: don't tell me you "understand" the reference image. Hand over the .blend. Openable, editable, renderable, spec-compliant — or it doesn't count.

For vibe coders, the benchmark's direct value is a reference coordinate for "how do I tell whether a visual agent can actually do the job." The market is full of visual agents and 3D tools with gorgeous demos, but between "gorgeous" and "shippable" lies the chasm of a production pipeline. BlenderWorld's 2,651 criteria are essentially a quantified version of a 3D production delivery checklist — the next time someone tells you "our agent does 3D," your first question can be: can it submit an editable .blend on BlenderWorld? What score?

There's a second layer of significance: who gets to write the exam. Agent evaluation infrastructure has so far been built mostly by Western teams — SWE-bench, WebArena, OSWorld, all in English-language contexts. BlenderWorld is a Chinese team (Qwen) formally writing questions in this space, and it's a hands-on practical exam, not a multiple-choice test. Benchmarks are discourse power: whoever defines "what makes a good visual agent" defines the optimization direction of the whole track. Qwen just wrote that definition into 2,651 binary criteria.

A bucket of cold water is still warranted: this exam is not cheap to run. Eight-view Cycles rendering plus three passes of VLM judging costs real compute and real time per run; and the choice of GPT-5.6-terra as judge leaves open questions about score reproducibility and judge-swap stability. But that doesn't blunt the direction — an evaluation should be harder, costlier, and more exacting than the thing it measures. Otherwise it has no reason to exist.

The one-line version: when AI evaluation starts demanding "files that open in production software," the industry's expectation of agents has moved from "sounds plausible" to "ships real work." BlenderWorld is the visual agent's first hands-on practical exam. The questions are public, the license is MIT, and the floor is open.