Z.ai Debuts GLM-Legal at 19.2% on Harvey LAB — and Crops Six Better Models Out of Its Chart

Law might be the harshest exam hall for AI agents. Get code wrong and tests fail, CI goes red — you roll back and try again. Get a contract review wrong and a company loses money, and a lawyer loses their career. On October 12, Z.ai sat that exam: on its official blog it launched GLM-Legal, a legal-domain agent model, claiming a 19.2% all-pass rate on the Harvey Legal Agent Benchmark (Harvey LAB) — nearly 3x its GLM-5.3-Flash base model at 6.7%. The numbers look great — great enough that the official launch chart invites a second look: GLM-Legal towering over Kimi K3 (12.92%), Claude Fable 5 (11.25%) and GPT-6 Astra (5.42%), with the unmistakable posture of a newly crowned "king of legal agents."

But that chart hides half the truth. On the Harvey LAB leaderboard run by independent evaluator Vals AI, GLM-Legal ranks 7th out of 79 models at 19.17%. The six models ranked above it — every single one — are missing from Z.ai's official comparison chart. The top of the board is Meta's Muse Spark 1.2 at 25.42%, a full six points ahead of GLM-Legal. A vendor's chart is never an X-ray; it's makeup. This article wipes the makeup off: what GLM-Legal actually scored, how it got there, what it cost, and what it means for people building on it.

What this system actually is

GLM-Legal is not a from-scratch model. It is a vertically specialized variant built on GLM-5.3-Flash: a 320B-total-parameter MoE with 18B active per token and a 1M-token context window — the same base model Z.ai open-sourced under MIT in August. Training ran in two stages: first, mid-training on an expanded legal corpus to bake the "facts to rules to conclusions" chain of legal reasoning into the model; then reinforcement learning across "tens of thousands of simulated legal environments" to sharpen agentic behavior.

The interesting detail is the second one: the model and the agent harness — the software that drives the agent and its tools — were evolved together. Edge cases surfaced in training first exposed bottlenecks in the harness; fixing the harness then exposed the model to more realistic constraints. In Z.ai's own words, "harness architecture and model training iterate in lockstep." That points to a key shift in agent models in 2026: benchmarks no longer test the bare model, they test the whole "model + scaffolding" system. From now on, every agent benchmark score deserves a follow-up question: how much of this is the harness's doing?

Why law? Z.ai's argument: legal work strings facts, statutory doctrine, analytical reasoning and client-ready work product into one unbroken chain. Given a professional task, the agent must extract the core issues from source documents, build defensible arguments, and format its output to exact specifications — satisfying all of these constraints at once is an ideal stress test for training models to operate in high-stakes professional environments. Translation: law is the textbook case of "one wrong step ruins everything," a natural fit for all-pass grading.

How the 19.2% was actually earned

Harvey LAB is an open benchmark from legal-tech company Harvey: 1,250 tasks across 24 practice areas, backed by over 75,000 expert-written rubric criteria. The grading is all-pass: a task counts only if every criterion is satisfied — one miss and the whole task scores zero, no partial credit. That strictness is precisely its value: it tests not "does the answer look right" but "can it deliver end to end." Vals AI reproduced Z.ai's number on a held-out set, with two LLM judges and internet disabled: 19.17%, 7th of 79.

What Harvey LAB was built for: 1,250 real law-firm workflow tasks and 75,000+ expert rubric criteria, graded all-pass. One missed step zeroes the whole task — no partial credit.

Lay the leaderboard out flat and the story completes itself. GLM-Legal's 19.17% genuinely beats Kimi K3 (12.92%), Claude Fable 5 (11.25%) and GPT-6 Astra (5.42%) — the three rivals Z.ai hand-picked for its chart. But the top six? Muse Spark 1.2 leads at 25.42% by a wide margin, followed by more Muse Spark models and Google's Gemini 4 Argon. Z.ai's official chart cropped all six out. Not one survived. This is the judgment that belongs in the body of the article: the 12.5-point lift is real and independently reproduced — harder than most vendor benchmark claims can say; but the "selective comparison" exposes a marketing instinct, and marketing instinct is not engineering. The next time you see any vendor chart, the first reflex should be to open the original leaderboard and read the full ranking. This episode validates that habit beyond doubt.

The other two benchmarks deserve a closer look too. On LegalBench, GLM-Legal scores 86.2%, tying Kimi K3 but trailing Claude Fable 5 (88.6%) and GPT-6 Astra (87.1%); on the Chinese-focused PLawBench (legal consultation, case analysis, document drafting) it scores 77.3%, trailing Fable 5's 77.7%. Z.ai's framing is "matching frontier closed-source models" — the data says "close but not ahead," a carefully measured choice of words. And one small wound: on Vals AI's Legal Research Bench, GLM-Legal scores 41.83%, worse than its own base model's 45.19% — vertical specialization has a price, and about three points of general legal-research ability were traded away. There are no free lunches in fine-tuning either.

The real selling point is cost and speed

On scores alone, GLM-Legal is "7th place" — not exactly stunning. But flip over the price card and the story changes: in Vals AI's runs, one Harvey LAB test costs $0.15 with GLM-Legal — versus $19.23 for Claude Fable 5, $26.16 for GPT-6 Astra, and $10.55 for Gemini 4 Argon. Note that the $0.15 is Vals' estimate using GLM-5.3-Flash's public rates ($0.075 per million input tokens), because Z.ai hasn't published GLM-Legal pricing at all. One-128th the cost for roughly three-quarters of the score (19.17% vs 25.42%): classic asymmetric competition on value. I'm not racing you for first place; I'm making it affordable.

The trade-off is time. An average task took 1 hour 12 minutes, while the board leader Muse Spark 1.2 needed 24 minutes. Three times slower, over a hundred times cheaper — that math may not work for law firms that are short on people rather than money, but it's genuinely attractive for legal-tech companies wanting to batch-run document triage. Don't reach for your wallet just yet, though: GLM-Legal is currently an application-only beta with no public price, no open weights, and not even published rate limits or data-retention policies. Z.ai is also bundling it into its AutoClaw desktop agent ($24 of signup credits) — a "collect case studies first, open up later" controlled rollout. Translation: you can play with it now, but don't bet production traffic on it yet.

Three demos: it's an agent, not a chatbot

Z.ai's blog post shows three demos, and notably all of them are agent tasks, not chat Q&A. Contract review: the model revises termination and handover clauses in a software services contract, and AutoClaw writes the edits into Word with tracked changes and comments — the lawyer's familiar workflow, preserved as-is. Evidence analysis: finding contradictions in a provider's progress reports across 40 dispute documents, each finding citing the Bates number (the ID stamped on files shared in litigation) — traceability maxed out. Compliance research: drafting a privacy plan for a website launching in Singapore, Malaysia and Indonesia, citing specific sections of each country's data-protection law.

These three demos effectively define GLM-Legal's product shape: not "a chatbot that knows more law," but "an agent that operates documents and delivers to law-firm process." That's also why Harvey LAB, which tests end-to-end delivery, says more about it than LegalBench, which tests single-point legal reasoning — and Z.ai clearly knows it, given where it aimed all its promotional firepower.

What this means for vibe coders

First, the recipe. The path GLM-Legal walked is becoming the standard playbook for vertical agent models: strong base + domain mid-training + RL in simulated environments + harness co-evolution. That playbook ports directly to healthcare, tax, compliance and other high-stakes domains. If you're building agent applications inside a vertical industry, the technical-routemap part of this blog post is worth more than the score — it tells you exactly where the "domain specialization" money goes: not longer prompts, but mid-training corpora and RL environments.

Second, chart literacy. This episode taught everyone a lesson: vendor comparison charts only include rivals that lose. The six models Z.ai cropped out happen to be the actual head of the board. The standard procedure for evaluating any "model X beats Y" claim from now on: find the original leaderboard, read the full ranking, check the reproducer's data. Independent reproduction (here, Vals AI) is the only yardstick separating "marketing numbers" from "engineering numbers" — GLM-Legal's 12.5-point lift survived reproduction, and that's its genuinely hard part; the selective comparison is its soft part. Read the two parts separately.

Third, the other side of 19.2% is 80.8% of tasks missing at least one step. Under all-pass grading, four out of five tasks don't get full marks — which means legal tools still need a lawyer in the loop, and compliance workflows shouldn't go fully automatic yet. Z.ai itself only dares to say "application-only beta," not "replacing lawyers." For builders, the right posture is "triage + draft": let the agent sweep 40 documents for contradictions first, then have a human verify — the Bates-number citation design in the demos was built for exactly this human-machine collaboration.

Fourth, the competitive picture. OpenAI launched Astra for Law in September (GPT-6 Astra wrapped in a legal search index and instructions, sold first to selected firms). Both are walking the same road: the legal-AI battleground has shifted from "whose model is bigger" to "whose configuration — harness, data, process — fits the business better." Z.ai's differentiation is cost and coverage of Chinese legal practice (PLawBench); OpenAI's is ecosystem and law-firm relationships. This fight has just started, and what deserves tracking isn't the next score update but: when GLM-Legal publishes pricing, when it opens weights, and what the first beta firms actually report.

Finally, a path for those who want to build now: the GLM-5.3-Flash base is MIT-licensed, with weights on Hugging Face and support in vLLM and SGLang. You can build your pipeline and tune your harness on the base today, then swap in the GLM-Legal model ID when it becomes available — Z.ai's pricing will most likely continue the base model's low-price line. Or try it inside AutoClaw first; the $24 signup credit covers plenty of test runs. One rule: before client contracts or discovery documents go into a beta API, get the data-retention and region policies in writing first.

GLM-Legal deserves a "cautiously optimistic" verdict: on engineering, it proves the vertical-specialization path works in a high-stakes domain like law, and the 12.5-point lift is real. On marketing, that chart with the top six cropped out is a reminder that what this industry lacks most isn't scores, it's honesty. Scores expire; methodology doesn't — and the habit of "always checking the original leaderboard" will outlive 19.2% by a long shot.