"Runs ≠ Secure": LSCBench Tests 6 Frontier Coding Agents, Best Combined Score Just 25.1%

On October 10, a paper with "vibe" right in its title landed on arXiv: Last Secure Code Benchmark: Is the Code You Vibe Secure? (LSCBench). Fifteen researchers from the National University of Singapore and Alibaba (Qwen Team, Alibaba Cloud Security) did something that was long overdue: they put six frontier coding agents in the same exam room and tested not "does it run," but "is the running code secure."

The report card is ugly. Functional pass rates reach 91.3%, but the "functional AND secure" rate tops out at 25.1%. In other words, out of every four pieces of code an agent hands you that run, three hide vulnerabilities it doesn't even know about. And that is the most honest picture of vibe coding's daily reality: you only asked for features, and the agent only cares about making them run.

How this benchmark "traps" the agent

LSCBench's design philosophy can be summed up in one line: trap it like a real user would, then grade it like an attacker would.

First, the trap. All 450 tasks come from 150 real vulnerability scenarios (115 from real CVE entries, 35 expert-written Java applications), spanning seven languages — Java, C, C++, JavaScript, Python, PHP, Go — and 24 CWE classes (21 of them in the 2025 CWE Top 25). Each task states only functional requirements — "write a feature that handles user uploads" — and never mentions security. That's a faithful reproduction of how vibe coding prompts actually look: when you talk to Claude Code, you don't append "remember to prevent SQL injection" to every sentence either.

Then, the grading. Every task hides two test suites: a functional suite checking "does it work," and a security suite that replays the vulnerability's proof-of-concept attack against the running program. Both must pass. The isolation is ruthless: the agent runs with no network access at all (model API only), the verifier runs in a separate container, and the agent never sees what the tests look like. Gaming the tests by memorization is off the table.

Finally, three granularities, each harder than the last: function removes only the smallest function containing the vulnerable code and asks the agent to fill it in; file removes the whole related file; repo removes the entire repository (all of it under 8,000 lines, or 8,000 lines around the patched code above that) and asks the agent to rebuild it. A scenario only counts as conquered when all three granularities pass both suites.

The report card: 91% and 25% are two different things

Six frontier models, all at maximum reasoning effort (max, xhigh for Qwen3.8-Max); five ran inside Claude Code v2.1.143, GPT-5.6-Sol inside Codex v0.154.0. All 2,700 run traces are public — every conclusion is checkable. The results (Func∧Sec = both functional and secure, the headline score):

/media/content-media/a56f3ccd-38fe-490f-a5d3-6edb6bc228d5/2026/10/f356beab-addc-4e81-88f4-270609ff5b90.png

A few numbers worth savoring one by one:

• Kimi-K3 is the most ironic case: the highest functional pass rate in the field at 91.3%, the lowest combined rate at 16.4%. It's the best at "making it run for you" and the least concerned with whether you survive it. The gap between "runs" and "secure" is widest right here.
• Claude Fable 5 took first place — at just 25.1%: three out of four working snippets it produces are insecure. That is the security level of today's strongest coding agent.
• The all-granularity column is the bleakest: even the best, Fable 5, manages only 12%, Kimi-K3 just 2%. Patching a small function, agents can bluff through; ask them to rebuild a whole repository and vulnerabilities reproduce almost by default — they're not accidental typos, they're systematic blind spots.

The paper's abstract doesn't mince words: "posing strong concern on the security of the code written by vibe coding."

Why this is vibe coding's bespoke trap

Traditional code has vulnerabilities too, but LSCBench exposes something new: an agent's "default settings" are insecure.

When a human programmer writes a login feature, "don't store passwords in plaintext" at least flashes through their mind. When an agent gets "write a login feature," its optimization target is turning the functional tests green. Security requirements absent from the prompt are absent from the code. This isn't the model being dumb — Fable 5 writes perfectly secure code when explicitly asked — it's that the vibe coding workflow naturally outsources security requirements to the quality of the user's prompts. And the user is vibe coding precisely because they don't know.

That closes a vicious loop: the less someone knows about security, the more they rely on agents to write code; the more obedient the agent, the smoother the code runs, the faster it ships — and the faster vulnerabilities go live. LSCBench's value is putting hard, reproducible numbers on that loop for the first time.

Three actionable takeaways for vibe coders

A benchmark's value isn't in the spectacle, it's in what you change on Monday. Three things you can do today:

First, put security requirements in the prompt, not in your prayers. Since agent behavior is entirely prompt-driven, constraints like "logins must resist brute force" and "validate all external input first" belong as permanent fixtures in your project's prompt templates — not improvised every time. Our guide "Don't Ship Your Vibe Project Naked: Web Security Baseline" is a good source to distill CSP, CSRF, XSS, and SQL injection baselines into context your agent reads before every session.

Second, quiz your own project the "double-blind" way. LSCBench's methodology scales down nicely: take a piece of auth/payment/upload code your agent wrote, hide the implementation, keep only the functional description, ask a fresh session to rewrite it — then play attacker against the result. The point isn't how many holes you catch; it's discovering whether your agent defends proactively when nobody reminds it. Run that test once and your calibration of "trust" will never be the same.

Third, keep human review on the critical path. The report card says: for function-level patches, agents' security level is at least a fair fight; for repository-scale rebuilds, a 12% all-granularity pass rate means you basically can't merge with your eyes closed. The pragmatic strategy is layered: let agents run free on CRUD pages, but code where "one mistake makes the news" — auth, payments, permission checks — gets read line by line before merge. Linus Torvalds using an LLM to untangle kernel code and digging up 3 decade-old bugs showed AI's bug-finding power; LSCBench shows the other side — AI's bug-writing power runs just as deep.

Open source, reproducible — 35 stars is just the start

Notable is how open this benchmark is: Apache 2.0, all 2,700 run traces public, Docker images and Harbor dataset ready, one command with Harbor 0.24 to reproduce. Any team can run their own agent (or fine-tuned model) through it and get their own "security waterline."

The repo sits at just 35 stars (as of October 17) with 11 commits — the paper is a week old. But given how urgent "agent code security" is, and how solid the design is (real CVEs, double-blind tests, network isolation, three granularities), its arrival in every major agent team's internal eval set is a matter of time. When you next see a model safety report citing LSCBench, don't be surprised — this paper likely lit the fuse.

The one-line take: vibe coding solved the efficiency problem of "writing code"; LSCBench reminds us "can this code ship" was never a one-dimensional question. When a 91% functional pass rate buys only a 25% secure pass rate, the biggest risk isn't that the agent wrote it wrong — it's that you believed it wrote it right.

Primary sources: the arXiv paper Last Secure Code Benchmark: Is the Code You Vibe Secure? (arXiv:2610.14766, submitted 2026-10-10); companion repo alibaba/last-secure-code-benchmark (Apache 2.0, full leaderboard and reproduction guide in README). All figures in this article come from the paper abstract and the repo README unless noted.