Stop staring at 'seems better': A/B testing for vibe projects

Almost every independent developer has lived through this moment: you change a button color or rewrite a line of copy, watch the numbers for a week, and conclude "it seems better, or maybe it doesn't." A/B testing exists to replace that "seems" with evidence: split users randomly into two groups, show one group the old version and the other the new version, and compare only the behavioral difference between the groups. The hard part of A/B testing was never writing the split code. It is the experiment design itself: what to test, which metric counts, how much sample is enough, and how to read the result. Broken split code can be fixed; a broken experiment design produces beautiful numbers that are simply wrong.

All examples in this guide are fictional: Nanqiao Bakery is a fictional bakery, its owner Fang Qing is a fictional person, and the order volumes and conversion rates are invented to illustrate method. They do not describe any real business. This guide covers product-level A/B testing — pages, copy, flows. It is different from canary releases: a canary release exists to ship safely with gradual rollout, while an A/B test exists to compare options and answer "which is better." The two are often used together, but their goals differ, and their results should not be mixed up.

Step one: ask whether you can test at all

Not every change deserves an A/B test. Three checks come first. First, is there enough traffic? How many users reach the step you are changing each day? If only 30 people a day see the checkout page and the improvement you hope for is 2%, you would need months to see a difference — and by then the product will have changed three more times. Second, is the change big enough? Swapping one shade of blue for another usually promises a tiny effect; cutting checkout from three steps to one promises something worth measuring. Third, can the result be attributed? If you change pricing, redesign the homepage, and run a promotion in the same week, the difference between test and control groups cannot be credited to any single change.

Nanqiao Bakery's example: Fang Qing wanted to test moving the "Add to cart" button from the bottom of the page to just below the product photo. She did the arithmetic first. The shop's mini-program sees about 800 product browsers a day, of whom about 400 reach product detail pages. At that traffic, detecting an improvement of 5% or more takes roughly two weeks; detecting 1% takes more than two months. She decided to run two weeks: if 5% shows up, that is enough; if not, the change is not worth more agonizing. That habit — doing the arithmetic before launching — is the single biggest time-saver in A/B testing.

Metric layers: one north star, several guardrails

Each experiment needs exactly one primary metric — the single number you most want to move: order conversion rate, signup completion rate, paid conversion rate. Only one, so that you cannot cherry-pick afterwards: across ten metrics, one or two will always "look better" by chance, and that is not a win, that is selection.

Alongside the primary metric, set guardrail metrics: numbers you must not make worse. Testing a simpler checkout, for instance, the primary metric is order conversion, and guardrails should be average order value, refund rate, and support inquiry volume. If conversion rises but refunds rise too, the "win" is fake. Also set process metrics: did users reach the new flow, where did they get stuck? Process metrics do not decide the winner; they explain the result. When an experiment shows no effect, process metrics tell you whether the change itself failed or users never even saw it.

Metric definitions must be frozen before the experiment starts, including the denominator: is conversion "visitors to product detail pages" or "visitors to the homepage"? Is the refund rate measured up to how many days after ordering? Change a definition mid-experiment and the result becomes unreadable. Write the definitions into the experiment record, stored next to the code — not only in someone's head.

Assignment: random unit, stable buckets, never split by time

Assignment has one core requirement: random and stable. The random unit is usually the user, not the individual visit: if the same user sees version A today and version B tomorrow, their behavioral data gets mixed. A common implementation hashes the user ID and takes the modulo, so the same user always lands in the same bucket, even across devices after login. Anonymous users can use a device identifier or cookie, but state the caveat: anonymous bucketing is inherently less stable than logged-in bucketing.

The classic beginner mistake is splitting by time: "old version this week, new version next week, compare the two weeks." The problem is that the two weeks themselves differ — holidays, promotions, weather, news cycles all move the numbers, and you cannot separate the version effect from the time effect. Unless traffic is so small there is no alternative, do not substitute time splits for random assignment. If time splitting is truly unavoidable, downgrade the language of the conclusion: "numbers changed after the new version shipped," not "the new version caused an improvement."

Another pitfall is assignment contamination: a test-group user shares a link with a control-group user, or support staff manually move a user to the other group. Links carrying experiment parameters are especially prone to this. The fix is to treat the server-side record of the user's bucket as authoritative, not the URL parameter; a shared link should not change the recipient's bucket.

Reading results: four classic misreadings

Misreading one: peeking early. On day three the data "looks significant," so you declare victory early. The problem with peeking is that you look many times, and each look carries a chance of catching random fluctuation — repeated peeking is repeated lottery tickets. The fix: set a minimum runtime (say, two weeks) or minimum sample size in advance, and do not call a winner before it is reached. If you must look mid-run, look only at process metrics — did it run, are there bugs — not at win/loss metrics.

Misreading two: p-value worship. A p-value below 0.05 does not mean "the improvement is real." It only means "if the two versions truly made no difference, seeing a gap this large would be relatively unlikely." A "significant" result on a small sample often comes with an exaggerated effect size — the winner's curse. A proper experiment report always states three things together: effect size, confidence interval, and sample size. A report that says only "significant 30% improvement" without sample size or interval is barely a report at all.

Misreading three: cherry-picking across metrics. Looking at ten metrics at once, one or two will turn up "significant" by chance. That is the multiple-comparisons problem. If you genuinely need several metrics, declare in advance which ones decide and which ones are merely observed, or tighten the significance bar accordingly. The pragmatic approach: the primary metric decides the winner; other metrics only surface problems, never add victories.

Misreading four: averages hiding subgroup differences. A 5% overall lift might be a 20% lift for new users and a 10% drop for existing users. That is Simpson's paradox: the aggregate and the subgroups tell opposite stories. Every experiment report should cut results by at least new versus existing users and by device type, checking whether the lift is uniform. If the lift comes from only one segment, the launch decision needs rethinking: ship to that segment only, or keep optimizing?

Alternatives for low-traffic products

If your product's traffic is genuinely too small for a rigorous A/B test, do not force one. Three alternatives exist, each with a price. First, before-and-after comparison: record two weeks of baseline before launch, two weeks after, and compare. The price is that time effects cannot be ruled out, so conclusions must stay conservative. Second, qualitative validation: recruit 5 to 10 real users, have each complete a task on both versions, observe where they get stuck, and record their words. The sample is small, but it finds problems A/B tests cannot — like "users never noticed this button at all." Third, staged rollout: give the change to a small slice of users first, confirm there are no bugs or obvious negative feedback, then expand gradually. That is essentially a canary release, not an A/B test — do not dress it up as an experimental conclusion.

Nanqiao Bakery ended up doing something like this. After two weeks, the button move showed a 3% conversion lift, but the confidence interval crossed zero — strictly speaking, no statistical conclusion. Process metrics, however, showed the new button position did get more clicks, while order conversion did not move — users saw the button, but willingness to order did not change. Fang Qing kept the new position (because the click experience was better) but stopped investing more time in that change. That decision is itself the value of A/B testing: not every experiment must produce a winner. "No conclusion" is also an answer — it tells you to stop spending time here.

A one-page experiment record before launch

Before each experiment starts, write a one-page record containing: the experiment name and hypothesis ("we believe moving the button below the product photo will help users find the purchase entry faster, raising order conversion"); the primary, guardrail, and process metrics with their statistical definitions; the assignment method and random unit; the minimum runtime and minimum sample size; early-stop conditions (such as a serious bug, or a guardrail metric breaching its threshold); and the decision-maker (who calls the winner, so it is not "whoever proposed the experiment declares victory"). After the experiment, write the actual results, subgroup cuts, and the decision back onto the same page. Months later, those pages will explain how the product became what it is better than any dashboard can.

One final ethical boundary: A/B testing measures product behavior; it does not test how far users can be pushed. Price discrimination (different users seeing different prices), manipulative design aimed at vulnerable groups, and collecting sensitive behavioral data without explanation should never be justified as "experiments." Before launching, ask one question: if users knew which group they were placed in, would they feel mistreated? If the answer is yes, do not run it. In the long run, user trust costs far more than any single experiment's lift.

A minimal assignment implementation and its checkpoints

For most vibe projects, assignment code can stay simple — the key is getting a few checkpoints right. A minimal working shape: after login, the server decides the user's bucket from a hash of the user ID and records it in the user profile or experiment table; the frontend only asks the server "which bucket is this user in" and never computes the bucket itself, because client-side bucketing is easy to tamper with and easy to compute inconsistently across pages. Experiment configuration — bucket ratios, start and end times, affected pages — belongs in server-side config, not hard-coded in frontend code, so an experiment can be shut down in an emergency by changing config without shipping a new release.

Before shipping assignment code, run four checkpoints. First, ratio check: run 1,000 test users through it — are bucket shares close to the configured ratio (say, 50/50)? A deviation of more than a few percentage points means the hash function may be broken. Second, stability check: does the same user keep the same bucket across page refreshes and device changes after login? Third, isolation check: can test and control users see each other's versions — through shared caches or shared server-rendered output, for instance? Fourth, logging check: is every assignment decision recorded? An experiment without records cannot be audited afterwards, which means it was run for nothing. Four checkpoints, half a day of work, and most assignment accidents are caught before users ever see them.

A post-mortem template for failed experiments

Failed experiments are more valuable than successful ones, provided the post-mortem is written properly. A five-column template is enough: hypothesis (why the change was expected to work); actual result (primary metric, guardrail metrics, confidence interval, sample size); process findings (did users actually see the change, where did they get stuck); hypothesized causes (labelled as "confirmed" versus "guess" — never write a guess as a conclusion); and next step (abandon, keep optimizing, or change direction). Written this way, Nanqiao Bakery's button experiment reads cleanly: hypothesis "button position affects ordering," result "clicks up, conversion flat," process finding "users saw the button but willingness to order did not change," next step "keep the new position, stop investing."

Post-mortems also prevent repeat work. Three months later, when someone else proposes "should we move the button below the product photo," you open that record and immediately know the idea was tested, what happened, and why it stopped — no need to spend another two weeks re-running it. Accumulated experiment records are, in effect, a team's product intuition turned into written assets. Intuition walks out the door when people leave; records do not. So after every experiment, success or failure, finish that one page before closing the task. It is the cheapest investment you can make in your future self.