How to Run an A/B Test: Setup and Bayesian Analysis

Most product changes still ship on a hunch. You redesign a button, a signup screen, or a whole product page, push it live, and hope the metric you care about moves. A/B testing is the method that replaces that hope: two versions of the same interface, both in production, judged on numbers you can compare. The idea is simple. The beginner mistakes around setup and analysis are not — and they are the reason a lot of tests teach you nothing.

What A/B Testing Actually Compares

An A/B test compares two live versions of the same solution, in production, against one shared success criterion.

Three words in that definition do the real work.

In production means you actually build and deploy both versions. They sit in front of a large volume of real users, with no facilitator in the room and no special supervision. Everyday visitors land on A or on B without knowing which one they got, and without you watching over their shoulder. This is production testing, not a lab.

Two versions can differ by a word or by an entire approach. What they must share is the same starting point and the same success criterion. Between those two poles, each version tries to complete the mission differently. That can be as small as button copy that changes from A to B, or as large as two opposite product-page or signup-screen concepts. Micro and macro both fit.

Compare is the word teams skip. You need quantitative, measurable, comparable criteria so you can pick a winner from data, not from a feeling. If you cannot name that criterion before the test starts, you do not have an A/B test yet. You have two designs and a debate.

There is a volume floor under all of that. A/B testing is very hard on a site or an app with fewer than 1,000 monthly users. Even at 1,000 you are on the edge. An early product with little traffic, or a large B2B product with a thin user base, is the wrong place for this tool. Past that floor, more volume is simply better: faster answers, more precise answers, including on micro-changes, and room to segment. Low volume makes the method painful. High volume lets you scale it.

An A/B Test Hypothesis Needs a KPI and Guardrails

A usable hypothesis names the change, the KPI you expect to move, and the guardrail metrics that must not degrade.

The first concrete step is not a mockup. It is an A/B test hypothesis.

Name three things. First, the change you think is worth making. Second, the KPI you believe that change will move — the metric, the data you expect to shift. Third, the guardrail metrics: the things this change could quietly hurt, and that you want to keep stable.

A concrete case. You think changing the wording on a given button will raise its click-through rate. The change is the wording. The KPI is click-through. The bet is that more people will click. The guardrails sit next to that click: the click rate on a neighboring option should not drop, and conversion on the next step after that first click should not degrade. Those surrounding metrics will not be your primary KPI, but a drop there can flip the final call.

Those three questions are what let you decide whether the test is even worth prioritizing, and later whether it succeeded. Take time on the hypotheses. Write several, then rank them. Testing every idea that walks into the room — including the ones with a weak expected impact — is how you waste the expensive part of the method.

Then check whether the hypothesis is plausible. If you already have analytics or past interviews, go look. The job is to prioritize the tests with the most potential and to cut the ones that will fail. An A/B test burns design time, engineering time, a production release, follow-up, and then either a rollback or a rollout of the winner. That is costly. A short plausibility check — has this been tried before, do competitors run this kind of test, is there writing on this type of change — is usually enough to know whether you are in a reasonable range.

There is no magic method. It depends on the hypothesis. And you should expect most tests to fail. The default is often already decent. That is the point of testing instead of shipping blind: you only roll out what actually moves something. If a test fails, you cancel it, you go back to the original, and you still learned enough to try again.

How to Set Up an A/B Test Without Biasing the Control

Version A is the live original — do not touch it. Version B is the variation, and the size of that change is a trade-off between impact and learning.

Once the hypothesis holds up, you design both variants.

Version A is the original. Do not change it. It is what you already have in production. You can adjust tracking so you can read the test. You do not restyle the thing currently live. If you start editing the production version, you compare two new things and lose the baseline. A is the reference. Leave it alone.

Version B is the variation. Anything is allowed, including a full rewrite of the page. The useful tension is this: the bigger the change, the higher the potential impact — and the blurrier the lesson. A wording change might lift click-through by 2, 3, 4, or 5%. Reworking a whole page might land closer to +20% or +30%. You will not know why conversion moved if you changed everything at once. Change the wording, then the layout, then an image, and you can attribute impact to each move.

That trade-off depends on the company. Ambitious bets and small refinements are both valid A/B testing strategies. There is no absolute best practice between them. Match the size of B to the question you actually need answered.

Then you have to ship both. Good analytics tracking is non-negotiable. In practice there are three main ways to do it.

A dedicated, often no-code tool — AB Tasty is one I used to work at; Optimizely is another — is installed once with a small JavaScript tag. You get a visual editor, close to Figma, where you can design variant B, deploy it to production without a developer, track the results, and read the analysis inside the tool. That autonomy has a price, financial and technical. You will do less than a developer who builds the test in code.

The second path is the one I would recommend to most companies: implement the test in code, and pair it with a dedicated tracking tool. Statsig, Amplitude, Mixpanel, and LaunchDarkly all offer a flag system that shows one variation to one slice of users and the other variation to the rest. A 50/50 split is straightforward. You then track the KPI and analyze it. You keep normal engineering freedom, and you get a clean split without inventing the allocation logic yourself.

The third path is your own code and your own tracking on a standard analytics stack. The hard part is the rule that should send half of users to A and half to B. It looks simple. Attribution makes it messy. What happens if someone starts on a computer and continues on a tablet — how do you keep them on the same version? How do you keep the version stable from one session to the next? How do you know it is actually 50% on A and 50% on B? Those questions always grow extra edges. Homegrown allocation is rarely as obvious as it first appears.

Your engineering team is the right partner for picking the path. None of the three is impossible. Once one of them is live, the test is running — and the job shifts to reading it without fooling yourself.

Bayesian A/B Testing Beats Raw Conversion Counts

Wait for volume and complete weeks, then read conversion rates through a Bayesian calculator — not raw clicks — and treat 95% probability of being best as the win line.

You have to wait for two things: enough volume for the result to be significant, and complete weeks of data. It is tempting to stop after three or four days if the numbers already look full. That is usually a bad idea. Behavior is cyclical: weekdays differ from weekends, Monday differs from Friday. That is even more true in ecommerce or leisure, where the weekly shape is loud. Wait at least 7 complete days. Fourteen is better — two weeks absorb the small week-to-week wobble. That sits on top of the volume requirement, not instead of it.

There is one exception. If the result is clearly negative in the first days, you can cut the test early to limit the damage. What you should not do is declare a win too fast, before you have respected volume and the cyclical-data constraint.

Then you have to read it. The beginner error is to stare at raw counts. In the button-wording example, the instinct is to ask whether people clicked more. Raw click volume is a bad answer. Attribution may already be off — you often discover the split was not exactly 50/50. And a raw count has no statistical reading. Ten extra conversions do not tell you whether that gap is real.

Start with conversion rates: what percentage of users moved from the starting point to the arrival point on A versus on B. That already respects allocation and the volume on each side. It is still not enough.

What you need is the significance of the gap. The simplest reliable tool is a Bayesian calculator. Dedicated A/B testing tools often include one. You can also find one by searching for a Bayesian calculator online — several come with visualizations that make the result easier to hold. You feed it your analysis data. You get a probability of being best, a probability of winning, for A and for B. The sentence you want looks like: variation B has a 93% chance of winning.

That 93% is not a vibe. Given the visitors and conversions you actually collected — a small sample — if you extrapolated to a much larger population and kept the test running for a very long time, you would have a 93% chance that B is truly the winner. At that point you can probably put it in production.

This is why a 3% conversion rate on A and 3.2% on B is not an automatic ship. Depending on volume, the Bayesian calculator will either back B strongly or tell you the gap is still compatible with chance. The usual bar: above 95% probability of being best, you treat that version as the winner. Below that, you do not really know, so you let the test run longer. Bayesian A/B testing — the probability that a variation is the best — is the reliable way to read a win.

From there the operational call is binary. Either B replaces A for everyone, or you cancel B and keep the original. Either way, share the result with the team and file it in an internal library. Reuse the hypothesis you wrote at the start, A versus B with a screenshot of each, a short conclusion, and the numbers that justified the decision. That history stops you from rerunning the same test every six months, and it gives you a knowledge base you can use to defend later calls.

Final Thoughts

A/B testing is less risky than a normal launch. You look, you measure, you roll the winner forward or you cancel. That is still safer than putting a new version in front of everyone and hoping. The craft is in the unglamorous parts: a hypothesis with guardrails, an untouched control, a wait long enough for weekly behavior to show up, and a Bayesian threshold instead of a slightly prettier conversion rate. Run a small first test. That is how the method becomes a habit.