The Klaviyo A/B Test: What to Test, What to Skip

A/B testing sounds like rigor. In practice, most email A/B tests in Klaviyo produce results that are either too small to act on or too muddled to interpret. The good news is that fixing this is mostly a matter of knowing what's worth testing in the first place - and what isn't.

The Kaydence TeamOctober 10, 20266 min read

Split testing gets treated like a best practice that runs on autopilot. You pick two subject lines, Klaviyo picks a winner, you move on. Nobody asks whether the test was big enough to mean anything. Nobody asks whether the winning version taught you something you could use again. The test happened, so the box is checked.

The problem with most email A/B tests

Small lists destroy the value of split testing. If you send to 2,000 people and split them evenly, you have 1,000 per variant. Open-rate differences of a few percentage points at that sample size are almost certainly noise. Klaviyo will still declare a winner. That number will feel real. It probably isn't. We use 5,000 recipients per variant as our own working floor - the logic being that below that threshold, a difference of five or more percentage points is more likely random variance than genuine signal. Most DTC brands don't hit that on a single send. That doesn't mean you stop testing. It means you stop treating every result as a decision.

The second problem is testing too many things at once. If the subject line, preview text, hero image, and CTA button color all differ between variants, you have no idea what moved the number. You've collected a data point, not a lesson.

What's actually worth testing

Some variables have enough impact on results - and enough generalizability across sends - that the lessons compound. Others are one-time curiosities that don't transfer. Focus your testing budget (time, list exposure, attention) on the former.

• Subject line angle, not just wording. Testing "Get 20% off today" against "Your cart misses you" isn't a wording test - it's a framing test. Curiosity vs. urgency vs. social proof vs. plain statement. The winner tells you something about how your audience wants to be spoken to.
• Send time by segment. Not universally - your wholesale buyers and your impulse shoppers probably have different peak engagement windows. Test within a segment, then apply that finding to every send to that segment.
• Email length. Long-form storytelling copy vs. a short punchy email with one image and two lines of text. The answer differs by brand and by list. Find your answer and stop guessing.
• Plain text vs. designed. Some audiences respond better to an email that looks like it came from a person. Others expect polished creative. If you've never tested this, you don't actually know which camp your list is in.
• Offer framing. "20% off" vs. "Save $12" vs. "Free shipping on everything" - same economics, different psychology. This is one of the most productive tests for promotional sends.

What to stop testing

Button color. Font choice. Whether the logo is centered or left-aligned. These variables exist in a testing graveyard because in email - unlike a high-traffic landing page - you rarely have the volume to detect a sub-1% lift, and design micro-variables almost never move the needle more than that. More importantly, even if one wins, you can't generalize the lesson. "Red buttons beat blue buttons" is not a strategic insight you can carry forward.

Emoji in subject lines is another one. It's not useless to know whether your audience responds to them - but test it once, apply the answer, and move on. Running emoji variants every send is not a testing program. It's indecision dressed up as process.

How to read the results honestly

Klaviyo's built-in winner selection is based on opens or clicks within a time window you set. That's a proxy metric, and like all proxy metrics, it can mislead. A subject line that drove a higher open rate might have attracted people who bounced immediately when they saw the content didn't match. If you can, follow the test through to clicks and, better, to placed orders. Open rate wins that don't convert to revenue are wins for vanity, not for the business.

Testing inside flows vs. campaigns

Flows accumulate volume slowly - a welcome email that sends to a few hundred subscribers a month will take a long time to reach a meaningful sample. That makes flow testing a slow game, but a valuable one, because the learnings apply to every new subscriber forever. Campaigns are faster because you're hitting your full sendable segment at once. Use campaigns to generate hypotheses quickly. Use flows to confirm which of those hypotheses hold across a wider range of audience states and timing.

One practical approach: run a test in a campaign first. If one angle clearly outperforms - substantial difference, large enough sample - apply that learning to the equivalent message in your flow. In Klaviyo, open the flow, clone the email step, and set the split using the A/B test block, defining your success metric before you launch. Don't let Klaviyo auto-select a winner until you've hit your sample threshold. You get the speed of campaign testing and the permanence of flow improvements.

The cadence question

One test per send is not realistic for most DTC teams. Two or three meaningful tests per month - on sends large enough to produce real data - is a better pace. That's 24 to 36 tests in a year, each adding one confirmed lesson to a growing body of knowledge about your specific list. That compounds. Random split tests on every campaign, regardless of sample size or variable isolation, produce noise that cancels itself out over time.

Where Kaydence fits

When our team manages a brand's Klaviyo program, we track what's been tested, what's been confirmed, and what's still an open question. Before we recommend a test, we check whether the send volume justifies it and whether the variable is isolated enough to produce a usable answer. The client approves the variants before anything sends - and when results come in, we explain what they mean and what we're doing with them. If you want an outside look at whether your current testing approach is actually teaching you anything, book a free Klaviyo teardown. We'll tell you how many of your recent tests had sufficient sample size to trust, which ones produced a lesson you can reuse, and which ones were just noise. Most programs we audit have more tests on record than real answers to show for them.

Ends