Message testing on Black Friday without false wins

Key takeaways

The success metric is confirmed sales per eligible contact; reply rate is an intermediate indicator and can rise while sales fall.
The control group is the current version, sent at the same time and to the same type of lead; without it, any improvement may be Black Friday seasonality.
With a 4% sale rate per contact, detecting a 2-point difference requires about 1,900 contacts per version; with fewer, the test is inconclusive, not a winner.
Change one variable at a time; price, send time and offer do not change between groups.
Define the stopping rule before starting; checking the result every day and stopping at the first good one is the most common form of false win.

What is a valid message test?

A valid test is a comparison between two or more message versions sent, at the same time, to groups of leads drawn at random, with a written hypothesis, a decision metric defined in advance and a sample size calculated in advance. Any test missing one of those five items produces an opinion with numbers, not evidence.

A written hypothesis means: "The opener with a question about the desired product (B) generates more confirmed sales per contact than the current opener with a greeting and a list of offers (A)." The hypothesis names the variable, the expected direction and the metric.

On Black Friday the test has one more enemy: time. Confi Neotrust (2025) recorded double-digit revenue growth in the first 48 hours of the event; a version sent on Thursday and another on Friday night are not comparable, because the audience and the urgency have changed. That is why the groups need to receive the versions within the same interval, drawn at random for each new lead.

Why reply rate is not a success metric

Replies measure curiosity; sales measure decision. A provocative opener ("Are you still overpaying for X?") usually generates more replies and more arguments, and fewer purchases. A direct opener with the offer generates fewer replies, but whoever replies is closer to paying.

Use three levels of metrics and decide by the third:

Level	Metric	Formula	What it is for
Intermediate	Useful reply rate	Replies that advance the conversation ÷ contacts delivered	Diagnosing why a version lost
Intermediate	Progression rate	Leads who reached the price or asked for a link ÷ contacts delivered	Seeing where the version stalls
Decision	Sale rate	Confirmed sales ÷ eligible contacts exposed	Picking the winner
Protection	Rejection rate	Blocks, "stop" and reports ÷ contacts delivered	Failing versions that sell by burning the list

"Eligible contact exposed" is whoever received the message and had a valid opt-in; undelivered messages stay out of the denominator but are counted separately, because a version with much lower delivery has a content or template problem. The protection metric fails a version even if it sells more: gaining 2 points of sales and losing 3 points of the list to blocks is a loss the following week.

How to set up the control group and the minimum sample

The control group receives the current version (A) in the same period and under the same conditions as the new versions. The draw is per lead, at the moment they enter the list or open the conversation, never by day of the week, by agent or by campaign, because each of those criteria carries its own built-in difference.

The minimum sample depends on the current sale rate and the difference you want to detect. A rough rule of thumb for two-version tests with a sale rate between 3% and 10%: to detect a 2-percentage-point difference with reasonable confidence, each version needs somewhere between 1,500 and 2,500 contacts. If the Black Friday list has 600 people, the test only detects large differences, on the order of 5 points or more; so test a big change (the whole offer) rather than a detail (an emoji in the greeting).

Decision rules:

If the observed difference is smaller than twice the variation expected by chance (calculation in the example below), the result is "inconclusive"; keep the current version.
If the new version sells more but has a rejection rate above 1%, fail it.
If the test reaches the sample before the deadline, end it; if it does not, end it as inconclusive at the deadline. Do not extend it to "give it time to win."

Step by step for the test

Write the hypothesis with variable, direction and decision metric (confirmed sales per eligible contact).
Define the minimum sample per version and the maximum duration (during Black Week, 48 to 72 hours).
Log the versions with a code (A, B), full text and date; without a record, nobody can explain afterward what changed.
Draw each lead into a version at the moment of entry. Use the same send time and the same approved template when the message is proactive.
Keep everything else identical: price, terms, link, the agent who takes over afterward, follow-up cadence.
Only decide once the sample is reached: calculate the sale rate and the expected variation, apply the written rule and document the result, including when inconclusive.
Confirm the sales in the CRM with the value recorded per lead; a CRM with attribution and a funnel, such as RASTRO, lets you link each sale to the lead and the version received.

Hypothetical example: 300 contacts per version decide nothing

A hypothetical example, with made-up numbers to show the calculation. A store tests two re-engagement openers for opted-in leads who stopped replying.

Raw result: A received 300 contacts and generated 12 sales (4.0%); B received 300 and generated 18 sales (6.0%). Difference: 2 percentage points, 50% more in relative terms.
Pooled rate: (12 + 18) ÷ 600 = 5.0%.
Variation expected by chance (approximate standard error of the difference): square root of [0.05 × 0.95 × (1 ÷ 300 + 1 ÷ 300)] = square root of 0.000317 ≈ 0.0178, or 1.8 percentage points.
Comparison: the difference (2.0 points) is about 1.1 times the expected variation. Under the "only decide above 2 times" rule, the result is inconclusive.
Sample needed: for 2 points to be twice the expected variation, each version would need about 1,900 contacts.

The correct conclusion is "B looks promising; repeat with a larger sample," not "B won, switch everything." With 300 contacts, 6 extra sales could be chance, a faster agent that day or two customers who would have bought anyway. How to structure the re-engagement being tested is in follow-up without pressuring the customer.

What cannot change between versions

Price, terms, offer, send time, opt-in and who takes over the conversation afterward. Changing any of those items along with the text turns the test into a comparison of two different campaigns.

Two limits are regulatory, not methodological. Under the WhatsApp Business messaging policy, proactive messages require opt-in: both versions go only to those who authorized it, and "let's test on a cold list" is not a test, it is a violation. And under art. 37 of the CDC (Brazil's Consumer Protection Code, Law 8.078/1990), misleading advertising is prohibited: a version with "last units" without real limited stock, or "from R$ 500 to R$ 300" when R$ 500 was never actually charged, fails before it starts; Procon-SP (São Paulo's consumer protection agency, 2025) recommends using the lowest price of the last 60 days as the reference. Test how you say it, not whether the offer is true.

Remember the cost: according to Meta's pricing documentation (2025), template messages delivered outside the 24-hour customer service window are charged; include the sending cost in each version's cost per sale. The same "sales, not clicks" reasoning applies to creatives, in creatives: evaluate sales, not just clicks.

Common mistakes that manufacture a win

Stopping on the first good day. With few events, the lead changes sides several times; whoever stops when their favorite version is ahead always "wins."
Changing two things. New text and new send time: if it sold more, which one did it?
Excluding those who did not reply. Measuring sales over respondents inflates the version that generated fewer replies. The denominator is eligible contacts exposed.
Letting the tool optimize on its own with every reply. Systems that swap in the "winning" version in real time based on replies pick the most curious one, not the one that sells the most.
Not logging the version sent per lead. Without it, the final report cannot say what each customer received, and the test does not make it into the report that helps the client decide.

Frequently asked questions

Sources

Do more replies mean the message is better?

No. Replies measure curiosity; the decision is by confirmed sales per eligible contact. A provocative opener can double the replies and reduce purchases. Use reply rate to diagnose why a version lost, never to pick the winner.

How many contacts do I need for an A/B test on WhatsApp?

It depends on the current sale rate and the difference you want to detect. With sales around 4% to 5% and a 2-point difference, somewhere between 1,500 and 2,500 contacts per version. With a smaller list, test only big changes and accept "inconclusive" as a possible outcome.

Can I test messages with people who did not opt in?

No. The WhatsApp Business messaging policy requires consent for proactive contact, and sending to a list without opt-in generates blocks and reports that damage the company's number. Test only with contacts who authorized it, or within the 24-hour window opened by the customer.

What if the test ends inconclusive?

Keep the current version and log the result with the sample used. Inconclusive is information: it shows that the difference, if it exists, is smaller than the list allows you to detect. Repeat after Black Friday with a larger sample or test a more radical change.