The A/B Test Where We Were Wrong, and What It Changed

What happens when the data contradicts everyone in the room, and you let it win.

Has a test ever produced a result that changed how you understood your audience? What did you do with it?

There is a version of experimentation where the data confirms what the team already believed, and everyone feels good about the process. That version is not very useful. The useful version is when the data tells you something nobody expected, and you have to decide whether to defend your assumption or follow the evidence.

This is about the second kind.

The hypothesis everyone agreed on

We were optimizing content for new users — people arriving at our platform for the first time, with no prior exposure to the product or the brand. The team had a strong hypothesis: video would be the most engaging format for this audience. The reasoning was sound. Video is accessible. It requires no prior knowledge. It meets a new visitor where they are rather than demanding they read their way into understanding a complex product.

Everyone expected video to win. I expected video to win. We designed the test, set up the tracking, and let it run.

What the data said

The result was unambiguous. New users were not watching the video. They were skipping it entirely and going straight to text. The written content — the part of the page we had treated as secondary — was what they actually wanted.

The first instinct in a room where everyone expected a different result is to look for a problem with the test. Wrong sample size. Tracking error. Seasonal anomaly. We checked all of it. The test was sound. The data was right. Our assumption was wrong.

What we did with it

The easier path would have been to note the result and move on. File it in the results repository, run the next test, let the video stay where it was. Many organizations do exactly that with results that contradict existing beliefs. The data gets acknowledged and then quietly set aside.

We did not do that.

Instead we asked a harder question. If new users are skipping the video and reading the text, what does that tell us about who they actually are? The answer changed something fundamental. We had been thinking of new users as passive viewers — people who needed to be shown the product before they could engage with it. The data said they were active readers — people who arrived already motivated, who wanted to evaluate the product on their own terms and at their own pace.

That is not a small distinction. It is a completely different person.

The thing I want to say directly

Letting the data win when it contradicts you is harder than it sounds. There is a version of data-driven decision making where the data is consulted when it confirms what you want to do and questioned when it does not. That version is not actually data-driven. It is opinion-driven with a dashboard nearby.

The test I am describing produced a result nobody in the room wanted. The team had invested real time in the video hypothesis. Letting that go required something more than analytical discipline. It required the organization to have already built the kind of trust in the process that made an uncomfortable result feel like information rather than failure.

That trust was the real output of the experimentation program. The persona update and the page redesign were downstream of it.

What this means for your program

If every test in your experimentation program confirms what you already believed, something is wrong. Either the hypotheses are not bold enough, or the results that contradict expectations are being quietly set aside.

The tests that change how you think about your audience are more valuable than the tests that validate existing assumptions. They are also harder to act on, because acting on them requires admitting you were wrong about something.

Build a program where being wrong is treated as a data point and not an indictment. The results will tell you things your assumptions never could.

Request Access to My Work