Let’s say experiment A is testing a small change, such as…
““Let’s say experiment A is testing a small change, such as the color of the sign-up button. As results start coming in, it becomes clear that the increase in the number of new visitors signing up is very small—garnering just 5 percent more sign-ups than the original button color. Besides the obvious assumption that changing the color of the sign-up button may not be the key factor holding back new users from signing up, it’s also an indication that you’ll have to let the experiment run quite a long time in order to have enough data to make a solid conclusion. As you can see from the chart above, to reach statistically significant results for this test, you’d need a whopping 72,300 visitors per variant—or, in other words, you’d have to wait 72 days to get conclusive results. As Johns put it in an interview with First Round Review, “That’s a lifetime when you’re a start-up!” In a case like this what a start-up really ought to do is abandon the experiment quickly and move on to a next, potentially higher-impact, one.””
About This Quote
This interpretation was drafted with AI assistance. It is one reading of the quote, not the author's own explanation.
Small A/B test changes may yield insignificant gains, requiring massive traffic for significance; startups should drop low‑impact experiments quickly.
In simple terms: Minor tweaks often need huge data, so abandon them fast.
Key Takeaway
Focus on high‑impact experiments.
Themes
growth
experimentation
efficiency
Mood
analytical
pragmatic
Type
strategic
educational
When to use this quote
- startup marketing
- feature testing
- resource allocation
- decision making
Key Concepts
statistics
product‑market fit
Questions to Reflect On
- When is it worth persisting with a low‑impact test?
- How can you identify high‑impact experiments early?
A Different Perspective
Dropping tests may miss hidden insights.