Navigating Pricing Tests: When to Change Plans, Test by Country, or Use Variants
You are implicitly testing your pricing even if you think you’re not testing it.
Though it is better to be intentional about your strategy.
There are three main different types of pricing testing:
- Changing the entire plan
- Testing by-country
- Testing variants
Here’s when you should pursue each of them:
Learn more in-depth in this deep dive.
Option 1 — Changing the entire plan
There are three main reasons you might just make a global change for everyone:
- If you don’t have sophisticated A/B testing infrastructure, like if you’re on a limited platform.
- If you have leadership on down conviction that global pricing changes are your strategy, like Fortnite or Apple.
- If you think that you won’t hit statistical significance in another scheme anyway, like an enterprise SaaS such as Snowflake.
While all three are totally valid, it can be very hard to read into your ‘tests.’ You have to take them as directional at best.
Option 2 — Testing by-country
Geographical testing is interesting. A lot of people have huge faith in it. But it violates even more tenets of statistics. The people are totally different with many confounds.
Thus, there’s no point in applying statistical significance to the data.
Nevertheless, you can still get much more specific data.
Retention > ARPA
The best use of Country testing is not for ARPA and conversion rate, since willingness to pay vary greatly from country to country.
Its best use is for retention and expected LTV. Single country testing is widely used in gaming for that very reason.
Testing in New Zealand, for instance, returns very representative D30 retention numbers to what you will get in a global roll-out.
Option 3 — Testing variants
Finally, there is the holy grail of pricing testing: shipping different variants that are randomly assigned to a user.
You tend to see this most with products where the price is not a widely discussed topic — because otherwise people would be confused at seeing different prices — but volume is high enough to get data relatively quickly.
Volume is key here.
For instance — If you have a 100,000 visitors, you can detect a 10% change in 1 week with 95% confidence.
But if you only have 1,000 visitors, that test becomes untenable. It will then take years.
That’s why you need to use the tools available to you.
