Price testing is probably some of the most impactful work you can do to grow your business.

Price testing is some of the hardest to design a proper test for.

Because

  • Split testing will inevitably show different prices to the same person causing immense customer frustration when encountering higher prices, and causing hasty conversions when the customer “discovers” a lower price. Your results will be skewed, your customers will be pissed, and the legality of a price split test is dubious.
  • Not split testing will cause a lack of benchmark data / put you in sequential test territory. You’ll need to find an alternative benchmark.

When price testing, try to design and analyse your test to be able to answer:

  • Did I just move sales to/from a different SKU? (i.e. price testing on a subset of a category will be very difficult, and you should at least evaluate category lift, not SKU lift. Your categories have a price structure and people change their behaviour when the comparison set changes)
  • Did this happen in only my test market? (i.e. can I go looking in other markets with similar macro-eco trends and see them jumping up and down to an equal degree as my test market?)
  • Has my growth changed meaningfully compared to the previous period? (i.e. compare growth rate both before test period and during test period. Plot growth % daily or weekly depending on volume and look at the data. Also put the data in a table)
  • Did the bottom line grow? (if evaluating price tests on revenue you’ll inevitably overweight the value of increased volume and underweight the value of better coverage. For scaled business, very often, a GREAT result of an increase-price test is to hold revenue steady. The higher price will generate a cascade of better P/L performance reaching all the way to the bottom. So make sure you can evaluate on your profit growth, not just rev. growth)
  • Is the effect durable? (i.e. if you drop prices for a week and customers buy… that’s not a positive test result, that’s customers acting urgently before prices rise again. Similarly if you increase prices - have you hurt the business or just caused a set of customers to wait a while. Price test runtime is tricky)
  • Is the effect repeatable (i.e. what if you slowly cycle back and forth between price points? How does that impact your test-market’s growth vs benchmark markets? Slowly meaning multimonth cycles)
  • Is the effect different across different categories? (often your price point won’t be exactly the same across categories, your brand position and perceived quality will be established more strongly in certain categories, and general elasticity will be different across categories. Are some categories outliers in your test?)