Photo by Landon Martin on Unsplash

A Product Manager’s guide to Painted-Door Tests

How your Product Team can learn faster and reduce your cost of experimentation

What is a Painted-door Test?

A painted-door test, sometimes called a fake-door or smoke-tests, is a method of early testing to gauge if your users would engage with a particular feature — it primarily focuses on tracking the completion of a main call-to-action. It is a way of exposing a partially or non-functional feature to your users to test their level of interest. This allows you to validate if this is something a customer truly wants in your product, or if you need to reevaluate the idea.

The windows in the centre of this picture are actually painted — adding some personality to the exterior an elevator shaft or some other internal that does not warrant windows. Image source: Coolstuffinparis.com

Using Painted-Doors

So what should a painted-door test look like? As close to the real thing as possible. There should be a clearly defined Call-to-Action, which is what you will use to measure take/conversion) rate. The metric you are measuring should be as close to the end (if not the end) of the funnel you are testing. The further down the funnel, the more accurate your data will be. It could be a setting toggle in the account area, a promotion banner with a buy button or even an out-of-stock product. For all these examples, the measures would be the amount of users that switch the toggle, click buy or attempt to add the item to their cart. All are valid methods but the approach will depend entirely on your product and your initiative

A hypothetical painted-door test to quantify interest in a tip the author functionality.

Drawbacks

The drawbacks for users are obvious. Those that have expressed an interest are faced with a switch-aroo. The are offered something, commit and then are ultimately told by your organisation it’s not available. It may feel like the rug has been pulled out from underneath them. Ensure you communicate well after the test. There are different levels of transparency you may want to present based on the test and on your brand. Use copy to empathise with your customer and apologise for the inconvenience. Let them know they are helping to make the product better and you’re working on having the feature available. You could even follow up with an email to reinforce the message and even offer a token compensation or discount.

5 extra tips for running Painted-Door Tests

  • Couple them with heatmaps (such as Hotjar) to understand if there is passive interaction with the test. Passive interaction is when a customer hovers around the test but does not engage — perhaps there is further potential.
  • understand if there is passive interaction with the test. Passive interaction is when a customer hovers around the test but does not engage — perhaps there is further potential.
  • Try basic segmentation to understand which users converted — new or high value customers? Low or high engagement? Desktop or mobile? It will provide extra insight when deciding how to proceed.
  • Follow up with the customers that demonstrated interested if you finally release the feature. They often appreciate that someone actually followed up.
  • Don’t over do them. They can be a frustrating experience for your audience.

Conclusion

Painted-door tests are a useful method in your experimentation toolbox — however, they don’t work in isolation. Painted-door tests are best placed early in the life of an initiative to act as a gate for the next stage. They answer, in a binary way, “will my customer use this?” They are a way to reduce the risk of a hypothesis with limited data — particularly ones that require extensive engineering or business investment.

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Curtis Stanier

Director of Product at @DeliveryHeroCom. Formerly @HelloFresh, @BBC, @Atos. Passion for product, business &tech. I like helping people solve problems. Berlin