The success of any growth experiment strongly depends on how well it is set up.
- Are you using a standardized and repeatable framework?
- Are your growth experiments prioritized with respect to other potential experiments that could also drive growth?
Here’s a simplified example of how to set up an experiment:
- Experiment Title: A/B test if a chatbot is a good substitute to a form to drive more registrations
- Step of the Pirate Funnel: Acquisition
- Hypothesis: Our visitors are more willing to provide their personal details to an interactive chatbot rather than to a traditional form since our audience is used to interacting with conversational interfaces in their lives.
- Implementation: We’ll build a chatbot using Chatfuel. This will ask our visitors the same questions as our existing signup form. We’ll use Google Optimize to split the traffic to the signup page as follows: 50% will see the form and 50% will see the chatbot. We’ll track the interaction in the page adding a heatmap (using Hotjar) and recording the sessions (also using Hotjar), and we’ll use Mixpanel to build a funnel showing which questions are harder for the visitors to answer and what is our conversion ratio.
- Duration: 2 weeks (to get a data set of at least 1,000 people).
- Win threshold: The experiment is a success (the hypothesis is validated) if the conversion ratio from unique page view to signup is 5% higher via the chatbot than via the form.
Once this process is internalized, you’ll start thinking about implementing features as an experiment.
As a result you’ll start getting better at feature prioritization based on real data.
If you’re currently running growth experiments, speaking to a growth mentor can come in useful.
You’ll be able to share with your mentor how you are structuring your experiments, what you’re measuring, and how you’ve split responsibilities across your team.
Getting a second perspective on your growth experimentation framework can help you head off potential mistakes down the road, increasing your long-term growth potential.