Advice on setting up your growth experimentation framework properly
Facing a similar challenge?
- Talk to someone who has been there before
- Get unbiased insight from industry veterans
- Work through the issue with a mentor by your side
The success of any growth experiment strongly depends on how well it is set up. Are you using a standardized and repeatable framework? Are your growth experiments prioritized with respect to other potential experiments that could also drive growth?
Here’s a simplified example of how to set up an experiment:
- Experiment Title: A/B test if a chatbot is a good substitute to a form to drive more registrations
- Step of the Pirate Funnel: Acquisition
- Hypothesis: Our visitors are more willing to provide their personal details to an interactive chatbot rather than to a traditional form since our audience is used to interacting with conversational interfaces in their lives.
- Implementation: We’ll build a chatbot using Chatfuel. This will ask our visitors the same questions as our existing signup form. We’ll use Google Optimize to split the traffic to the signup page as follows: 50% will see the form and 50% will see the chatbot. We’ll track the interaction in the page adding a heatmap (using Hotjar) and recording the sessions (also using Hotjar), and we’ll use Mixpanel to build a funnel showing which questions are harder for the visitors to answer and what is our conversion ratio.
- Duration: 2 weeks (to get a data set of at least 1,000 people).
- Win threshold: The experiment is a success (the hypothesis is validated) if the conversion ratio from unique page view to signup is 5% higher via the chatbot than via the form.
Once this process is internalized, you’ll start thinking about implementing features as an experiment. As a result you’ll start getting better at feature prioritization based on real data.
If you’re currently running growth experiments, speaking to a growth mentor can come in useful. You’ll be able to share with your mentor how you are structuring your experiments, what you’re measuring, and how you’ve split responsibilities across your team.
Getting a second perspective on your growth experimentation framework can help you head off potential mistakes down the road, increasing your long-term growth potential.
MENTORS WE RECOMMEND FOR THIS TOPIC
“Jason was a rockstar. Asked lots of insightful questions, listened, and ultimately delivered 3 very actionable tips, that I am in the process of implementing. From start to beginning, great energy, I knew I was talking to a veteran in Growth Hacking space.”
“Just finished a great session with Jason. He was very tailored in applying his extensive experience/background to what I was looking for help on – hiring a remote marketing team. I could have easily chatted another hour! Super helpful, and an enjoyable experience with some clear takeaways. Thanks Jason.”
“Had a great chat with Michael regarding app analytics and measurement! I loved his unconventional approach to certain metrics! Very much looking forward to our next call. :))”
“This session was really helpful. I needed to get more knowledge on tracking tools and more specifically on how to track new activity from affiliates. I liked Michael’s approach as it was a good mix between taking a step back looking at my need and giving context on each of the tool I could use, and more precise guidance on how to implement the recommended solution in Mixpanel. Thank you!”