Experimenting is at the core of building and tying new ideas. A opposed to having a clear problem to solve, experiments are designed to “try” out ideas that you have and yet know if they will work or not.
At most startups, I notice two primary “ends of the pendulum” issues. Most (over 90% of startups) dont run enough experiments. The rest (< 10%) run too many experiments in parallel.
If you fall into the first category, then my only suggestion is to think about experimenting and commit to doing one. The AirBnB blog is a great place to start, in terms of understanding the user experiments they run. They actually have a experiment reporting framework, which shows how evolved their thinking is in terms of this facet of work.
I also am a fan of the “test and learn” mindset. Since I believe that working on your startup is more a journey in self realization than of markets, problems and customers, it is important to keep learning. I do also believe the best way to learn is to try and do small experiments which you can scale.
I have been a fan of disciplined experimentation.
This post is about the < 10% who run too many experiments in parallel. That’s the biggest challenge I see with startups that hire amazingly entrepreneurial talent for their first few hires.
Since each of the first 5-10 employees are entrepreneurs themselves, they all tend to run multiple experiments, either with product, marketing, customer acquisition, sales, etc.
The framework I have for thinking involves 3 “sets of steps”. I call it “Trail, Nail, Scale”.
The “Trail” comprises of 5 steps, the “Nail” comprises of 3 steps and the “Scale” comprises of 2 steps.
Here is a visual to think about it.
Trail Nail Scale Disciplined Experimentation
Obviously this is very early thinking, but I’d love your feedback.
The way to think about experiments is you to pass through gates and assign the appropriate resources at each stage and have a “rough sense” of what you are trying to achieve. If you know exactly what you want to get out of your experiments, you are not “experimenting”.
What I have noticed is that the 3 stages end up being a funnel. There are many experiments you run, a few of them you will nail and a fewer of them you will scale.
If you have 100% of your “experiments” when you start, (on the left of the graphic), then 20% (or less) will be nailed and 10% you will scale.
In terms of allocating time and resources (if you dont have a large team as a startup, allocate your time), 10% is spent on “Trailing”, then twice that time or 20% on nailing and 70% on scaling.
There are many questions that this throws up, which I want to address over the next few days.
1. How many experiments should you run at the same time?
2. How do you define the success of an experiment?
3. How do you internalize and document the learning from your experiment?
4. How much “money” should I spend on trailing? How about in nailing?
5. How do I leverage lean principles into this thinking of Disciplined Experimentation?
Anyway, I’d love your feedback on this framework. As I share more of my work, which I am interviewing people in larger (Unicorn) startups at, I will also give you some case studies to see what they learned.
