Business experimentation is a process of testing and validating new ideas in a business setting. The goal is to reduce the risk of failure and increase the chance of success. It involves taking a hypothesis about what might work in a business and then testing it out in a controlled way to see if it's actually effective.
By using a structured approach to testing and validating new ideas, businesses can reduce risk and increase their chances of success.
In this post, we will explore the benefits of business experimentation and how it can be used as a key driver for innovation and growth in modern businesses:
- When Blockbuster went bust
- A key ingredient to startup success
- Types of experiments
- How to conduct a business experiment
- Caveats of business experiments
- Advantages of running experiments
- Additional Resources
When Blockbuster went bust…
For those of us old enough to remember, or perhaps those living close to the last standing location, Blockbuster was a dominant force in the video rental industry. It was a big hit retail store with millions of customers, massive marketing budgets and efficient operations. I remember the excitement of walking in and looking through the video racks, picking up a new release on the weekend. Blockbuster was the go-to destination for movie rentals. However, their grip on the market was about to be challenged by a new player - Netflix.
The game-changing introduction of streaming by Netflix ultimately led to the downfall of Blockbuster. As Netflix continued to grow and innovate, Blockbuster struggled to keep up. They filed for bankruptcy in 2010 and eventually closed all of their remaining stores.
The story of Netflix's rise to dominance toppling down a much larger competitor is one of many. Another example may include Amazon’s rise and Macy’s struggling to remain competitive in the retail industry.
A key ingredient to startup success
The charts above illustrate a comparison of survival rates between Fortune 500 companies (left) and US startups (right). Over time, it’s become harder and harder to remain dominant as a large scale company in the US while startups, even in the face of great change, tend to have very long-time survival rates.
So what is it about start-ups that attributes to their superior ability to survive, if not thrive, over a longer period of time? We can say it’s their ability to pivot, to respond quicker to market changes, etc. and these are all right.
Turns out, research points to one key factor - It’s experimentation.
Not just any experimentation. It turns out just making up business experiments isn’t good enough. There’s a set of studies that show that going through the process the right way makes all the difference.
The key to start-up success is business experimentation, and going through this process the right way makes all the difference for launching it.
Types of experiments
The purpose of running an experiment is to give you an answer to your hypothesis quickly and with the least possible resource expenditure. There are many kinds of experiments you can run to test your hypothesis, including the following six:
- Customer interviews - Really good way of learning about a marketplace
- Surveys
- Smoke tests - Appearance of a product before even having it built
- Prototyping and/or wireframes
- A/B or split testing
- Market trials
How to conduct a business experiment
So you can’t just perform an experiment and expect a positive outcome. The goal is to reduce your uncertainty about the world. Turns out there’s an optimal way to do so as there are 3 general steps for conducting a successful business experiment
Step 1 - Surfacing Assumptions
The first step of experimentation is surfacing assumptions. Assumptions are the things that need to be true about the world for your business idea to work. What needs to be true about the world for this to work? What do you know? What do you not know?
You ultimately need to figure out what needs to be tested and how to test it. This helps to clarify what the team believes to be true about their product, market, or customers. Identifying and challenging assumptions can also encourage creative thinking and generate new ideas, leading to better products and more successful outcomes.
Step 2 - Developing a Hypothesis
The next step in the process is to turn your assumption into a hypothesis, and the key to developing a valid hypothesis is anchored on the following 3 criterias: You are testing one key assumption - You build your hypothesis around your key assumption, and you run an experiment that contributes to your hypothesis. Your hypothesis is testable - You are able to build a testable hypothesis that you can run today. Your hypothesis is falsifiable - Essentially a yes or no answer. For example, you can’t say “people will like our product” because “like” is subjective and not measurable. It should have an acceptable threshold
Here's an example of a good business experiment hypothesis:
If we offer a personalized onboarding experience to new users, we will increase their engagement and retention by 25% within the first month.
This hypothesis is good for several reasons. Firstly, it is specific and measurable, with a clear target metric (engagement and retention) and a specific percentage increase goal (25%). Secondly, it is testable and has a clear cause-and-effect relationship between the intervention (personalized onboarding) and the outcome (increased engagement and retention). Finally, it is grounded in a plausible assumption that a better onboarding experience can positively impact user behavior.
Step 3 - Testing the Hypothesis
Once your hypothesis is formulated, you want to then design and conduct and experiment to test your hypothesis. An experiment gives you an answer to a hypothesis quickly and with the least possible resource expenditure. Generally speaking, designing a good business experiment requires careful planning and consideration of several key factors such as a defined objective, sample size and possible control groups, measurement metrics, etc.
Let’s explore some possible types of experiments you can conduct in your business to test your hypothesis.
Caveats of business experiments
Don’t just climb hills. Look for the mountains.
This is especially true in larger organizations and at organizations that are good at doing experiments. Tweaking certain elements can get you a slightly higher or slightly lower outcome.
The problem with experimenting on very small changes means you end up missing the mountains - missing big opportunities because you’re too focused on the small ones. Experimenting on very small changes takes away resources from spending more time on radical experiments. You can try different small experiments that will give you quick indications about the huge opportunities out there instead of small experiments that optimize what you already have.
Unexpected outcomes
Experiments can sometimes produce unexpected results, and it's important to be prepared for this possibility. It's important to approach experimentation with a mindset of learning and to be open to revising or even discarding initial hypotheses based on the results.
Bias
There is always a risk of bias in business experimentation, whether it's in the design of the experiment, the selection of participants, or the interpretation of results. It's important to be aware of potential sources of bias and take steps to minimize their impact.
Advantages of running experiments
Data-driven decision making
Business experiments provide objective data that can inform decision making. Instead of relying on intuition or guesswork, businesses can use experimentation to test hypotheses and make decisions based on empirical evidence.
Risk reduction
Experimentation allows businesses to test new ideas on a small scale before committing significant resources to full-scale implementation. This can help reduce the risk of costly mistakes and failure.
Innovation and creativity
Experimentation can help foster a culture of innovation and creativity within a business. By encouraging employees to generate new ideas and test them through experimentation, businesses can stay ahead of the curve and remain competitive in rapidly changing markets.
Business experimentation is a powerful tool that can help organizations make data-driven decisions, reduce risk, and drive innovation. By constantly testing new ideas and taking calculated risks, companies can significantly increase their likelihood of success and ability to adapt quickly to changing market conditions.