Most organizations are constantly trying to improve their service; both to maximize customer engagement and increase revenue. Furthermore, just about everyone seems to be a self-proclaimed expert on how to fix things. But as ideas start flying, it quickly becomes clear that ideas themselves are not enough. Ideas conflict with ideas. Some ideas take longer to implement than others. To top it off, ideas come from many different perspectives ranging from designers and developers to call centers and sales and marketing. How do you decide which ones to do?
A few years ago I was introduced to Strategyzer. This is the group who brought us the Business Model Canvas, the Value Proposition Canvas, along with many tools to understand your business, customer segments and products you offer. Strategyzer also has a process for testing ideas to improve a product or service. Who knew? Well, now you do.
This is it in a nut-shell. 5 Stages to discovering if an idea has merit. To help you understand with process more, I’m using an actual idea my team ran through this process.
A Word of Caution
Alright, I’m gonna be straight with you. The example project I use in this article probably shouldn’t have gone through this process. But why? Jakob Nielsen has a set of heuristics; a broad, rule-of thumb principles for designers. The very first one on that list is Visibility of System Status which says: “The system should always keep users informed about what is going on, through appropriate feedback within reasonable time.” This feature didn’t need to be tested. Any experienced designer could tell you that. In fact one of our designers saw an issue with our service months earlier, built out some prototypes and proposed this as a solution. The name of the project was even VOSS (Visibility of System Status). But it never gained traction. It just sat in a folder collecting dust. So when I formed a new team, 6 months later, focused on testing ideas, it was the first idea I chose to test.
Here’s the takeaway: At some organizations, designers are second-guessed and not trusted to do their job, which is to design. This process for testing design ideas is best suited for ideas that are risky or ones where you are uncertain of their value. But I have used it to get projects pushed through too. I hate to admit it, but sometimes that’s what it takes. I sincerely hope this process helps you out in your work.
Understanding the Problem
The hypothesis or idea we had to improve our service, centered around a screenshot counter. The service we were working on takes screenshots of a person’s digital activity, then uses AI to determine if there was any explicit activity, then send a report to pre-determined individual who reviews the report of the activity. This service is typically used by people who struggle with pornography addiction as well as parents who want to keep their kids away from explicit activity. Here’s the problem we had though; people didn’t know if the software was actually working until they got a report of activity. There was nothing in the software that indicated screenshots were being captured. A designer came up with a couple of ideas; a screenshot counter and a more detailed log of screenshots taken in a period of time. One was costly to build while the other was cheaper to build. When everything was said and done; time and resources spent and value delivered, which one would have been worth the effort? We wanted the most bang for our buck.
Now, let’s look at the process.
Hypothesis
These are assumptions, educated guesses. Ultimately, this is what we think will address customer pains and gains. These can be pulled from various sources in the company and impact various places in the service.
This was ours: “We believe that users that can see the service is working on their device have a greater confidence that the system is actually working than users that only receive a daily report.”
This is what we were testing.
Design Tests
Step 1: State Hypothesis
We believe that… users that can see the service is working on their device have a greater confidence that the system is actually working than users that only receive a daily report.
Critical: Medium
This is addressing a concern in our service to that makes it higher than your average idea. But it’s not an attempt to put out a fire in the service. It’s a medium.
Step 2: Test
To verify that, we will…
Conduct 4 unmoderated tests (surveys) with 4 different groups of 50 participants per test. Create 4 separate prototypes consisting of current state, screenshot capture counter, screenshot history and screenshot capture counter/screenshot history combo. The current state is the control group. The other 3 are the test groups.
Test Cost: Low
We are using flat prototypes and surveys. This is very cheap in cost.
Data Reliability: Medium
We are using a control group and 3 different types of tests, all to compare to one another. We have a good sample size too. But these are unmoderated surveys using a single, flat UI. The data reliability would be higher if we did interviews or multiple screens within an interaction flow or things like that.
Note: Gave up some data reliability for test cost. We needed to prove we could test hypotheses quickly and cheaply without giving up too much data reliability.
Step 3: Metric
And Measure…
We were primarily concerned with measuring Confidence in the system and Clarity of what was happening. We were concerned with answering questions like “Do they know that screenshots are being taken?”
We were also measuring Usefulness. Basically we wanted to know; “Is having Clarity and Confidence useful to them.”
Lastly, we wanted to know which combination of features gives the most confidence and clarity?
Time Required: Low
From the time we sent out the surveys and collected results, it was going to take about a week.
Note: Decided not to test findability. This means, we decided where to put the counter and the history within the menu of the app. Before releasing this to users, we’ll want to test usability and make sure people know where to go to find this info. But that’s out of scope for this experiment. We just need to know if this feature is valuable.
Step 4: Criteria
We are right if…
Any test group has 50% increase in confidence over control group that screenshots are being taken.
Any test group has 50% increase in clarity over control group.
Test
Alright, we’ve got the thing we are testing. We’ve got the plan in place. Now we actually do some testing. This is a snapshot of the parts to that.
Test: Build
Prototypes designed by a Product Designer
Surveys and testing pool set up by Design Researcher
Test: Measure
Running Unmoderated testing through Usability Hub
Test: Learn
Collecting data directly through Usability Hub then compiling in Google Sheets.
Capture Learnings
Step 1: Restate Hypothesis
We believed that users that can see the service is working on their device have a greater confidence that the system is actually working than users that only receive a daily report.
Step 2: Observation
We observed that…
- people had a 25% confidence that the system was working when they saw a design for the Current State.
- people had a 25% confidence that the system was working when they saw a design for the Current State plus the Screenshot History (list).
- people had a 64% confidence that the system was working when they saw a design for the Current State plus Counter.
- people had a 65% confidence that the system was working when they saw a design for the Screenshot History plus Counter
Data Reliability: Medium
Step 3: Learnings and Insights
From that we learned that… the presence of a Screenshot Counter provided a greater degree of confidence that the system was working.
When the Screenshot Counter AND Screenshot History were together in the design, confidence was at 65%. This was the highest confidence. However, the Screenshot History on its own scored a 25%, the same as the original design. Whereas, the Screenshot Counter on its own scored a 64%. This told us there was tremendous upside to the Screenshot Counter alone.
Action Required: High
Step 4: Decisions and Actions
Therefore, we will… recommend Screenshot Counter as a project to Product Design Team.
Revisiting the Criteria: We are right if…
Any test group has 50% increase in confidence over control group that screenshots are being taken.
Any test group has 50% increase in clarity over control group.
The control group was 25%, the Screenshot Counter was at 64%. That is definitely more than a 50% increase.
In addition to the criteria there is a business case for the counter over the history.
Out of the two options, the Screenshot Counter was cheaper to build by far and it was more effective. So though we could build both if we want the maximum increase in value. It’s not worth the cost to add on the counter history as well.
Extended: Recommend Screenshot History to the Hypothesis Hopper (backlog of ideas to be tested) as part of Image Sharing (customer pain and potential project).
Assessment
With every hypothesis we test, by the end of it we want to tie it up with a neat little bow. Was this idea Validated, Invalidated or Unclear? With this project, the answer seems fairly clear and with others you could likely determine that from Step 4 of the Learning Card, Decisions and Actions. Even so, being able to wrap up your testing in this way is extremely beneficial.
Invalidated
Put it on the shelf or in a dark corner. pick up something else to test.
Unclear Results
Conduct further tests to seek confirmation.
Invalidated
Hand-off to Product for future planning, design and development. Figure out a way to get this to your customers.
Validated!
For this project, our hypothesis was Validated.
This has since shipped. Check out the screenshot I took while writing this article.
My phone’s been charging so there’s only the one screenshot in the last hour. Normally there’s closer to 60 within an hour.
Wrapping Up
In summary, this works. It’s a cheap, effective way to validate design ideas to improve your products and services. It can used in small or large organizations with huge teams or teams of one. This process can be adjusted to fit your needs and will help you in the end to have a more success business.
So, like, do it. Seriously. The process I’m using and highlighted today comes straight out of Value Proposition Design. It’s not original to me and it’s not a trade secret. Go buy the book. And while you’re at it, but their other books.
One last thing. I’d like to point out that I don’t work for Strategyzer nor do I get kickbacks for recommending their tools. My team has been using them for a few years and found them very powerful. Go try them yourself.
