Tom Bowen — March 20, 2017 
There’s no doubt that conversion optimization is made up of many activities and tactics. Analytics analysis, A/B testing, email marketing, heat mapping, these are just some of the important tactics that go into optimizing a site for conversions. But to me, one of the most under-utilized CRO tactics is definitely usability testing.
By observing real users, who are thinking out loud and attempting to accomplish real tasks on your website, you can gain insight into the user experience that can be provided in no other way. Sure, a heat mapping tool may tell you what is and what is not being clicked on a page, but it will not tell you what users are trying to click but cannot find. Form analysis tools might show you what field users are getting stuck on, but they don’t explain why. Even session replay tools give you only half the picture. You may be able to see where users are having troubles on the site, but without hearing what the user is thinking, you don’t know what is causing the difficulty. You are left to hypothesize.
A heatmapping tool can show you what is and what is not being clicked on a page, but it cannot help you learn what users are trying to click but can’t find. It can’t point you to user frustrations.
The Value of Usability Testing
Usability testing uncovers conversion issues with a website that we are just too close to discover ourselves. You may be a designer, a developer, an optimizer, or the site owner. You may know your site inside and out, but it’s always from your own perspective. That’s an inherent bias that is probably not shared with most of the users of your site.
When you watch strangers use your site and hear what they’re thinking as they do so, you start to experience the site from a broad range of perspectives. This uncovers barriers to conversion that you won’t typically be aware of otherwise. Usability testing can also uncover user concerns such as trust issues, privacy questions, how customers compare you to competition, and more.
I frequently use the term “low-hanging fruit” when it comes to usability testing. It’s very common that a usability study uncovers a problem that was a major obstacle to conversions and is simple and quick to fix. Such high-value, low-cost issues are indeed the low-hanging fruit any conversion optimizer looks for.
Moderated vs Unmoderated Usability Testing
Usability testing can be broken down into two basic methods: moderated, and unmoderated (or remote) testing.
Traditionally, a usability study consisted of a series of test sessions, each with one test user and a moderator. The moderator’s role is to observe the users in their attempts to complete the tasks given to them and to ensure that each test user consistently verbalizes his or her thoughts while performing the tasks.
As technology has emerged to make it possible, Remote Unmoderated Testing has become quite popular. In this type of testing, test users are delivered task instructions electronically and perform the tasks from their own home or office using the software installed on their computers to record the sessions. This can dramatically reduce both the cost and the time required to perform usability tests.
Advantages of Remote Testing
There are two huge benefits to remote unmoderated testing: cost and time. Let’s talk about time first.
Lightning-Fast Turnaround
In the traditional moderated approach, it can take at least a couple weeks to recruit users and schedule a time for them all to come for the testing. And then that testing itself will take several hours–about one hour for each user being tested. So just this component of the project is measured in weeks.
But for unmoderated testing, this component can literally be measured in hours. Once you upload the list of tasks into the testing tool, test subjects are immediately recruited and are often performing the tests simultaneously, within minutes. You can often start watching videos of these tests within the first hour. This is a big-time savings over the traditional moderated approach, and as we all know, time is money!
Low-Cost Investment
Time is indeed money, but an unmoderated approach saves money in other areas as well. Most remote testing tools either charge a flat fee per test/participant, or a flat monthly fee for an unlimited number of tests. Per-test rates typically range from $ 25 to $ 100 per test, and monthly rates tend to range from $ 200 to $ 500 per month. Which provider & which plan you choose obviously depends on your testing needs, but in all cases, it is much less expensive than the cost of a moderated study.
If you are conducting your own moderated usability study, you will incur costs for recruiting test users, paying incentives to them for participating, hiring a moderator, and rental of a test facility at a minimum. For a project consisting of just 12 users, we are already talking about several thousand dollars, compared to several hundred for an unmoderated study.
Users in Their Natural Environment
One other advantage of remote testing is that users are performing your tasks from their own devices and in their own homes or offices. This is good because it more closely matches how people would typically be using your site. It’s a more natural testing environment than for users coming to a strange test facility and using a computer they might not be familiar with. Because of this, it can produce more relevant results.
Benefits of Moderated Testing
So we know that moderated testing takes longer and costs more than unmoderated testing. But surely there are advantages to this approach that make it worth the extras costs, right? Well in many cases, yes!
More Targeted Test Subjects
With most remote testing tools you can specify some very basic demographic requirements in screening who will and who will not serve as test participant for you. But these are indeed very basic. They are typically limited to things like gender, general age, general income levels, and sometimes proficiency with the use of a computer or the internet.
But every site is different. And so are its target users. If your site’s primary target is, for example, a heavy gamer, or an architect, or a single mom, tests performed by a more general population may be less representative than what you need. In a moderated usability study, you recruit your own participants, either yourself or with the aid of a research firm, and you are able to specify exactly the criteria you wish. (Note also that some remote testing tools allow you to recruit your own participants but it can be logistically challenging to equip them with the hardware & software needed to work with the testing tool.)
Most remote testing tools, like this example from UserTesting.com, allow you to specify basic criteria for screening who will participate as test subjects, but the criteria options are typically quite broad.
More Time With Your Testers
With remote testing tools, tests are targeted to last about 10 to 15 minutes for each tester. Some tools have a specific time limit such as 15 or 20 minutes. For those tools that do not have a defined maximum time limit, the testers still have an expectation to be testing no more than about 15 minutes. If you submit long tasks or a great number of tasks that exceed this expectation, test users often tend to hurry through those or get frustrated with the amount of work or questions being asked of them. And the results suffer accordingly.
But the mindset is different for participants in a moderated usability study. The participants drive to the test location. They are being paid more for their time and effort than a remote tester is. They expect to be with you for an hour or more. And you should set that expectation with them before they come.
This gives you much more time with each tester to help you learn more about your website. You can take advantage of this by having them test many different areas of your site, perform several different tasks, or even the same task more than once with slight modifications. You may want them to perform similar tasks on a competitor’s site. You can get significantly more information, about a great deal many more areas of your website through a moderated usability study, due to the extended amount of time you get with each test participant.
More Quality Time = More Quality Feedback
More time with the user is great, but having a moderator there to observe and react can also provide tremendous value. If the test user gets too focused and stops thinking out loud, the moderator is there to prompt them and ask what they are thinking. If the participant misunderstands the requested task, the moderator can clarify. Occasionally a test user might do something completely unexpected or seemingly illogical. In such cases, a moderator can ask the participant why they took that action or what they expected to see. The ability to ask relevant questions in the middle of the test based on a participant’s action often yields some of the most important feedback in the entire project.
The moderator is also able to observe body language, facial expressions, and other non-verbal cues that will not show up on a video of a remote tester’s screen. Our moderators often keep an eye on the feet of test users. Discomfort or uneasiness often manifests itself through nervous feet!
A good test moderator can add tremendous value to the project by observing reactions & body language, keeping the test user thinking out loud, and preventing the user from going off on unproductive tangents. Image Source: flickr
A moderator’s role is certainly not to aid the test subject in performing the actual tasks they are performing but to keep them on task and help them understand the objective of the task, as well as being a keen observer. A good moderator can add tremendous value to the project by doing his or her job well.
The Professional Tester Syndrome
It’s hard to overlook the huge benefits of the low-cost and fast turnaround of an unmoderated remote user study. It may leave you asking, are there any other disadvantages to this method? There is one, and it’s what I call the “Professional Tester” syndrome.
Each remote testing tool has a pool of everyday people who serve as their testing population. Each of these testers has installed the necessary software to receive and perform a test and receives a small payment for each test they perform. It’s an efficient system which enables the low-cost and quick turnaround that these testing tools can offer.
But it can also result in some testers performing an inordinate amount of tests. And this creates its own bias. It’s not uncommon at all to run a test where a test user is more focused on giving you advice about how he or she thinks you should change the site instead of just performing the task. I have seen tests where the tester has said things like, “I think most people would probably do this, but I think you mean that, so I’m going to go about it this way.” I’ve even seen test videos where the user spent more time critiquing the wording of the task itself than actually trying to perform it.
Sessions like these do not mimic a real-world situation at all. Quite the contrary, they introduce so much unnatural bias that they are of little value to the overall project.
If you do have tests which do not provide relevant feedback, most testing tools are extremely obliging about giving you a credit for that test so you can run it again with a different participant. But this still adds additional time to the project, and in a remote testing situation, you probably have not budgeted for this additional amount of time.
So Which Method Is Better?
For pretty much any subject, the answer to that question is usually, “It depends.” And it’s no different this time! The choice of moderated versus unmoderated really does depend on your situation: your website, your users, your objectives, etc. Certainly, budget and time requirements are key inputs to this decision.
There are some situations which clearly lend themselves to one method over the other, however. If your site has been in business for a while but has never had any type of conversion optimization done with it, I would say it is very ripe for a complete, moderated usability study. There is probably a lot of low-hanging fruit to pick and simple changes to make for some high-yielding improvements. Similarly, if you are about to start a consistent A/B Testing program for your site, but aren’t sure where to start, a moderated usability study can give you lots of feedback from which to form hypotheses to test.
There are a lot of great applications for remote testing as well. One is definitely to show the value of usability testing to someone unfamiliar with it, to acquire more budget to run more in-depth tests. Few things can be so enlightening to a director or a vice president as a 30-second video of a user being unable to find a checkout button or how to add a product to a cart. A handful of inexpensive remote user tests can quickly bring to light. This can create new budget money to run much more extensive tests (either moderated or unmoderated).
Iterative Testing
Another great application for remote user testing is the concept of “Iterative Testing”. Especially useful in the development phase of an application or website, iterative testing refers to conducting several tests with a small number of users as opposed to one study with a larger number of users.
The idea here is that the earlier in the design and development phase, the more likely it is for major usability issues to show themselves. It only takes a couple of user tests earlier on in the development cycle to identify the major showstoppers. As those major items get identified and ironed out, the number of testers needed grows somewhat with future tests further down the development cycle. In other words, if you have a budget to test with 20 users, you can get much more bang for your buck during the development phase by spreading those over four or five testing rounds of just 2 to 6 participants, as opposed to just one project with 20.
Iterative testing involves more rounds of tests with fewer participants earlier in the development cycle. Here, testing was split into 7 rounds, often with just 1 user to identify showstoppers.
Iterative testing is a great application for unmoderated testing, because you do not have to invest in things like a facility or moderator for each round of testing. You can easily and quickly ship the newest version of the page or app out to a couple of testers and watch them use it. Then you can make the necessary changes & adjustments before going on to another iteration. It is inexpensive, and fast, and ideally suited for the development phase of a project.
Conclusions
In the end, the better approach really does depend on your situation. If you are new and inexperienced with usability testing and are looking to just test the waters, remote testing is an ideal place to start. Likewise, if you have a limited budget or are short on time, or just want to test one very specific function on your site, remote testing will be your friend!
On the other hand, if you have a very specific or specialized target audience you need to test with, or your testing needs are broad, needing to see people use many areas of your website, then you may want to look into a more in-depth, moderated usability study. If you are looking to start a new conversion optimization program and don’t know where to start with your site, or don’t know where the biggest problems lie, then again a full usability study is probably the better direction for you to go.
There are many of you out there with experiences with both kinds of usability testing. I’m interested to know what your thoughts are regarding the advantages and disadvantages of each method. Please leave your thoughts in the comments below to share with all of us!
Digital & Social Articles on Business 2 Community(62)
