Taking shortcuts in UX research

In my family, when I was younger there was something known as an “Alexander shortcut.” My family determined that any route that took longer than the original route fit into this category. One route in particular was known for this: my route home from school. I’m still convinced it was shorter, as it did cut off a fairly significant corner, although it did meander through the woods, and over and around the mounds from the dirt dumped from some new builds. Perhaps it seemed shorter because rather than a long hill that was half the route, I’d climb a very short very steep hill. There were still the dirt mounts, and a hill to go over, that the normal route would have bypassed. If it was longer, it seemed shorter to me.

Heuristics is a shortcut

Of course, there are a lot of places we try to take shortcuts. Recently, I’ve seen more than one person attempt to take an Alexander UX shortcut. They post two pictures of a product on a forum, asking the community to use voting buttons to signal “which one is better.” The problem? Unless respondents also add comments, you don’t get feedback on why they responded the way they did. You don’t know if they are responding based on visual appearance, interaction design or some other factor. And you don’t know what their training, experience or background is — are they even qualified to comment? Unlike my route home, this shortcut might not get you to the pace you want to go.

At best, someone familiar with UX heuristics could do a heuristic analysis of your designs. Psychology Today describes a heuristic as:

A mental shortcut that allows an individual to make a decision, pass judgment, or solve a problem quickly and with minimal mental effort. While heuristics can reduce the burden of decision-making and free up limited cognitive resources, they can also be costly when they lead individuals to miss critical information or act on unjust biases. (Psychology Today)

Asking for usable feedback is never a bad thing; however, relying solely on a heuristic analysis isn’t sufficient to get the full picture. Your two designs may be heuristically equal, in which case, your shortcut just walked you in a big circle. Alternatively, users’ experience might cause a heuristically good design to be a failure. Heuristic analysis is good, but it is no substitute for testing your design with real users.

Heuristics alone can fail

For example, Jakob Nielsen’s 10 Usability Heuristics, identifies consistency and standards as a heuristic. This heuristic could be applied in multiple ways. Jakob’s law states that “users spend most of their time on other sites.” Applying this heuristic principle along with Jakob’s law would mean that your design should follow similar patterns to other products or websites; however, if you have users that have expectations about how to navigate your product, you might alienate them or drive them elsewhere. You will still want to make sure your site or product maintains internal consistency. In a case like this, you might need to gently move your users closer to consistency with other sites rather than jumping straight there.

When creating errors messages, another heuristic, the language needs to be understandable and help users recover from errors; however, applying language used by other sites (again, the consistency heuristic) that doesn’t resonate with your users could create more problems than it solves.

Research is the solution

Taking heuristics into account when creating designs to test is a good starting point. Multiple designs can meet all of the heuristic challenges catching things you can predict and identify. It will prevent your test designs from making predictable errors.

When designing user experience tests, asking a question like which do you like better? isn’t going to be helpful. It is too broad and general to elicit more than an unqualified preference. You want to observe and understand where users get tripped up, so hearing details of what they think and see is important. Following up during the test to ask questions about things you want to better understand or clarify are important.

Ask questions that don’t place judgment on the user’s abilities like While you were doing the task, did you notice…?, Did you find anything particularly easy or difficult about that task? or, when someone is successful, If you’d not been successful in completing the task, what would you have done? to better understand the results.

And presenting two designs and having users work through both of them could bias the results as they will see one design before the other. As you will need to ensure that some users see the second option first, you’re unlikely to save resources by doing this. If you do, you’ll want to ensure that your study is balanced, ensuring that you have users that match each of your personas in both groups.

Conclusion

As a starting point, heuristics can help to get you to where you want to go. Alone, they might not help you meet your organisation’s goals and they might not help your users be successful in their goals for using your product.

David Pullara stated:

You don’t evaluate creative work on whether you like it or not. If you’re not the target audience, you can absolutely hate it… and it can still be a fantastic, effective piece of work.

You evaluate creative work based on how well it delivers on the creative brief. Full stop. (Evaluating Creative)

The same is true for UX. Does it deliver what your users need, does it accomplish the goals and needs of your organisation and does it allow users to be successful in their goals for using your product? That you can only know this based on testing.

What matters is not whether people prefer one design over another, nor does it matter which one your boss likes. Heuristics alone are not going to answer that question. They are general and focus on known problems, only catching the predictable.

If all you’re looking for is superficial feedback because two designs have tested equally well, voting buttons may be appropriate, but they will only provide a superficial response. Which works best for your users is what matters: which one generates the conversions you require or allows you to meet the goals of your product. As with voting buttons, heuristics alone is not nuanced and does not take into account your users. The goal is to make your product a success. Sometimes the best shortcut is not to take a shortcut at all.