Lab, ethnography, remote: How should we test usability/UX?

This blog follows from talks and discussions at UXBrighton on October 14th. Thanks to speakers from Ethnolabs, Pidoco, Feralabs and Flow. The discussions on the validity and utility of remote testing inspired this blog.

Any UX expert will tell you that testing with real users is the key to successful usability research, but there are a number of methodologies that can be applied in UX research, and applying the right one is dependent on the particular context. The researcher obviously wants the most accurate data they can attain, but they are constrained by time, budget and resources.

Planned lab testing can be useful to measure efficiency because session tasks can be set up in very specific ways. Its also much easier to make use of sophisticated bio-metric techniques such as eye tracking to measure attention and facial expression & galvanic skin response to track emotion. However it may also create a pseudo-context that eludes the reality of the natural user-system interaction.

In many instances ethnography may be more appropriate. This may be done by holding face to face sessions in users natural contexts (e.g. at work or at home), encouraging the user to demonstrate their natural interactions with the system under test, and recording as much data as possible and analysing properly later. This is a research technique I applied in my MSc thesis research with Ubuntu users. This has the advantage of flexibility, because the researcher can control the flow and focus of the session depending on the kinds of interactions and problems that naturally come up.

However, I belief it also has the possibility to create un-natural behaviour through the observer effect, i.e. that people change their behaviour when they feel they are being watched. They may begin to interact with the system in ways intended to meet the researcher's expectations or to impress them. This may lead to in-accurate usage data and ultimately a false picture of user-system interaction. The ability of the researcher to avoid these instances by recognizing un-natural behaviour is crucial to solving genuine usability problems and developing accurate user profiles.

On the other hand remote user testing of screen activity may be a better alternative. We should not forget that in many cases, (particularly with PCs and websites) users interact with systems on their own. The context is that of user and computer, not user and computer and observer. Remote data collection may lead to more accurate usage data. By letting users interact with their system in their natural context and remotely recording data, perhaps at random times of the day, I believe the observer effect is reduced.

Of course there is something missed by using this technique; facial expressions, tone of voice and an explanation of why the user is behaving in the way they do. Furthermore, it will often lead to a large amount of raw data, which can be difficult to properly analyse. However, in terms of actual usage data I believe it may often be an improvement on face to face ethnography. Combining remote observation with other techniques, such as getting users to keep learning diaries as they encounter problems, and face to face feedback sessions; may lead to an accurate picture of user-system interaction and a product's usability.

Ultimately there is no one best technique. Researchers must look at the context of the product under analysis to assess suitability. This context includes things like who the users are, what sort of interactive product is being (re)designed, what stage of development the product is under, and so on.

Likewise, which technique is most appropriate depends on the research question. Do we just want to collect usage data? Is there a particular usability problem we want to solve? Do we want to specifically understand the needs and contexts of the users? Do we want to access diverse user groups? Also other factors such as, time, resource and budget availability, inevitably effect the technique that is chosen.

Knowing how to answer these questions is as much a part of being a UX researcher as performing the research itself.