Highlights of “Usability Engineering in the Wild: How Do Practitioners Integrate Usability Engineering in Software Development?”

Back in 2014, I published this study together with a co-author. To make it more accessible, I have written the following summary highlighting the essential findings and implications of the study.

In the paper, we presented an explorative study about how usability engineering is conducted and understood in practice. Our research question was: “How do practitioners perceive and integrate usability engineering in software development?”

Through 12 qualitative semi-structured interviews with participants from 12 different software development organizations, we wanted to get an understanding and extending existing research about how usability is conducted and integrated into practical software development. By extending existing research, the intention is to support the ongoing process of refining and developing new methods and approaches for supporting practical usability engineering.


We found that:

  1. By practitioners, usability is used as an ambiguous term often mixed with other concepts, especially user experience (UX). Usability was by some described as a good feeling when designing. What makes usability ambiguous is that the practitioners rarely set and defined the objectives and purposes of usability. Regarding UX and usability, UX was linked to the overall design. Usability was linked to individual functionality and narrow design aspects, for example, the design of an interface object such as a button. Missing usability competencies in some organizations made it challenging to customize evaluations and decide when and what to evaluate.
  2. Usability is not easily explicable and was found difficult to justify and make credible. Initial designs are based on domain knowledge, and user input is used for adjustments, feedback, and visions. Visions for usability are set, but measurable goals are not.
  3. Informal and ad-hoc evaluations are common because developers need ongoing feedback, that cheaply, easily, and fast can be implemented. The practitioners did not follow a clear plan, or organized evaluation steps. The organizations had built up a body of principles and standards for individual software types – local de facto standards. These local de facto standards are developed because it’s easy, fast and useful, to keep following the same principles used in previous projects.
  4. Reporting of usability problems is done in a wide variety of ways, with the common denominator being lightweight. This can take the form of simple notes, presentations, workshops and user stories. This fits into the light hierarchies, the fast pace of projects, and makes it possible to act upon identified problems during the development process. Problem-solving is done through discussion with colleagues, and by getting inspiration from other sources such as blog-posts and online forums.
  5. All organizations followed an agile development approach. The advantage is that this approach is dynamic and allows changes during development. The disadvantage is that the focus on sprint completion can become the main factor leaving out proper or any usability engineering.

In conclusion, we found that local de facto standards highly drive practical usability engineering, and the practitioners conduct usability engineering informal and lightweight. This includes setting goals, including users, evaluating, and reporting usability problems. A majority of the development projects are completed within short time spans, and the participants need ongoing feedback that can be acted upon during development. Especially these factors are central, as such fast-paced environments do not leave much time for planned and organized usability engineering. Especially the local de facto standards were seen as an efficient approach towards creating robust designs fast. Agile development is a double-edged sword. It allows dynamics between developers and designers, but the focus on sprint completion can take away attention from usability engineering. Missing and limited usability competencies also have an impact. Some examples are ineffective inclusion of users, as well as challenges planning and customizing evaluations to specific conditions.


Some implications of the study are:

  1. Developing scales for prioritizing different user needs could be used for structuring, and facilitating discussions and decision-making when prioritizing usability problems and which product features to include. We support the idea of such straightforward scales. They are easy to understand, learn, and can be used by teams consisting of people, with a wide variety of backgrounds and roles.
  2. Developing approaches and facilitation for workshops seems relevant. Workshops are already popular because it’s a dynamic approach, but practitioners have problems using workshop sessions optimal. This is especially true when including users, because the practitioners experienced problems both running the right activities, and directing the users towards providing relevant feedback. We see a potential for optimizing an already popular method, and using existing guide lines when conducting workshops.
  3. Local de facto standards were used as local interaction design patterns. Future methods for documenting, sharing, and communicating local de facto standards within organizations could support building up a corpus of local design patterns.

Bornoe, N., & Stage, J. (2014). Usability Engineering in the Wild: How Do Practitioners Integrate Usability Engineering in Software Development?. In International Conference on Human-Centred Software Engineering (pp. 199-216). Springer Berlin Heidelberg. DOI: 10.1007/978-3-662-44811-3_12

A few words about selling UX

A while ago I attended a seminar for practitioners about the differences and similarities between usability and UX and not least the problems of understanding, separating, and combining the two into something specific. During this seminar, a problem discussed among the practitioners was how to sell UX.

There are a number of challenges when presenting and selling UX to clients. UX is a “fuzzy” term not easily explicable when it comes to what is delivered, what UX looks or feels like, and how the value of UX is made quantifiable. For example, a chief product officer of a small software development organization told me once: “…how does one get usability included into business cases so that they are credible higher up in the system?”

On top, several different UX definitions exist, and UX is easily confused with or used as a synonym for ‘usability’ or “user interface.”

The following advice was mentioned at the seminar when talking to clients:

  • Use facts about UX.
  • Include UX people during sales meetings.
  • Include UX explicit into business cases.
  • Outline the intended UX design process.
  • Show examples of how UX methods can form a product.
  • Provide examples of how existing knowledge about UX design can be used in new projects.
  • Compare with UX strategies taken by competitors.
  • Show something “beautiful” early in the process.
  • Get allies in the organization.

“Help users recognize, diagnose, and recover from errors”

Outlook password changeAt my organization, Aalborg University, it is a requirement to change the campus account password once every 90 days, a security imitative implemented last year. This is a widespread security policy used in many organizations, but also a policy whose significance has been questioned for more than a decade. I have very mixed feelings about this security measurement. A major advantage is of cause that leaked passwords will be unusable at some point (not considering the option that backdoors etc. can have been installed.) However, this approach is also associated with several obstacles from a user perspective. These include: coming up with a new easy to remember secure password and the hassle of changing password on all associated services requiring authentication. At Aalborg University, this applies to basically all IT-services such as access to WiFi, e-mail, printers, databases, etc. The new password has to be changed manually in several of these services.

Perhaps it’s because I’m a Mac user, but no notice is given about the upcoming expiring password. When I suddenly no longer can access different services I know it’s time to create a new password (after some frustration about trying to figuring out what the problem is.) Our passwords are changed through the Outlook Web App. To make sure that the password meets a certain security standard some requirements are in place. If the new password does not match this standard, the following error message is displayed:

“The password you entered doesn’t meet the minimum security requirements.”

Unfortunately, this error message does not tell anything about what the requirements are or how to get this information leaving the user in the unknown. This is a textbook example of a usability problem directly linkable to one of Jakob Nielsens’s ten heuristics:

“Help users recognize, diagnose, and recover from errors”.

I’m surprised to find this classic usability problem in software such as Outlook managed by a large organization with thousands of users. This must make the support phones glowing (update: after talking to our IT support department it actually does increase support requests.)

Using design cards to facilitate redesign: initial findings

When developing new design concepts and redesigning existing ones the idea of using design cards to facilitate the process and creativity has been explored in several different fields. The idea of using design cards as part of design workshops has been explored when developing new design concepts such as game designs [7], “playful experiences” [6], and tangible interactions [5]. For example, design cards have been used to “…probing and provoking users’ past experiences and current assumptions.” [3]. Several design cards have been developed based on theoretical frameworks and used to rephrase abstract frameworks into something more operational. Through this transformation the theory can be more tangible and applicable by making cards with keywords, pictures, and questions [5]. It is out of the scope of this blog post to include a detailed review of related literature. Instead, I will briefly outline recent HCI-related research about design cards.

In recent research design cards have mainly been evaluated during initial design phases, and a few during other phases such as evaluation of designs, and redesign. Past literature has pointed out the several general strengths and advantages of design cards. On an abstract level design cards can be considered a design material useable in a collaborative setting when making designs [4]. Wölfel and Merritt (2013) [8] conducted a review of 18 different forms of design cards or card methods used as part of a design activity, and highlight that design cards have been reported to:

  • Supporting design dialogues.
  • Cards can act as a common reference among participants.
  • Cards are something specific and concrete to talk about.
  • Making the design process visible and less abstract.
  • Facilitating a design process, for example, by providing structure in the process.
  • Physical tokens in the form of cards are easy to include, use, and manipulate.

They divided the purpose and scope of the card systems into three different categories:

  • General (8 card methods) (open-ended inspiration.)
  • Participatory design (7 card methods) (engage designers and users in the process.)
  • Context-specific/agenda driven (3 card methods) (focused on a particular context or design agenda.)

They divided the methodology of the design cards into three categories:

  • No methodology (3 card methods.)
  • Suggestion for use (7 card methods.)
  • Specific instructions (5 card methods.)

In usability engineering, a challenge for developers is how to correct usability problems, especially non-trivial problems. Running redesign workshops with developers and designers actively collaborating has been proposed as one method for exploring redesign opportunities [1, 2]. In a recent study, we decided to explorer if including design cards in redesign workshops would improve the proposed redesigns and support the process of fixing usability problems.

In summery we asked groups consisting of two – three students, following a class about designing and evaluating user interfaces, to make redesign proposals of a given web shop. We provided the groups a list of known usability problems. The groups were divided into four group clusters. The groups in three of the clusters were provided a design card system, a different system for each cluster. The groups of one cluster were acting as control groups and did not receive any design cards. During the redesign exercise, we observed a subset of groups. Afterward, we had the students filling out a survey about their impression of the quality of the redesign, and usefulness of the design cards. Finally, we interviewed a few groups. In addition, six evaluators, three academic researchers, and three practitioners involved in the development of the web shop assessed the quality of the redesigns.

Our initial findings indicate that design cards did not, at least not how we used them in this study, have any major positive effect on the quality of the redesigns. When comparing the quality assessment of the redesigns, we did not find any significant differences in comparison to the control group. The students did not find the design cards particular useful. However, some mentioned that the cards did provide some initial ideas and inspiration. These initial findings might be surprising when looking at the positive results reported from previous studies [5-7]. One major difference in this study is that design cards were included in a redesign workshop, and not during an initial design development or ideation workshop. The students were provided both an existing design, and a list of known usability problems. These seemed to be the main drivers and basis for the redesign proposals.


  1. Bornoe, N., Billestrup, J., Andersen, J. L., Stage, J., & Bruun, A. (2014). Redesign workshop: involving software developers actively in usability engineering. In Proceedings of the 8th Nordic Conference on Human-Computer Interaction: Fun, Fast, Foundational (pp. 1113-1118). ACM. DOI: 10.1145/2639189.2670288
  2. Bruun, A., Jensen, J. J., Skov, M. B., & Stage, J. (2014). Active Collaborative Learning: Supporting Software Developers in Creating Redesign Proposals. In Human-Centered Software Engineering (pp. 1-18). Springer Berlin Heidelberg. DOI: 10.1007/978-3-662-44811-3_1
  3. Bødker, S., Mathiasen, N., & Petersen, M. G. (2012). Modeling is not the answer!: designing for usable security. interactions, 19(5), 54-57. DOI: 10.1145/2334184.2334197
  4. Halskov, K., & Dalsgård, P. (2006, June). Inspiration card workshops. In Proceedings of the 6th conference on Designing Interactive systems (pp. 2-11). ACM. DOI: 10.1145/1142405.1142409
  5. Hornecker, E. (2010). Creative idea exploration within the structure of a guiding framework: the card brainstorming game. In Proceedings of the fourth international conference on Tangible, embedded, and embodied interaction (pp. 101-108). ACM. DOI: 10.1145/1709886.1709905
  6. Lucero, A., & Arrasvuori, J. (2012). The PLEX Cards and its techniques as sources of inspiration when designing for playfulness. International Journal of Arts and Technology, 6(1), 22-43. DOI: 10.1504/IJART.2013.050688
  7. Mueller, F., Gibbs, M. R., Vetere, F., & Edge, D. (2014). Supporting the creative game design process with exertion cards. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (pp. 2211-2220). ACM. DOI: 10.1145/2556288.2557272
  8. Wölfel, C., & Merritt, T. (2013). Method card design dimensions: a survey of card-based design tools. In Human-Computer Interaction–INTERACT 2013 (pp. 479-486). Springer Berlin Heidelberg. DOI: 10.1007/978-3-642-40483-2_34

Free Wireframe GUI Toolkit From GUIToolkits

GUIToolkits is currently offering a free wireframe GUI Toolkit for Keynote, PowerPoint, Visio, OmniGraffle, Illustrator and Fireworks. The package contains hundreds of wireframe GUI components for iPhone, iPad, Android, Blackberry, Windows Phone, Facebook, The Web, Mac OS X, and Windows.

To download the toolkit a predefined Twitter message must be posted from your Twitter account (what they call “Free with a tweet”). For more info and to download the package go to their website: http://guitoolkits.com/wireframe-gui-toolkit/

Free Keynote Mockup Templates from Keynotopia

Keynotopia is currently offering free Keynote mockup templates for mobile, web and desktop apps. In return a predefined Twitter message (called “free with a tweet”) must be posted so for those that have yet to join in on the Twitter vibe it might be possible to alley with a friend or colleague having one of those Twitter accounts.

The templates can be downloaded here.

This quick guide provides some startup advice when first playing with the Keynotopia templates and Keynote.

I recently attended the conference MobileHCI 2012 in San Francisco and was made aware of this offer during the tutorial “Designing and deploying mobile user studies in the wild: a practical guide” organized by Karen Church.