Highlights of “Usability Engineering in the Wild: How Do Practitioners Integrate Usability Engineering in Software Development?”

Back in 2014, I published this study together with a co-author. To make it more accessible, I have written the following summary highlighting the essential findings and implications of the study.

In the paper, we presented an explorative study about how usability engineering is conducted and understood in practice. Our research question was: “How do practitioners perceive and integrate usability engineering in software development?”

Through 12 qualitative semi-structured interviews with participants from 12 different software development organizations, we wanted to get an understanding and extending existing research about how usability is conducted and integrated into practical software development. By extending existing research, the intention is to support the ongoing process of refining and developing new methods and approaches for supporting practical usability engineering.


We found that:

  1. By practitioners, usability is used as an ambiguous term often mixed with other concepts, especially user experience (UX). Usability was by some described as a good feeling when designing. What makes usability ambiguous is that the practitioners rarely set and defined the objectives and purposes of usability. Regarding UX and usability, UX was linked to the overall design. Usability was linked to individual functionality and narrow design aspects, for example, the design of an interface object such as a button. Missing usability competencies in some organizations made it challenging to customize evaluations and decide when and what to evaluate.
  2. Usability is not easily explicable and was found difficult to justify and make credible. Initial designs are based on domain knowledge, and user input is used for adjustments, feedback, and visions. Visions for usability are set, but measurable goals are not.
  3. Informal and ad-hoc evaluations are common because developers need ongoing feedback, that cheaply, easily, and fast can be implemented. The practitioners did not follow a clear plan, or organized evaluation steps. The organizations had built up a body of principles and standards for individual software types – local de facto standards. These local de facto standards are developed because it’s easy, fast and useful, to keep following the same principles used in previous projects.
  4. Reporting of usability problems is done in a wide variety of ways, with the common denominator being lightweight. This can take the form of simple notes, presentations, workshops and user stories. This fits into the light hierarchies, the fast pace of projects, and makes it possible to act upon identified problems during the development process. Problem-solving is done through discussion with colleagues, and by getting inspiration from other sources such as blog-posts and online forums.
  5. All organizations followed an agile development approach. The advantage is that this approach is dynamic and allows changes during development. The disadvantage is that the focus on sprint completion can become the main factor leaving out proper or any usability engineering.

In conclusion, we found that local de facto standards highly drive practical usability engineering, and the practitioners conduct usability engineering informal and lightweight. This includes setting goals, including users, evaluating, and reporting usability problems. A majority of the development projects are completed within short time spans, and the participants need ongoing feedback that can be acted upon during development. Especially these factors are central, as such fast-paced environments do not leave much time for planned and organized usability engineering. Especially the local de facto standards were seen as an efficient approach towards creating robust designs fast. Agile development is a double-edged sword. It allows dynamics between developers and designers, but the focus on sprint completion can take away attention from usability engineering. Missing and limited usability competencies also have an impact. Some examples are ineffective inclusion of users, as well as challenges planning and customizing evaluations to specific conditions.


Some implications of the study are:

  1. Developing scales for prioritizing different user needs could be used for structuring, and facilitating discussions and decision-making when prioritizing usability problems and which product features to include. We support the idea of such straightforward scales. They are easy to understand, learn, and can be used by teams consisting of people, with a wide variety of backgrounds and roles.
  2. Developing approaches and facilitation for workshops seems relevant. Workshops are already popular because it’s a dynamic approach, but practitioners have problems using workshop sessions optimal. This is especially true when including users, because the practitioners experienced problems both running the right activities, and directing the users towards providing relevant feedback. We see a potential for optimizing an already popular method, and using existing guide lines when conducting workshops.
  3. Local de facto standards were used as local interaction design patterns. Future methods for documenting, sharing, and communicating local de facto standards within organizations could support building up a corpus of local design patterns.

Bornoe, N., & Stage, J. (2014). Usability Engineering in the Wild: How Do Practitioners Integrate Usability Engineering in Software Development?. In International Conference on Human-Centred Software Engineering (pp. 199-216). Springer Berlin Heidelberg. DOI: 10.1007/978-3-662-44811-3_12

Active Involvement of Software Developers in Usability Engineering: Two Small-Scale Case Studies

Together with co-author Jan Stage we got a short paper accepted at the conference INTERACT 2017.


The essence of usability evaluations is to produce feedback that supports the downstream utility so the interaction design can be improved and problems can be fixed. In practice, software development organizations experience several obstacles for conducting usability engineering. One suggested approach is to train and involve developers in all phases of usability activities from evaluations, to problem reporting, and making redesign proposals. Only limited work has previously investigated the impact of actively involving developers in usability engineering. In this paper, we present two small-scale case studies in which we investigate the developers’ experience of conducting usability evaluations and participating in a redesign workshop. In both case studies developers actively engaged in both activities. Per the developers, this approach supported problem understanding, severity ratings, and problem fixing. At the organizational level, we found that the attitude towards and understanding of the role of usability engineering improved.

Nis Bornoe and Jan Stage. 2017. Active Involvement of Software Developers in Usability Engineering: Two Small-Scale Case Studies. Human-Computer Interaction – INTERACT 2017. Lecture Notes in Computer Science. Springer.

A few words about selling UX

A while ago I attended a seminar for practitioners about the differences and similarities between usability and UX and not least the problems of understanding, separating, and combining the two into something specific. During this seminar, a problem discussed among the practitioners was how to sell UX.

There are a number of challenges when presenting and selling UX to clients. UX is a “fuzzy” term not easily explicable when it comes to what is delivered, what UX looks or feels like, and how the value of UX is made quantifiable. For example, a chief product officer of a small software development organization told me once: “…how does one get usability included into business cases so that they are credible higher up in the system?”

On top, several different UX definitions exist, and UX is easily confused with or used as a synonym for ‘usability’ or “user interface.”

The following advice was mentioned at the seminar when talking to clients:

  • Use facts about UX.
  • Include UX people during sales meetings.
  • Include UX explicit into business cases.
  • Outline the intended UX design process.
  • Show examples of how UX methods can form a product.
  • Provide examples of how existing knowledge about UX design can be used in new projects.
  • Compare with UX strategies taken by competitors.
  • Show something “beautiful” early in the process.
  • Get allies in the organization.

Joining the ACM IUI 2017 program committee

I have joined the ACM IUI 2017 (The 22th International Conference on Intelligent User Interfaces) program committee. The program committee of the IUI conference is slightly different organized in comparison to other ACM conferences. Usually, the members of a program committee will assign each submission a number of external reviewers and write a meta-review. At IUI the program committee is divided into two subgroups, a senior program committee, and a (junior) program committee. Each submission will be assigned three reviewers, two (junior) PC members, and one senior PC member. What diversifies the (junior) program committee from being external reviewers is that we are more linked to the process and participates in activities such as paper bidding and discussion. As a Ph.D. student, this is an excellent opportunity to get a gentle introduction to the work and assignments handled by a conference program committee.

The 22th International Conference on Intelligent User Interfaces (IUI ’17) will take place March 13-16, 2017, Limassol, Cyprus and the scope of the conference is described as:

At ACM IUI, we focus on the interaction between machine intelligence and human intelligence. While other conferences focus on one side or the other, we address the complex interaction between the two. We welcome research that explores how to make the interaction between computers and people smarter, which may leverage solutions from data mining, knowledge representation, novel interaction paradigms, and emerging technologies. We strongly encourage submissions that discuss research from both HCI and AI simultaneously, but also welcome works that focus more on one side or the other.

Facilitating Redesign with Design Cards: Experiences with Novice Designers

Together with coauthors Anders Bruun and Jan Stage we got a paper accepted at OzCHI 2016, the Annual Conference of the Australian Computer-Human Interaction Special Interest Group. OzCHI 2016 is taking place in Launceston, Tasmania from November 29th – December 2nd 2016.


While effort has been put into developing and evaluating usability evaluation methods less attention has been paid to shifting usability feedback into improved designs. We report from a study with 44 novice designers creating redesign suggestions. Some were provided with domain specific design cards to facilitate the redesign process. Design cards are physical cards used to structure a collaborative process, and providing design cues such as keywords and questions. Afterward, three developers assessed the quality of the suggestions. We found that the cards diversified the range of system aspects that novices considered, supported ideation, and kept the discussion going. However, the cards did not compensate the limited design experience, and the participants had challenges understanding the value of the cards, and implement them in the process. Having developers assessing the subjective quality of the suggestions turned out to be challenging due to low inter-rater reliability.

Nis Bornoe, Anders Bruun and Jan Stage. 2016. Facilitating Redesign with Design Cards: Experiences with Novice Designers. In Proceedings of the 28th Annual Conference of the Australian Computer-Human Interaction Special Interest Group: Connceted Futures (OZCHI ’16). ACM, New York, NY, USA.

Review submitted to OzCHI 2016

OzCHI 2016Submitted a review to the OzCHI 2016 conference with the formal name “The Annual Meeting of the Australian Special Interest Group for Computer Human Interaction.” Here at our group, The Research Centre for Socio+Interactive Design at Aalborg University, we have for several years participated in and supported this event. This is the first time I am reviewing for OzCHI and the second time submitting a full paper. This years theme is “Connected futures.”

OzCHI 2016 will be held in Launceston, Tasmania, November 29th through to December 2nd 2016.

Work in progress: case study about the impact of usability work

TonePrint Editor old version
Figure 1: Screenshot of the original version of the TonePrint Editor.

Currently, I’m spending a part of my summer writing a small-scale case study about the impact of usability work. Back in 2014, I was part of a team arranging a redesign workshop [1] for a development team at the company TC Electronic, now known as Music Group Innovation. They wanted to evaluate and improve an application called TonePrint Editor (see figure 1.) The essence of the workshop was to facilitate the development team fixing a list of previously identified usability problems. I recently returned to Music Group Innovation to conduct a small-scale case study about the redesign process of the TonePrint Editor. I wanted to do a follow-up to explore the process around making design changes to the application, and what had happened with the identified usability problems.

Project introduction

The use of usability evaluation methods is a widely accepted and used approach during iterative software development. One form of usability evaluations is the formative approach often conducted with a think aloud method. Formative usability evaluations are used to get feedback about users’ behavior when using an application and to get the users qualitative feedback about the concepts and designs used. The feedback reveals insights about how the users perceive, understand and interact with a system, insights that can be used to improve and develop an application. A lot of research has focused on both developing different usability evaluation methods, and evaluating the effectivity of existing methods [6]. Less attention has been paid to the relationship between the output from evaluations and improvements made to a given application [7], such research is complicated, time consuming and resource demanding [5]. Dennis Wixon boils this down to that: “…problems should be fixed and not just found” [8]. Since the point of making usability evaluations is to end with an improved interaction experience it is relevant to investigate what happens with identified usability problems, how the developers use the feedback from evaluations, and what they perceive as useful about the insights.

TonePrint App new version
Figure 2: Screenshot of the new version of the TonePrint App.

In the redesign workshop running two years ago [1], the main focus was to have the developers engaging in an innovative redesign suggestion process through active involvement. As part of the workshop, we included a short lecture about basic principles of interaction design. The intention with the lecture was to have the developers think about UI design in more broad terms and get inspiration for coming up with redesign proposals. Before the workshop, the developers had conducted a formative usability evaluation and compiled a usability problem list consisting of 19 problems. The outcome of the workshop was ideas for changing the flow of the main screen and minor changes to the interface. After participating in the redesign workshop, the development team has continued the redesign process, and have made several changes to the application.

During my revisit at Music Group Innovation I conducted a semi-structured interview with two members of the development team, a product manager, and a developer. We spent a couple of hours talking about the changes made to the application, the redesign process, and impact on the organization after engaging in user-centered design.

Preliminary insights

I’m still in the process of analyzing the interview in detail, but I will here outline a couple of interesting insights.

Prioritizing usability problems

When we talked about the list of the 19 identified usability problems, the first step after the evaluation was to prioritize the problems to decided on which ones to fix with. During the compilation of the list, a classic severity rating in the form: minor, moderate and severe was given. Additionally, two other ratings were added. The interviewed programmer would give a complexity rating (1-8). This rating is the estimated technical complexity of fixing the problem. The interviewed project manager would give a business value rating (1-8). This rating is the estimated importance related to the functionality of the application. Both are also related to the resource requirement estimation. The three ratings were then used to decide which problems to prioritize. Through this prioritizing process, the development team was able to understand and analyze the problems from more angles. This also served to make the fixing of usability problems more goal oriented. Initially, they prioritized seven problems. In the end, the team made fixes for 14 problems.

Getting problems confirmed and extended

Consistent with conclusions from another study [3], including the development team in the formative evaluation provided the developers with a more specific understanding of the usability problems. This understanding is more detailed that simply reading a usability problem report [3]. They were already aware of, or had ideas about possible usability problems, but in line with findings from another study [4], they found it useful to get confirmation or disconfirmation. What is more interesting is that their fuzzy ideas about problems were concretized and extended. For example, the flow of operations on a particular screen was not in line with the flow of operations found logic by the users, and how the users wanted to interact with the application. Getting insights into this design flaw was by the project manager characterized as a big eye opener. This was not identified as a specific usability problem but was by the development team identified as a more generic design problem leading to usability problems. Interestingly, the most significant redesign considerations sparked around feedback mainly gained through the involvement in the evaluation and redesign workshop, and less on the specific usability problems.

Design changes

During the redesign process, a couple of significant design changes was decided.

As mentioned above the flow of operations and order of options on a screen was found to be problematic. While this was not a specific usability problem, the development team decided to work on this problem during the redesign workshop. During the initial design of the application, they wanted to make the application ‘flashy’ (see figure 1.) During the workshop, they instead created redesign proposals based on the insights from the evaluation and the basic interaction design principles introduced during the short lecture. Afterward, they further evolved these proposals into a specific deign (see figure 2.) Similar findings have been reported by previously work [2].

At the time of the usability evaluation and redesign workshop, two applications existed the TonePrint Editor and the TonePrint App. The two applications have been merged into one application that runs on all major devices to make it easier for the users.

A couple of take-aways

The process of prioritizing the identified usability problems made the fixing of usability problems more goal oriented. For example, instead of simply adding problems to the backlog, there were some clear thoughts behind what problems to prioritize. This included considering severity, complexity, and business value ratings, as well as the estimated resources needed to fix a given problem.

Having the development team actively involved in both the formative usability evaluation and the redesign workshop provided insights about the current application design that would not have been gained if both the evaluation and creation of redesign proposals had been outsourced. Regarding the identified usability problems the developers got a more specific understanding and extensive understanding besides what was reported in the usability problem list. Insights about the current state of the usability of an application do not merely come from reading a report based on a formative usability evaluation.

The redesign workshop provided a frame for working with the insights. This sparked new ideas and a set of redesign proposals that were later matured and evolved into implementable designs. The final design shows that the development team was able to combine insights from the evaluation with basic principles of interaction design.

The short conclusion is that usability work makes sense and have an impact as long as the understanding of usability work goes beyond purely conducting usability evaluations.

Upcoming work

During the upcoming weeks, I will do a more comprehensive analysis of the interview and investigate the above themes in more detail. Along with my co-authors, we will be submitting this case study to the Industry Experiences track at the NordiCHI 2016 conference, so crossing fingers that the reviewers will find the paper interesting enough for a presentation.


  1. Bornoe, N., Billestrup, J., Andersen, J. L., Stage, J., & Bruun, A. (2014, October). Redesign workshop: involving software developers actively in usability engineering. In Proceedings of the 8th Nordic Conference on Human-Computer Interaction: Fun, Fast, Foundational (pp. 1113-1118). ACM. DOI: 10.1145/2639189.2670288
  2. Bruun, A., Jensen, J. J., Skov, M. B., & Stage, J. (2014, September). Active Collaborative Learning: Supporting Software Developers in Creating Redesign Proposals. In International Conference on Human-Centred Software Engineering (pp. 1-18). Springer Berlin Heidelberg. DOI: 10.1007/978-3-662-44811-3_1
  3. Hoegh, R. T., Nielsen, C. M., Overgaard, M., Pedersen, M. B., & Stage, J. (2006). The impact of usability reports and user test observations on developers’ understanding of usability data: An exploratory study. International journal of human-computer interaction, 21(2), 173-196. DOI: 10.1207/s15327590ijhc2102_4
  4. Hornbæk, K., & Frøkjær, E. (2005, April). Comparing usability problems and redesign proposals as input to practical systems development. In Proceedings of the SIGCHI conference on Human factors in computing systems (pp. 391-400). ACM. DOI: 10.1145/1005261.1005274
  5. Law, E. L. C. (2006). Evaluating the downstream utility of user tests and examining the developer effect: A case study. International Journal of Human-Computer Interaction, 21(2), 147-172. DOI: 10.1207/s15327590ijhc2102_3
  6. Nørgaard, M., & Hornbæk, K. (2009). Exploring the value of usability feedback formats. Intl. Journal of Human–Computer Interaction, 25(1), 49-74. DOI: 10.1080/10447310802546708
  7. Uldall-Espersen, T., Frøkjær, E., & Hornbæk, K. (2008). Tracing impact in a usability improvement process. Interacting with Computers, 20(1), 48-63. DOI: 10.1016/j.intcom.2007.08.001
  8. Wixon, D. (2003). Evaluating usability methods: why the current literature fails the practitioner. interactions, 10(4), 28-34. DOI: 10.1145/838830.838870

Locked out of Windows XP

padlockandchain2Today I decided to find out if it would be possible to bring back life in my dated Dell Latitude D820 I got back in 2006. I haven’t used it for years, and Windows XP was last reinstalled back in 2008. Originally the plan was to replace XP with Windows 7 to figure out if this old guy still has a few years left. Of cause, I couldn’t remember either the user or administrator passwords and was locked out of Windows XP… Now what to do?

Several websites describe two simple methods to rest the password of a user. Both assumes that it’s possible to get access to the system with an account having administrator rights. The most simple method is to have someone with administrator rights resting the password. The other method is a bit more complex. The essence is to reboot in “safe mode with command prompt” and log in with the ‘hidden’ administrator account. With some simple commands, it’s then possible to reset the password of a user account. It turned out that the administrator account was also password protected, and I was unable to log in at all… While preparing for the big operation to remove the hard drive and secure a backup of as much data as possible before whipping the disk, I found Ophcrack. Ophcrack is an open source password cracker that can be used to crack LM and NTLM hashes. Fortunately, this tool works well for recovering Windows XP account passwords.

One option is to download ophcrack liveCD, an image, and burn it to a CD/DVD or USB stick. This process was easy, I downloaded the ISO image, burned it to a DVD and booted the Latitude laptop (after changing the boot order in the BIOS). The application more or less starts automatically, and less than a minute later I had recovered the passwords of the user and administrator accounts.

While Windows XP is a dated operating system first released back in 2001 I was quite surprised that it was possible to recover passwords this fast. Anyway, this tool saved me a lot of time and will make it a lot easier to take a system backup before upgrading to Windows 7.

Automatic generated birthday wishes

Happy birthdayA few days ago I celebrated my birthday. During the day, I received several birthday wishes through different media. This included several e-mails from both friends and different companies. Getting greetings is something I appreciate, but what is the point of getting auto generated greetings? This comes down to what the meaning of sending a greeting to someone is. Essentially the sender is showing friendliness and respect. However, I always had trouble appreciating greetings from a machine and found these kinds of greetings to be an illusion. Clearly no one is actually greeting you, it’s simply a scheduled job fully automated and executed by a machine like any other automated task. Two of the auto-generated messages I received were even identically since both companies happen to use the same e-mail provider. This strengthens the feeling of ‘industrialized’ greetings. The only plus is that sometimes these auto-generated birthday greetings will contain a coupon code or link to some special offer or gift even though this was not the case this year.

“Help users recognize, diagnose, and recover from errors”

Outlook password changeAt my organization, Aalborg University, it is a requirement to change the campus account password once every 90 days, a security imitative implemented last year. This is a widespread security policy used in many organizations, but also a policy whose significance has been questioned for more than a decade. I have very mixed feelings about this security measurement. A major advantage is of cause that leaked passwords will be unusable at some point (not considering the option that backdoors etc. can have been installed.) However, this approach is also associated with several obstacles from a user perspective. These include: coming up with a new easy to remember secure password and the hassle of changing password on all associated services requiring authentication. At Aalborg University, this applies to basically all IT-services such as access to WiFi, e-mail, printers, databases, etc. The new password has to be changed manually in several of these services.

Perhaps it’s because I’m a Mac user, but no notice is given about the upcoming expiring password. When I suddenly no longer can access different services I know it’s time to create a new password (after some frustration about trying to figuring out what the problem is.) Our passwords are changed through the Outlook Web App. To make sure that the password meets a certain security standard some requirements are in place. If the new password does not match this standard, the following error message is displayed:

“The password you entered doesn’t meet the minimum security requirements.”

Unfortunately, this error message does not tell anything about what the requirements are or how to get this information leaving the user in the unknown. This is a textbook example of a usability problem directly linkable to one of Jakob Nielsens’s ten heuristics:

“Help users recognize, diagnose, and recover from errors”.

I’m surprised to find this classic usability problem in software such as Outlook managed by a large organization with thousands of users. This must make the support phones glowing (update: after talking to our IT support department it actually does increase support requests.)