Posted by: Arild | April 12, 2010

Usability testing in Scrum projects

Scrum projects are perfect for usability testing. Make usability testing an integrated part of every sprint and you will reap the benefits of usability testing throughout the project.

In Scrum projects it is usually recommended to do interface design one or two sprints before the implementation of the design. This allows you to do usability testing of the paper prototypes or mock-ups, and actually refine the design after feedback from the users before any time is spent on the implementation.

Usability testing guru Steve Krug recommends using an observation room where the developers, designers and anyone else who wants to can watch the usability test live. This can be any normal meeting room with internet access. Your first step should be to read his book “Rocket Surgery Made Easy” (Amazon). This gives a very good overview of how you can do low-budget usability testing. The setup is quite simple. All you need is…

  • screen sharing software (for testing of working software or websites)
  • or a camera/webcam (for testing paper prototypes),
  • a microphone (for the test room)
  • and speakers (for the observation room).

Set aside one morning at the end of each sprint to do usability testing. If you do the testing the day before the demo you should have a reasonably stable version, and you still have time to do a few minor fixes before the sprint demo. You can test with 2-3 users before lunch. This should be enough to uncover the most prominent usability issues.

To make things even easier you can simply use your backlog use-cases as test cases. Make sure you leave 10 minutes between sessions for you to reset your test and write up your notes, and for the observers to have a short break and grab a cup of coffee. When the test is done, you can do a short debriefing with the rest of the team to agree on the top 3 issues to fix before the next sprint.

Alternatively you can time-box the fixing of usability issues. That is, you fix as many issues as you can within a time frame of e.g. 1 day or set aside x function points in each sprint to fix usability issues. Make sure you have a specific task for this in the sprint backlog.

Whichever approach you choose, it is usually a good idea to start with the quick wins that will give you the most improvement from the least amount of effort. Often all it takes is a few tweaks, some clarification, re-wording or minor changes to the layout, to remove some of the most important issues. Other important problems are put into the product backlog and prioritized by the product owner. (At this point it helps that the product owner has viewed the test session from the observation room.)



  1. great post – it’s a great summary of what processes need to be in place and what benefits people can expect.

    We’ve been using discount usability testing (as described in Steve Krug’s book) at Rypple.

    We perform user testing every week (our iterations are week long). It’s helped us improve usability tremendously over the last couple of months.

    I also wrote about Krug’s book at

    • I would love to hear more about your experience with weekly user testing! 🙂
      How early in the project did you start? How much time do you spend every week on the testing? Do you test in the beginning or the end of the sprints?

      I had a read of some of your other book reviews at as well. I’m just getting started on “Freakonomics”. As you point out, quite a few interesting insights in this book. 🙂

      • We’ve been using for quick & inexpensive remote user testing for over a year now. As you can imagine, there are some drawbacks. Most significant one is that your test scenarios cannot get too complex because the more complex they are the greater the likelihood the tester will deviate or misunderstand something.

        To supplement that, we always tried to get on the phone with our real users and share screen shots or even go through new changes together (Skype with screen sharing is an inexpensive option).

        We also tried to run some more formal user testing by going directly to where our users work (Rypple is a workplace tool). But, as you can imagine, this is expensive and time-consuming.

        Then finally in January we settled on a great compromise that Steve Krug outlines in his book. Discount usability testing lets us get plenty of insight every week, it’s personal, fast and cheap. (I recently put up an add on Craigslist and got over 20 interested and qualified users within a couple of days – we pay $30 for about 50 minutes of testing.) And we still continue to use for simple scenarios or when we need to test something immediately.

        We usually run user testing once a week with only one person at a time. We do this over lunch time and after the testing session is over we quickly go over major takeaways and agree on what we can implement/change by the end of the day (anything bigger/more complex goes into our backlog). We tried it with 2 people at a time (back to back) but that would take an extra hour and would not get us that much benefit. We’re careful about screening our users as well to ensure maximum benefit from that hour we spend watching them complete tasks. Sometimes, if the team is working on multiple themes during the week we even have more than one user testing session per week. The key is to keep them lean so that everyone can move fast.

        Let me know what you think about Freakonomics after you’re done reading it.

  2. HI.

    For screeen sharing software, Morae is great. It captures the screen, the user’s face and voice, and allows observers to type in comments. That’s all synchronized together, and make analysis really easy.

    It’s only for in-person testing, not remote.

    — Hal Shubin

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s


%d bloggers like this: