Posted by: Arild | March 17, 2010

When experts become novices

I have just finished the first rounds on user testing on an expert application. The application is migrated from an existing VB6 application to a new C# application.

To allow for realistic user testing we needed to recruit (somewhat) representative users. As this is an expert application where the users need a fair amount of business knowledge, our only source of test users are the users of the existing application.

This kind of testing, using expert users, gave us a few lessons, both expected and unexpected. I will share some of our lessons through my next few blog posts.

  1. Expert users of the old application, novice users of the new application

Expert users tend to focus on what they know from the old application. This involves how they are used to doing thing, including shortcuts and workarounds. The tasks they are used to perform in the old application will often take longer to perform in the new application; simply because they are well accustomed to the old application. This can make them negative towards the new application. However, with a little practice they will hopefully get just as efficient, or more, using the new application.

This is one reason to let some of the test users participate in several of your rounds of testing. As they get better at using the new application, some of their initial concerns may disappear, while others prevail and may prove more serious than you initially thought.

Just make sure you don’t confuse these expert-novice issues with real usability issues in your application. 😉



  1. This sounds a bit familiar 😉 How involved has the users been in the developement of the c# port? If they are given a chance to shape the new application, this might result in a more understanding of the new one, shifting the focus to the new functionality rather than focusing on how it used to be.

    If the users testing the c# application isn’t involved in the development, is this a choice made on purpose?

  2. None of our test participants have been directly involved in the development of the application. However, one of our participants is one of our “expert users” and participates in our demos at the end of each sprint. This participant has also been involved in user interface workshops and is an important business resource to the developers of the application.

    The other test participants have not had this kind of experience and involvement in the development of the application. We have purposely tried to get both users who have some experience with the application (or at least have seen it in use in demo sessions), and users who have never seen the new application before.

    The new application will be rolled out without any proper training of the users. Testing with users of the old application, who have not yet seen the new application, may be the best indication we can get on how the new application will be received.

    I hope this answered your questions. 🙂

  3. These questions might require a blogpost of its own…

    To get proper user feedback, how close to the real application must the test environment be?

    Furthermore, I’ve got an upcoming session with user feedback. In our case, we are still in the design phase, but would like to get feedback on our thoughts as early as possible . Do you think it is enough to give them “screenshots” that show them what we are planning to implement? The persons that are attending this sessions are all expert users of the old system.

  4. Very good question!

    The short answer is:
    Test as early as possible!
    Test before you even think it’s possible to test!
    The earlier you test the more you will save in later stages of the development.

    But as you said, I think this deserves it’s own post. 🙂

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s


%d bloggers like this: