IA Summit 2009 – Dave Cooksey – Taxonomy Validation

About dave:  saturdave ux consulting

Testing and taxonomy – why testing is important 

Taxonomy is an “expert” activity that comes from a priveliged view. Taxonomy is more closely aligned with business/technology goals than user goals.

The facts about taxonomies and metadata

• relationships are implied (as opposed to an ontology which defines relationships).  Users must interpret the relationships.
• A user’s individual experience informs their interpretation. Their experiennce very much differs from ours. This is why we need to test.

Why taxonomy validation is necessary

• justifies the cost of creation
• helps resolve organizational conflict
• vets ideas with real users
• reassures the business that the project focus is on success

Taxonomy validation techniques:

1. delphi card sorting – allows for direct interaction with participants (can ask follow ups and ask people what they mean, to describe their experience, justify their choices)
2. remote card sorting
3. usability testing - focuses on interaction (step-by-step vs. bird’s eye view)
4. search analysis - indirect (no follow-ups, no “what did you mean?”)

Delphi Card Sorting: About

• Qualitative method
• Small sample
• Based on the delphi method
• developed at UB
• Article by Celeste Lyn Paul, A Modified Delphi Approach to a New Card Sorting Methodology

Delphi Card Sorting: How it works

• Heirarchy is laid out in cards (seeded or not)
• Inform the user what the vocab is for 
• Users are instructed to turn over cards that they don’t think are necessary, move cards around, make new cards (add ones for missing concepts). 
• While they work, ask the participants about what they are doing (e.g. Why are you collapsing x and y categories? Why did you turn that one over? etc.)
• Users work in a cumulative fashion (don’t restart the test everytime, let them build on each other’s changes)
• The process generally runs for about 15-20 participants until the heirarchy stabilizes. 

Delphi Card Sorting: Tips

• If possible, get a video recording of the sorting. Video recording is essential
• Take pictures of the table before they start and after they finish (each participant)
• Remember, this exercise is to inform YOU - the end result is the final heirarchy, with notes. (which ones people had problems with, what flip flopped a lot).
• Considerations: What kinds of users are you testing: do they have domain expertise? do you have multiple user types?
• Plan for lots of space. Allow time between participants
• Create a starter sheet to make notes

Remote Card Sorting

• quantitative, large sample
• open or closed (closed – tests structure, open – creates categories, gathers many perspectives)

Remote Card Sorting: How to do it

• recruit via intercept or email (keep these separate, don’t combine the analysis/results)
• determine sample sized needed
• choose software
• focus each sort to 15-20. NOT TOO MANY ITEMS
• put items/categories on cards
• run study
• analyze the data

Online tools for card sorting:

• optimal sort
• mind canvas
• web sort
• cardsort
• socratic card sort
• web cat

Remote Card Sorting: Tips

• run a few qualitative to test, if possible
• pick the most important categories and items (stereotypical problems)
• read the comments
• deliverables will depend on the software used. 

Regular card sorting

• -open or closed: closed – write down your terms and let them organize them. open – let them come up with their terms and then organize them

Usability Testing

• piggy back on system testing
• task based – Where would you go to do this?

Usability Testing: Tips 

• have a clear goal
• keep it simple
• ask follow-ups, probe a little bit.
• illustrate how data is driving the experience.

Search Analysis – Tips

• Try to get as much info as possible about the data.