Why Card Sorting Doesn't Work: Create and Test Taxonomies Instead

Let me start out by saying that "doesn't work" may be too strong of a phrase here. There are valuable things that can be learned from card sorting. However, I would suggest that you will be far better off having a information architecture expert create a taxonomy or site map for you and then use that to test users. This " Create and Test" method is especially useful for large, deep taxonomies.

The reasons to have a knowledgeable expert create a taxonomy first, and then test it are the following:
1. You are not asking people who know little to nothing about usability or logically organizing information to create your site structure for you.
2. What is card sorting really? It is having a number of people give suggestions of possible groupings and possible labeling of those groupings. I would argue that you can get the same kind of feedback while users test your taxonomy.
3. Usability testing time required to test a taxonomy is far shorter than the time to do a card sorting exercise. During testing you will be watching users, measuring their success rates at finding the information they are looking for, and hearing and seeing feedback as to what issues they are having with naming and/or organization. What do you have at the end of card sorting? An oftentimes, very subjective set of data, the statistical significance of which is questionable if you are not testing hundreds of users. It is possible to test hundreds of users at once using online tests. I would argue that the online tests are far more valid that small sample sizes of 10 and under.

Some suggestions on the Create and Test method are:
1. Try to limit each category within your taxonomy to having 7-10 items if possible.
2. Create a click-through prototype of the taxonomy. This is not a full-blown site, but simply the categories listed as blue links on a page and when you click one, it displays the next level of categories. This allows you to see users interacting and allows them to talk through what they are thinking. It also helps you see if they were successful on the first attempt, second attempt or not at all.
3. Test a range of items or questions. Since you are doing a less time-consuming click-through test of the taxonomy itself, you can have the user find 10-15 things that would be contained within the taxonomy. If it is a product taxonomy, for instance, test some of your best known products, some obscure products, and other products where you are a unsure as to whether or not the taxonomy is solid. If you are testing a support site, maybe you give the user a list of 10-15 FAQs that most people seeking support ask and have them click through to where they think the answers are. Be creative here.

In the end, if you use the Create and Test approach, you will:
1. Get valuable feedback on naming and trouble points in your taxonomy.
2. Have strong measurable data about user success rates.
3. Have completed the test (and analysis) in a far shorter timeframe than with card sorting.

Overall, card sorting is effective in finding synonymous names for categories and identifying user agreement on category names that are strongly indicative of the users' mental models. However, results are often not directly applicable because users are not as familiar with all of the content on the site, users don't think of the confusing synonyms that could be a result of the words they choose, their categories sometimes mix unlike things in a confusing fashion, among other issues. In contrast, the Create and Test approach gets you much of the same feedback, without the longer timeframe, and gives you measurable success results rather than possibly applicable yet conflicting and not well though out suggestions.

If you liked this article: Maybe you'd like to hire us at:

The Usability Review

Subscribe to This Blog (RSS)

The Myth of Limiting the Number of Clicks

Time after time, you hear clients and even designers say we need to get the user to their destination in 2 or 3 clicks. The end result of this point of view is usually a home page with so much stuff on it that you can't find anything you need OR a navigation scheme that has 50 items top level category. Most of the time, neither of these approaches is efficient or effective.

So, what is the right approach? Well, my first instinct is to refer to an article written on Jared Spool's site www.uie.com. Jared Spool is one of the most well-known usability and user interface experts around. The name of the article is The Truth About Download Time.

What does download time have to do with number of clicks you ask? The article is all about user perception versus reality. The basic premise of the article is that user's perception of the speed of a site were strongly correlated with their successful task completion on the site and not at all correlated with the actual speed it took to load pages on the site. In fact, the "fastest" sites according to the test participants actually had the slowest download speeds. Now the article was written in 2001 when download speed was very important as bandwidth constraints freqently made us wait for pages to load.

I would extrapolate these results to a user navigating through a deep site taxonomy or through a series of tasks. When designers and clients force too much stuff up to the home page or put too many links in a category, they are actually making it MORE difficult for users to complete their tasks. When site maps and taxonomies are designed such that there are fewer options that are more clearly identifiable, task completion will be improved. I usually try to keep categories to 7-10 items.  A good reference point is the classic journal article by Miller in 1956: The Magical Number Seven, Plus or Minus Two.

There are those who would argue that the use of Miller's paper with respect to navigation is a misinterpretation and that far more links (and presumably fewer clicks) will work just fine.  But what has often been absent from discussions about limiting the number of categories in a taxonomy is users employing the process of elimination during navigation.  Process of elimination of the incorrect options is far easier with a small set of options.  In fact, I recently observed a usability test session where the main menu had only 5 options.  This menu had a lot of usability problems to begin with: 1) the options were hidden in a rollover by default; 2) the client's design firm had used icons instead of labels (the latter were only visible on rollover); 3) the nomenclature was not very clear for each option; and 4) the placement of the rollover menu was in the upper right of the screen (one of the last places users typically look).  Yet, despite all of these negative usability factors, the menu was largely successful.  I attribute that to the fact that there were very few options and users were able to single out choices by process of elimination (even though the couldn't read the labels on the icons and had to keep the choices in memory).  The argument to use much larger numbers of links also largely ignores the time time it takes to read those options and the memory required to recall the possible options that could be the right one.   Also, users often click the first option they find that sounds right.  In many cases, the greater the number of links displayed, the greater the likelihood that there is some overlap within those links which can lead to users exploring the wrong path first and wasting even more time.

There are several caveats to the above situation, though:
1. The number of clicks should be reasonable. If you are forcing people through 15 levels of a taxonomy, they better be doing their taxes or something complex that they have a strong motivation to complete.  I am not a proponent of making navigation paths any longer than necessary.  For instance, on the Lenovo.com computer site, their new customization screen for purchasing a laptop has a carousel with 39 options.  You literally have to click the next button 39 times see each option.  And this is only for the "System Compoents" tab.  There are more options for the other 3 tabs.  And while you can skip around, who knows what you are missing.  They should take heed of the Dell example which has more options per page because who knows what you are missing in terms of configuration if you aren't clicking through the 39 options.  It could be something important to you.

2. If the list of links are things that most people are familiar with, you can get away with more links. US states for a US-centric site, or city names within a state are examples. For groups of things that are more subjective (like product categories), it is best to stick to ~7-10 links.
3. You must do a good job of creating descriptive, easy-to-undestand, mutually exclusive categories. "Descriptive" does not mean make the category name 8 words long. It should be as short as possible, yet meaningful. "Mutually exclusive" means that a user should not be confused between two categories when looking for an item within the taxonomy. There are cases where items will appear in two categories (or more) within a taxonomy, but those cases should be minimal.

While The Truth About Download Time is not specifically about building taxonomies and site architectures, it IS all about user perception and the correlation of that perception with site performance. I think it is a great analogy to the number of clicks issue. So, the next time you hear someone say "Everything on the site should only be two clicks from the home page", you will have some sound reasoning about why that is not always the case.

If you liked this article: Maybe you'd like to hire us at:

The Usability Review

Subscribe to This Blog (RSS)