Usability Study

Home page of in early 2011

The Problem

The city had the same website for nearly a decade. That sort of UI consistency would have been nice if it wasn’t so difficult to use.

To be fair, fully not bad for 2000s web design

It was time to upgrade. Not just to get a more modern aesthetic, but also to offer more functionality for Seattle’s citizens and visitors. A little more than halfway through that effort, they wanted to make sure that they were on the right track, to demonstrate to stakeholders (like, say… the Mayor) that they were making forward progress. While contracting with the (now defunct) NGO Knowledge As Power, I designed and implemented a usability study that was the first of its kind for the city, run on a shoestring budget.

The Process

I drafted off Krug’s excellent methodology for quick-and-dirty-yet-effective usability studies. Two important changes had to be made, however. First, unlike with most UX projects, a city’s website can’t be targeted at a specific audience. They have to try to do the impossible and please everybody. So grabbing 5 random people probably wasn’t going to be robust enough. We decided to run two studies, one for Everyday Citizens, one for Business Owners. This made it easier to pick sets of tasks that we could be sure everyone in a given group would understand.

Second, as stated above, the city wanted to have an understanding of how the redesign improved over the old design, so I modified the methodology slightly to make it an A/B usability test (both within subjects and between subjects, just in case that kinda jargon gets your juices flowin’).

As always, running the study was the easy part. Putting the study together and making sure the recommendations were heard were much bigger challenges. We spent more than a week diving into the site to see what what was there (spoiler alert: A LOT) and coming to consensus with the city about what tasks we’d test. (And I literally spent a day hiking around the city, hanging posters on telephone poles to recruit participants from all walks of life and all corners of the city. Hey, the important work isn’t always glamorous.)

I knew from the beginning that, for the city to take the issues participants encountered seriously, decision makers needed to see what the users were experiencing first-hand. I leaned heavily on Krug’s authority as a usability guru to convince all the major players to spend a morning watching people use the site and debriefing with official reports to follow.

The Product

Feel free to flip through one of the reports I gave the city – technically that was the product of the work.

But what I’m most proud of was what happened in that observation room. The shared understanding of users’ struggles helped these high-powered stakeholders build new professional relationships and made it easier for them to accept recommendations for improvement, most of which they adopted. (We also had the full backing of the mayor’s office, and that authority didn’t hurt, either.) One stakeholder even confided that the observation sessions helped clear log-jams that had been intractable for years.

This combination of renewed collaborative purpose and my usability expertise resulted in a website that won a 2011 Best of the Web award for city government websites.