Creative Usability Test Methods—or My Brief Career as a Robot Voice

When you want to do a usability test, sometimes you have to step out of your comfort zone and get creative to get the job done. That’s just what happened to us.

We’re well practiced at usability testing at USAGov—in person, remote, hallway tests, first-click tests—all of these things we manage without blinking an eye. But this spring, we tried something new. Our office was planning to make some changes to our IVR script. You know what an IVR is even if you don’t know the term. IVR stands for interactive voice response, you hear it when you call an organization and hear a recorded voice that greets you and gives you the option of pressing different buttons for different topics (e.g.,“To schedule an appointment, press 1. For billing questions, press 2.”). That’s an IVR.

At USAGov, we wanted to test the recorded information that people hear when they call our toll free number, 1-844-USA-GOV1 (1-844-872-4681) during non-business hours. If you call after hours, you get a greeting, and then some options to listen to recorded information about our most common questions, like passports, housing, taxes, scams, and government benefits and grants.

The goal for our test was to identify major issues or areas that USAGov Contact Center can improve in the IVR messages. Specifically, we looked at:

• Can our callers get the information they need to progress on their task, or do they get stuck or frustrated?
• Are we missing vital information?
• Is there too much information?
• Do we have enough messages, and are they the right messages?
• Are the messages clear and easy to understand?

The test planning progressed like a normal usability test. We had goals, tasks, scenarios, and scripts. We had participants, observers, and a facilitator. Here’s where it gets tricky—how do we pull it all together? How does the participant call the number and interact with the IVR, while interacting with a facilitator, while the observers are able to hear everything that’s happening and record the session?

Since the test was scheduled during business hours and the IVR message is only available outside of business hours, using the live number wouldn’t work, and some advanced planning was in order. We had access to the test system, and that seemed like a viable option, so we did a dry run with staff playing the roles of remote observer, facilitator, and participant. It was tricky. Really tricky. We used GoToWebinar to record and bring everyone together, but when we added the element of the test IVR, it became really complicated.

There are a series of menus and passcodes that you have to navigate to get to the recordings, and we would have to send our participants through all the menus and passcodes at the start of each scenario. It was just getting too difficult. So we came up with plan B. Have someone on staff actually perform the IVR script live, in person. That lucky person was me.

But how can you make a live person sound like an automated voice recording? We wanted the experience to be as authentic as possible, so this question haunted us. Like any performance, the key was in rehearsal.

All those years of high school theater performances were about to pay off.

First I printed the script in a large font to make it easy for me to read. I used separate pieces of paper for the greeting, the closing, and each of the 6 menu items so I could spread the script out on my desk and easily jump from task to task. I read the whole script many times, and then I read the script out loud over and over until I was comfortable reading everything without making mistakes. That’s all good, but next I had to learn to read it like it was an automated recording—and they’re just not the same as a conversational voice. So I called the system (we called her IVR Lady) and we spent hours on the phone talking to each other. I had to learn her timing, her pacing, her tone, and her inflection. I became IVR Lady.

When test day rolled around, I was ready. I was able to go through all the scenarios doing my best IVR Lady impersonation. Still, I wondered if I was fooling anyone – it seemed impossible to me that they thought they were talking to a recording, how could they not know the difference? At the end of each session, we had a few minutes to ask our participant questions about their experience, so I asked, “While you were going through our scenarios, did you think that you were speaking to a recording or a live person?” Everyone we asked told us that they thought they were speaking to a recording, and they were surprised when they found out it was me. It worked!

Our creativity, determination, and willingness to take a risk paid off. We learned things about our existing script, and we have implemented changes to improve the experience of our callers.

This is what happens when new UX testing frontiers open up. I remember the first time we tested our website on mobile devices, we had to solve the problem of how to project the screen so our observers could watch what was going on. Our solution involved a desk lamp, a cheap webcam, and some binder clips. So don’t be afraid to come up with a low-tech hack to make a test work; the worst that could happen is that they tell you that you have a lousy robot voice.