Testers in a dangerous time: Rethinking experience testing during a global pandemic
In the face of COVID-19, Canada Aviation and Space Museum staff came up with a creative solution to move forward with public testing.
When the Canada Aviation and Space Museum closed its doors on March 14 due to the COVID-19 pandemic, its exhibition team was working full-tilt on Eyes on the Skies. Developed in partnership with NAV CANADA, this upcoming exhibition will explore our complex air navigation system, including the people and technologies that keep Canada’s skies safe.
As the exhibition’s interpretive planner, I was gearing up for Ontario’s March Break — a time of increased visitation. I was planning to interact with some of these museum visitors, to conduct proof-of-concept testing for a number of the exhibition’s interactive elements or games. Pairs of staff were planning to engage with volunteer participants. One tester would talk with the public, holding up instructional signs and pretending to “be the computer” as visitors made choices that drove the gameplay. The other tester would observe the interaction, noting perceived areas of enjoyment or confusion, and recording visitor feedback.
Well… that wasn’t going to happen.
My dining room after the shipment of testing supplies. So many cool widgets, so few people to share them with — or maybe…
The day before the shutdown was announced, staff were preparing for the likelihood of working from home. Our designers, a Montreal firm called TKNL, were just about to ship out the testing materials — elaborate mock-ups for three of our interactive experiences within the exhibition. These materials included mock buttons, slider inputs, gameboards, and white boards to test word puzzles. Given growing uncertainty, I asked to have the materials delivered to my house instead of to the museum. I didn’t know what I would do with them from home, but having them on-hand would give me more options.
In the days that followed the closure, I realized that we could only proceed if we moved the testing online. I took stock of our most pressing questions, and considered which could effectively be answered remotely. Elements of the original plan involved having staff manipulate the materials and observing visitors’ reactions. That was out. We could, however, test experiences where the participants themselves would manipulate all game elements. I was able to proceed in testing two of the three games — one where people use the phonetic alphabet to solve word puzzles, and another where they track the location of airplanes on a grid.
I had never tried something like this before, and I was a little nervous. I’m not always technically savvy, and new things tend to make me worry — which can be inconvenient during a global pandemic.
Adapting the process
The team at TKNL was very supportive. They adjusted the resources, creating two packages that participants could print and assemble in their own homes. I then proceeded to facilitate testing sessions, meeting with volunteers one-on-one by videoconference. At the start of each session, we would take a few minutes to review each of the “widgets” and set them up for gameplay. I would demonstrate the items we had intended for them to use, and discuss how they could pretend that they were using the same items at home.
Gavin Mullins sits at his family’s dining table testing interactive elements for an upcoming exhibition at the Canada Aviation and Space Museum.
I started with a few trusted colleagues who would bear with me as I got used to the rhythm of delivering testing through a new medium — fumbling with the cue cards, and taking too long to record answers as I balanced the roles of two testers. I then broadened the circle, working with 13 participants who ranged in age from seven-and-a-half to forty-something. Sessions were spread over a week, each lasting roughly 45 minutes.
As I could only see their faces, I asked participants to narrate their actions (“I’m sliding the airplane to letter P, and pressing ENTER”). I also asked participants to read instructions out loud, helping me to gauge readability. I asked the younger participants’ parent to stay within earshot — close enough help with logistics if needed, but far enough away not to influence their child’s interaction.
Reviewing the results
In the end, the results we obtained were useful. We were able to gauge interest and enjoyment, and identify patterns of confusion — spots where the instructions didn’t feel intuitive, or where participants weren’t sure “what to press next.” This information in particular is crucial.
It’s important to consider some of the drawbacks of remote testing. You learn a lot when you can directly observe a person’s actions. This is lost, to a degree, when we interact remotely. Also, the flow of remote testing tends to be more stilted, as one tester plays the role of both the interactor and the observer/recorder. Timelines are another factor — a group of four testers, working in pairs, could likely have gathered the required data in a couple of afternoons rather than a week.
I believe that our success hinged on looking critically at our questions and ensuring that our testing medium — the videoconference — was suited for the job. While this meant that we only tested two of the three scheduled experiences, we were able to move these two into the next phase of development.
As I write this article, many factors are still uncertain. We don’t know when we will be back at the office, when the design and fabrication teams will be back in their shops, or when we will once again be able to meet in person. We don’t know when our next phase of testing — prototype testing — will take place, and whether we will be able to test these items in-person or if we will still be looking for remote solutions.
While I’m looking forward to a return to normalcy, I believe that there’s a certain opportunity for professional learning when challenges force us to find creative solutions. Perhaps that’s a silver lining, or consolation prize, when our expectations are upended. This experience made me think critically about how best to get to the core of my questions, and which questions would have to wait (no matter how badly I wanted to move things forward). In uncertain times, it’s important to recognize the need for balance — we can only do what we can do. I am pleased that we were able to move forward with elements of our testing plan. This will ensure that we are able to offer our visitors the best possible experience — when we are finally able to welcome them through our museum doors once again.