Testers in a dangerous time: Rethinking experience testing during a global pandemic

5 m

May 13, 2020

Thinking fast

The day before the shutdown was announced, staff were preparing for the likelihood of working from home. Our designers, a Montreal firm called TKNL, were just about to ship out the testing materials — elaborate mock-ups for three of our interactive experiences within the exhibition. These materials included mock buttons, slider inputs, gameboards, and white boards to test word puzzles. Given growing uncertainty, I asked to have the materials delivered to my house instead of to the museum. I didn’t know what I would do with them from home, but having them on-hand would give me more options.

In the days that followed the closure, I realized that we could only proceed if we moved the testing online. I took stock of our most pressing questions, and considered which could effectively be answered remotely. Elements of the original plan involved having staff manipulate the materials and observing visitors’ reactions. That was out. We could, however, test experiences where the participants themselves would manipulate all game elements. I was able to proceed in testing two of the three games — one where people use the phonetic alphabet to solve word puzzles, and another where they track the location of airplanes on a grid.

I had never tried something like this before, and I was a little nervous. I’m not always technically savvy, and new things tend to make me worry — which can be inconvenient during a global pandemic.

Adapting the process

The team at TKNL was very supportive. They adjusted the resources, creating two packages that participants could print and assemble in their own homes. I then proceeded to facilitate testing sessions, meeting with volunteers one-on-one by videoconference. At the start of each session, we would take a few minutes to review each of the “widgets” and set them up for gameplay. I would demonstrate the items we had intended for them to use, and discuss how they could pretend that they were using the same items at home.

A boy, his back to the camera, is seating at a dining table in front of a laptop, taking part in an online testing session.

Photo Credit

Lindsey Ehman

Gavin Mullins sits at his family’s dining table testing interactive elements for an upcoming exhibition at the Canada Aviation and Space Museum.

I started with a few trusted colleagues who would bear with me as I got used to the rhythm of delivering testing through a new medium — fumbling with the cue cards, and taking too long to record answers as I balanced the roles of two testers. I then broadened the circle, working with 13 participants who ranged in age from seven-and-a-half to forty-something. Sessions were spread over a week, each lasting roughly 45 minutes.

As I could only see their faces, I asked participants to narrate their actions (“I’m sliding the airplane to letter P, and pressing ENTER”). I also asked participants to read instructions out loud, helping me to gauge readability. I asked the younger participants’ parent to stay within earshot — close enough help with logistics if needed, but far enough away not to influence their child’s interaction.

Reviewing the results

In the end, the results we obtained were useful. We were able to gauge interest and enjoyment, and identify patterns of confusion — spots where the instructions didn’t feel intuitive, or where participants weren’t sure “what to press next.” This information in particular is crucial.

It’s important to consider some of the drawbacks of remote testing. You learn a lot when you can directly observe a person’s actions. This is lost, to a degree, when we interact remotely. Also, the flow of remote testing tends to be more stilted, as one tester plays the role of both the interactor and the observer/recorder. Timelines are another factor — a group of four testers, working in pairs, could likely have gathered the required data in a couple of afternoons rather than a week.

I believe that our success hinged on looking critically at our questions and ensuring that our testing medium — the videoconference — was suited for the job. While this meant that we only tested two of the three scheduled experiences, we were able to move these two into the next phase of development.

Moving forward

As I write this article, many factors are still uncertain. We don’t know when we will be back at the office, when the design and fabrication teams will be back in their shops, or when we will once again be able to meet in person. We don’t know when our next phase of testing — prototype testing — will take place, and whether we will be able to test these items in-person or if we will still be looking for remote solutions.

While I’m looking forward to a return to normalcy, I believe that there’s a certain opportunity for professional learning when challenges force us to find creative solutions. Perhaps that’s a silver lining, or consolation prize, when our expectations are upended. This experience made me think critically about how best to get to the core of my questions, and which questions would have to wait (no matter how badly I wanted to move things forward). In uncertain times, it’s important to recognize the need for balance — we can only do what we can do. I am pleased that we were able to move forward with elements of our testing plan. This will ensure that we are able to offer our visitors the best possible experience — when we are finally able to welcome them through our museum doors once again.

Whispering Loudly: An Update about the “Quiet Updates”

A mid-sized biplane with distinctive yellow wings, photographed outdoors from the back right.

From cobwebs to Interwebs: Uncovering vintage photos of the museum’s aircraft

Five men, suited in their flight gear, walk towards the camera with a bomber airplane, facing forward and to the right, in the background.

The British Commonwealth Air Training Plan: Canada’s most important homefront contribution

A wide-angle view of a large space inside a museum, with white walls and a black floor. Colourful panels are visible, as well as a computer screen and a spacesuit inside a glass case.

Six steps for planning clear communication

“Worth Defending” – A brief look at the 1954 advertising campaign of a major Canadian aircraft manufacturing firm, Canadair Limited of Montréal, Québec, part 4

The advertisement “La liberté d’expression,” in English “Freedom of Speech,” published by Canadair Limited of Montréal, Québec, as part of its 1954 Worth Defending advertising campaign. Anon, “Canadair Limited.” La Presse, 13 April 1954, 25.

“Worth Defending” – A brief look at the 1954 advertising campaign of a major Canadian aircraft manufacturing firm, Canadair Limited of Montréal, Québec, part 3

“Freedom of Worship,” one of the advertisements issued by Canadair Limited of Montréal, Québec, as part of its 1954 Worth Defending advertising campaign. Anon., “Canadair Limited.” The Vancouver Province, 5 January 1954, 5.

“Worth Defending” – A brief look at the 1954 advertising campaign of a major Canadian aircraft manufacturing firm, Canadair Limited of Montréal, Québec, part 2

One of the advertisements issued by Canadair Limited of Montréal, Québec, as part of its 1954 Worth Defending advertising campaign. Anon., “Canadair Limited.” Toronto Daily Star, 6 October 1954, 4.

“Worth Defending” – A brief look at the 1954 advertising campaign of a major Canadian aircraft manufacturing firm, Canadair Limited of Montréal, Québec, part 1

A block of photographs showing some of the people involved in the bombing of beluga whales in the estuary and gulf of the St. Lawrence River. Anon., “La chasse aux marsouins [sic]. » Le Devoir, 15 August 1929, 6.

“The bomb that will kill the porpoise” – A shocking use of air power in interwar Québec: The bombing of the beluga whales of the St. Lawrence River, part 4

A typical pod of beluga whales, Hudson Bay, near Churchill, Manitoba, July 1999. Ansgar Walk, via Wikipedia.

“The bomb that will kill the porpoise” – A shocking use of air power in interwar Québec: The bombing of the beluga whales of the St. Lawrence River, part 3

The Franco Canadian physician / pilot Louis Cuisinier and one of the 45 or so kilogram (100 or so pounds) handmade aerial bombs with which he hoped to kill beluga whales. Anon., “La bombe qui tuera le marsouin.” La Presse, 5 August 1929, 11.

“The bomb that will kill the porpoise” – A shocking use of air power in interwar Québec: The bombing of the beluga whales of the St. Lawrence River, part 1

Several exhibition display cases, panels, and screens in a large, brightly-lit air hangar-style room with a high ceiling and tall, exposed metal walls.

Diane Pitre and Steven P. Deschamps’ LGBT Purge Stories Told at the Canada Aviation and Space Museum

An Institute for Aerospace Research staff member conducting an inspection on a Shuttle Remote Manipulator System, or Canadarm, around 2000. National Research Council Canada.

“It is only with the DAIS that one can see rightly; what is essential is invisible to the eye” – A respectful look at a great Canadian aerospace invention, the Diffracto DAIS non destructive testing equipment, and a bit more, part 2

$A technician conducting the first corrosion detection trial, on a Boeing Model 727 test airframe, using a Diffracto DAIS-500 non destructive testing equipment, 1992. National Research Council Canada.$