I’m coming towards the end of my latest set of blind playtests for Escape the Nightmare. The testing was divided into three waves and the feedback from each wave lead me to change things for the next wave – one of the things that changed every time were the questions that I asked. I thought that it might be interesting to share the questions that I’ve ended up asking the third and final wave and have some discussion of how our choice of questions influences the quality of the feedback that we receive as designers.

I like to start feedback forms with a little bit of preamble about what I’m looking for from feedback. Some players are happy to give negative feedback, but some won’t unless you make it clear that you value the opportunity to improve. Some players are happy to let you know about problems that they think are unsolvable (or at least that they don’t think that there are good solutions for), some won’t if you don’t specifically make that okay. Some are happy to say whatever comes into their head, others will self censor ideas without the reminder that the best ideas can start out as bad ones.

“Did you enjoy the game?” The feedback from this question is almost always pretty much worthless. If you’ve done any testing in person you’ll know that sometimes players will (by body language and disposition) have a rubbish time but say that they enjoyed it because people don’t like being mean (or want to be invited to the next test or have a feeling of obligation or whatever). However I think it’s important to ask it because if someone wants to say nice things that aren’t representative they’re going to find a way to do it, so throwing this out as a sacrificial lamb to enrich the answers to other questions is worth the space.

“How would you summarise the rules?” This is a context question that I find helps a lot in interpreting other responses. Without it it’s often impossible to tell the difference between “The testers did not play the game as intended, which lead to problem X, the rules need improving and the components could offer more clarity.” and “The game has problem X, which needs a mechanical change to fix.” In the latest wave some testers have used this box to describe what they thought of the rules (as if the question read ‘how would you summarise your opinion on the rules?’) so in the future I might add the line “as if for a new player”.

“How would you describe the game to a potential new player?” This tends to get at what the players found to be the most important aspects of the game. Introducing a game to a new group is largely about setting appropriate expectations, the expectations a person would choose to set says a lot about what they felt about a thing.

“How hard was it to learn to play?” This question was added as earlier waves revealed a learning curve issue and steps were taken to try to blunt it. It’s the more concise version of “Previous groups had trouble learning this so I changed X, Y and Z to make that easier, did that work or are things still tricky at the start?”

“What difficulty setting did you play at?” “How many players did you have?” These are important context questions necessary to interpret other answers. In particular comments on difficulty or game length could be sensitive to these factors so it’s worth being able to tell if certain types of feedback correlate with certain difficulty settings or player counts.

“How many games did you play?” Normally I wouldn’t ask this, but Nightmare takes less than ten minutes so normally a session involves more than one game – especially if one of the record breaking ten second lose game states occurs. Here it’s helpful because it gives an indication of the level of mastery obtained by the players before giving this feedback and it’s also good overview of what a group thought of a game. When groups are giving answers in the double figures I can generally look forward to them having had overall positive experiences, if a group reports one or two then it’s a signal that something likely went very badly wrong in one or both of those games.

“How many games did you win?” As this is a cooperative game the win-loss ratio is very important. Players don’t like to lose continuously, but also tend to find things boring if they win too often, so this gives an indication of where the difficulty curve is at right now. It also acts as another “Did they play the game right?” test, as when a group’s win-loss ratio is massively out of whack with the average responses it very often means that the group got a rule wrong (Sometimes a particular group is just exceptional, that shouldn’t be ruled out, but odd answers here make it worth probing further).

“Was the length of the total play session too long or too short?” I tend to ask this about games rather than sessions, but here the session makes more sense as the unit for analysis. A game can be ten seconds long and that can be a terrible or a wonderful experience – I’ve had players claim both. However their subjective opinion on the session as a whole is important, too short is generally good news but can indicate that a group wanted more depth and a longer payoff for their successes. Too long is typically bad news, indicating that something made the time drag on, whether the game was tedious or too hard and the group insisted on scoring one win or the reset time between rounds generated too much downtime.

“What’s the best thing about the game?” “What’s the worst thing about the game?” It’s always good to include this pair of very open ended questions, to capture experienced on subjects that it wouldn’t occur to ask about. It’s also telling when a group really likes the game or really doesn’t like the game – seeing what they still find to put for the question which opposes their views can get at some of the most fundamental strengths and weaknesses of the current version.

“What made the biggest difference in whether you won or lost the game?” If anything in the game gives the subjective feeling of unfairness it’ll get brought up here. It’s also a question that catches a lot of “broken” game states, if it’s coming up with stuff like “Victory hinged on drawing the such and such card” then that’s a specific problem that needs work.

“Are there any particular cards or abilities that you felt were too influential / not influential enough?” The second question is often more interesting than the first. As a designer everything that you put into a game tends to have some purpose or other and if something’s not serving its purpose it often needs to be changed or removed. Naturally players are more inclined to comment on things that made a big difference than things that merged into the background (they’re more salient after all) but it’s worth going fishing for options that made players go “Eh, I’ll probably never want to use that.” since they’re adding complexity to your game without contributing anything meaningful. Of course some things are supposed to create a background against which the rest of the game can happen and it’s okay for them to be serving that purpose without always standing out as the most influential thing, but if something that was supposed to be important shows up here then it’s time for a change.

“Were there any rules or cards that struck you as odd, out of place or difficult to follow?” Sometimes players will find something that doesn’t make sense and throw up a house rule or simply replace it with something that they understand. This is easy to do and keeps a game going, but it’s easy for them to forget that they’ve done it if you don’t bring it up. It’s also worth identifying things that don’t mesh with the theme or simply individual parts of the rules that are written badly.

“If you could make one change to the game what would it be?” Nintey nine times out of a hundred suggestions from people who’ve played something a couple of times won’t take into account something very important that you’ve found in hundreds of hours of designing the thing and will be less good than the solution that you’re generating yourself. So if you can get a hundred suggestions the chances are that one of them will be an improvement to the game, best not to miss the chance! Also I think that people actively enjoy answering this sort of question.

Do you want to play again (either as part of another playtest or when the game’s released)? This serves two purposes, the first is to make sure that you’ve got a pool of playtesters that you can draw on again once you’ve made changes based on this wave of playtesting. The second is that it acts as a much stronger “Did you enjoy the game?” question. The difference between being asked about enjoyment in the abstract and specifically subscribing to play again is very meaningful and generally this is the question that tells you that you’ve got a good game on your hands if you get lots of yesses – not the enjoy question above.

Any other comments? Some people will have amazing insights that they won’t share if you don’t give them permission. Never don’t give a tester permission to say something that they want to say, always welcome enthusiasm with open arms and hear everything there is to be heard 🙂

You’ll notice that a lot of these questions involve probing in a way that’s likely to generate some sort of follow up – I think that’s okay. It’s impossible to ask everything that you might want to know and a form that runs to more than a couple of pages becomes intimidating in a manner that means people won’t bother to fill it in.

Reaching out directly and getting back in touch with specific questions asking specific details has been a positive experience for me. I think that testers like to know that their work is taken seriously and the feedback that they’re giving is being listened to – intelligently following up questions reinforces that. Getting higher resolution data by having opening questions and then being able to probe into specific aspects of specific tests “Did this happen?” “Would it have been the same if that had been included?” “What do you think to this approach?” improves the game more than trying to get the same generalised data from every group. It’s not good to badger anyone with repeated follow ups, but this approach suits me.

Finally I’d like to give a shout out to Noc’s Box, these guys has a look at the game and did me the fantastic favour of passing me an unedited video of the hour that they spend playing it. There’s no series of questions that could hope to rival the richness of being able to see the whole thing play out naturally, so I’d conclude this post about questionnaire design with the thought that if you can see the whole thing happen live then you can dispense with most (but not all) of the questions that you might want to ask.

What do you ask your playtesters? Is there any place that we disagree on what’s important? I’d be interested to hear how other’s approach this issue.