Transcripts

[Aaron Hodder] Ummm, I'm sure we are all familiar with testing against quality criteria such as capability, reliability, usability, maintainability, portability,

[Laughter]

performance, and so on.  

But what about Fun?
When was the last time we tested for Fun in our products?
What even is fun?

Well, to help us answer this question, I am so excited to have Emma and Jonathan from PikPok. 

Now, as a gamer, I have wanted, forever, to go to a testing conference and have someone come up and talk about computer games. 

It's a whole area of software that basically all of us use, but no one talks about for some reason. It's almost a shame. Like

“What did you do last night?”
”Yeah, I went to the rugby.”
“Ah, I got my ass kicked in Elden Ring for the fourth time.”

Please join me in welcoming Emma and Jonathan from PikPok. 

[Applause and cheers]

[Emma Procter] Hey, everyone. We're from the research team at PikPok. It's a Wellington based gaming company. We're up on Willis street. 

Now, we are particularly excited to be here today because, um, we usually spend all our time with game devs. And as awesome and exciting as that is, they don't necessarily always appreciate the value of testing and measuring stuff, which is our jam, obviously.  

Games user research is often described as a niche within a niche. There aren't that many of us in the world and there's even fewer in New Zealand. In fact, I think we might be the only established game user research team in the country. 

But, yeah, as Aaron said, we're here to talk about the key metric that is very important and unique to our products and industry. And that's Fun.  

So, a little bit about PikPok, we sell Fun.  

We make all kinds and different types of fun. So we've got a really diverse portfolio. 

We make games on mobile, PC, and console. 

We make grown up, sort of violent, zombie apocalyptic games, if that's what you find fun.  

We make a very casual mini golf game.  

We've got quite an absurd duck breeding game, which is kind of a surprise hit called Clusterduck. 

And if horse breeding is your thing, we have a game for you. We have a very good horse breeding and racing game.  

A little bit about us. 

So, this gentleman here, Jono Shirley. He’s a Senior Games User researcher at PikPok, He's an industry veteran. He's seen a lot of games succeed, seen a lot fail. He's been at PikPok for a while.

I'm the Research Manager. I've been at PikPok for about three or four years now. But my background is a bit more diverse. I've worked in a lot of different industries, and in more market research, customer insight type of stuff.  

And that is our research team there, today.

So, before we dive into how we go about thinking about and testing for Fun. We want to give you a little bit of context on the market that we operate in. 

The global gaming market is huge. In-fact, the global player base makes up 41% of the entire earth's population. Gaming, today, is worth $200 billion, which is obviously a lot.  To give you context, the movie and music industry combined, doesn't make up a third of what gaming's worth. 

Revenue, especially for mobile, which is a lot of what we do at PikPok, is massively on the rise. With the 5G era imminent, it is going to make mobile gaming even more exciting, and more accessible. So growth is predicted to soar. It's big. It's really big.

[Laughter]

Navigating the mass market that is gaming, the competitive landscape and players' needs can be complex. Honestly, it can sometimes feel quite overwhelming.  

Not only that, the complexity in gaming products is really high.

So I've worked in lots of different industries. I’ve worked in startups, corporates, public sector, private sector, agencies…but nothing comes close to navigating the complexity, in delivering gaming products that I’ve experienced.

There's actually no standard coherent framework for understanding the complex nature of gaming genres, sub genres, and mechanics. 

So up here, is one that we use. This is from data.ai, recently acquired, now Sensor Tower.  

But as games innovate and mash unexpected categories together, something like this can be quickly outdated. So the one thing that is certain is, rapid market and innovation are part and parcel of developing games.  

So, on the one hand, we have a deep, competitive landscape, and elaborate taxonomies.

And on the other hand, a simple question of “What is a game?” still has an avid debate around it. 

Which leads us to the point of this speech. In this ever changing environment, can we answer “What is Fun?”

Now, it goes without saying that having a dance with Danny DeVito would be fun. 

Game developers, unsurprisingly, have spent a considerable amount of time thinking about this. And there are lots of different theories.

The two that I'm going to talk about here, there is one by a guy called Csikszentmihalyi...I looked that up and practised pronouncing it. It’s definitely chik-set-me-hi. 

He developed Flow Theory. You’re probably all familiar with Flow Theory. It's a little bit like Goldilocks. You know, not too challenging and difficult. Not too easy and boring. Right in the middle, is Flow.

Flow includes the idea of fun. So, some developers believe that this state is needed to achieve Fun.  

Whereas, this other guy, Raph Koster. He reckons that games are puzzles to be solved. And he sees that there are lessons to be learned right? So this is where the notion of fun comes from. Koster thinks that fun from games arises out of mastery, and it's the act of solving puzzles, and the process of learning, that makes them fun. 

But flow and learning still don't necessarily describe how us, in the research team, can measure Fun throughout development.

What we do know, for sure, is that Fun is a very personal activity and can look very different from one person to the next. 

Gamers are not a monolithic group. Some people are like Hello Kitty, and cute experiences.

And some people enjoy horror, allegedly. So Fun is very subjective and varies in meaningful ways.

At PikPok, we don't approach product strategy and development and testing with what is fun. We start by thinking about player motivations. 

So, specifically, at PikPok, we use this model, the Quantic Foundry Gamer Motivation Model. And this was established by a very smart man, over in California called Nick Yee.  He has a company called Quantic Foundry, and they use psychometric methods and data from over 1.65 million gamers to develop this. 

So very briefly, and probably doing a disservice to Quantic Foundry, I'll explain how they made it.  

After many iterations, Quantic Foundry used exploratory factor analysis, to identify how variables cluster together. They then used this to create their assessment tool for the motivations that they identified. This is the tool that we use at PikPok.  

So these motivations reflect intrinsic motivations: a person's interests, values, and desires. These aren't extrinsic, although they are important in games. If you do this, you get gold, for example. Importantly, these are intrinsic. So, there's a lot to unpack in this model and I'd be more than happy to talk about this later on, but I'm going to quickly explain some of this.  

So, there's 12 up here. We’ve got excitement and destruction. 

Excitement i.e. fast paced games. Destruction's pretty self explanatory. They tend to cluster together to be games that we describe as Action games. So, Call of Duty is a game that heavily leans into these motivations.  

Social. These are people who like working together. That could be competitive or cooperative. A game like World of Warcraft would be a Social game.  

Mastery. These are people that don't mind a higher cognitive load. So strategic games is enjoying having…making complex decisions. The consequence of your choices matter in the game. And Challenge is all about getting better at something. You practise, you get better at it. So, Dark Souls, if you're familiar with Dark Souls, is a game that leans into Mastery.  

Achievement. Power and Completion clusters together to be more around Achievement. Power is about getting more equipment, better characters, becoming more powerful in the game, so you can be more efficient and effective at what you’re doing.

Completion, very important in a lot of games: ticking all the boxes, doing all the missions, doing everything that’s in the game that you can do. Candy Crush is a great example of a game that leans into this motivation. 

Immersion. Virtual Reality has a head start in immersion because of the platform they have. Immersion is all about stories that’s world building: cool characters, cool world, maybe elaborate plot lines. Fantasy, in this context, isn't the Sci-Fi or Game of Thrones kind of fantasy. This is more about just being someone else, somewhere else.  

Finally, Design and Discovery cluster together for creativity. Discovery is about if I do this and I do that, what will happen, right? Open world games, they lean into Discovery a lot. And Design, that's a lot about self expression. Often customisation in games is a good way to express design. So, Roblox, The Sims, Minecraft…these are games that all lean into creativity.  

Another thing that's important to know about this model that we use for understanding fun is that these motivations are not measured like a thermometer; where zero is none, and more is better. They’re measured on a spectrum. 

So if a game scores low on story, for example, it doesn't mean that their motivations are any less real or lesser than. In-fact, they may have an equally strong preference. Instead of them being interested in lots of characters and lots of plot lines. It’s more about preferring a blank canvas, with no overarching narratives that has to draw you through the game.

One of the ways that we use this in our research team is to carefully screen for research participants via this model.  We do this to understand their full motivational profile. 

How do they feel about all these different 12 motivations we're looking at?

This gives us invaluable context of what and why our research subjects may find something fun, or not, and it gives us a blueprint of how we measure Fun, going forward.  

So, an interesting finding about these motivations is that we're discovering that fun is not just one emotion or one motivation. 

It's a process between multiple emotions, and it's a state that emerges as a result of some of those motivations being catered to, and the mix of feelings that comes with that.  

So what you can see here, on the left hand side, is an example of a game design hypothesis at a very basic level. This was developed in the concept phase of game dev.  

So, these motivations are what we believe reflected the game we were building. So our hypothesis, here, is that Fantasy, Discovery, Challenging situations…uh, what else have we got here? And, some complex decisions and engaging and strategic thinking, with a little bit of Destruction, were key experience pillars for our target market. So you’ll note that Community and Competition is quite low. That means we were going for a solo play experience, not a more social experience. 

So after we had run a prototype test. On the right hand side, the average motivational profile of the people who actually came through, they're all quite similar right? 

The patterns are looking quite similar: Fantasy, strategy's pretty high, Power’s sort of low leaning. 

But the one thing that surprised us is that Community is super high for this group. 

This is the profile that reflects how they like to play games, in general, not the context of this game. 

And knowing that they're high on Community doesn't necessarily mean we need to change our strategy. But it did give us an understanding of how we may be measuring Fun for this group.  

Okay. Cool. So, using these motivational measurements and testing them with real players, we now have a better idea of what we believe our players are going to find fun. Which we've used to develop product concepts and strategy in the Ideation phase. 

This is where we want to start heading into testing in Pre-prod, which is where Jono comes in.

[Jonathan Shirley] I'm going to grab a glass…a bit of water, even thoughI haven't talked. Thank you, Emma. 

So yeah, here's our game development process at PikPok. It’s similar to how most games companies do it.

Like Emma talked about just then. We had the Ideation stage with the concept, and thinking of the best ideas that we can go with from the research, understanding the game and motivation in a hypothesis form.

All going well, we can greenlight the project; going to preproduction with a small team, building out the best idea we have. 

Following on from that, you go into production, where we obviously need to add more content. Therefore, ramping up the team, and finally into launch if all goes well, and we ship the game.

But from our perspective and research, this research format by a fellow games user researcher, Steve Bromley, is probably a better way to look at it.

Here, we have the phases of development. We also have potential team goals at the time, during that stage of development. 

If we're going to look at pre-production for example, this is when we're building out the best idea we had from the Ideation period. And from here, questions that we might be able to help them with in research is

“How can we understand each system that we're creating in isolation? and
“How can we test the Core Loop to ensure it comes together as a fun experience?”

So, what do I mean when I'm talking about a Core Loop or a Game Loop?

Well, in every game, there's this action and this loop that occurs. 

Often in a game, you have to do something. And by doing something, you get something; whether that's an item or a piece of equipment. Once you get something, you're able to change something; whether that's levelling up, or building out your building, or anything like that. Often, this is a feedback loop, it repeats and goes around.

And as games get more complex, what we start to see is this idea of more loops involved, or loops coming off loops. And this is how we make a game more complex in development. So, when I talk about game loop, this is kind of what I'm referring to.  

So, how do we, in research, help with pre-production testing? Well, for one, we do usability testing.

Here, we're trying to validate the game’s Core Loop. Unlike many other industries, where we try to eliminate any friction in the user experience. In games, we are very conscious about players encountering friction. You need the element of friction for it to be a game.  

But bad friction or unwanted friction is in fact one of those main blockers in players achieving that Flow State that we referred to before.

So when we're conducting usability testing, we're analysing that amount of friction.  

Here's one of our games in early development. It's a grey box with the UI, put together pretty quickly. But it has all the elements of that Core Game Loop.

In this game, you go out.
You go scavenging, which is the "do" part.
You get resources, that’s the “get”.
And you craft items and upgrade your base, which is the “change”.

Questions we have during this period are

“What stands out as particularly good to the player?”
“What should be emphasised?”
“What parts of the game are not landing well during this development?”

Another one is usability testing those individual systems.

Here is another example of us testing out Rival Stars Horse Racing VR in its very early prototype staging.

Here, we're testing out various controls; collecting objective insights on usability and learnability. We're asking questions such as

“Do players understand the interactions?”
“Are they able to do what is required?”
“Does this interaction feel the way we intended?”

I will note, at this point, that we're not actually looking for bugs here. In fact, in research, we don't really focus on bugs. We have a great QA team, at PikPok, that does all that. 

We are looking for UX issues, or things in the experience that we don't intend for it to be there. And sometimes releasing a UX issue that is quite critical, can have just as much impact, as releasing a bug that shouldn't be there.  

Cool. So, we send this back to the team. We've done all these usability tests. The team's been watching, giving them deliverables,making sure there isn't a fraction too much friction, and now it's back to make some changes on their end. This is a cycle, ideally, for testing and iterating that just keeps going.

Cool. Okay. So, now, through that long stage of pre-production, and now we’re going to production. We're adding more content to the game, the team size is getting bigger, and there are further questions to answer around the player experience.  

All right. This leads me nicely to playtesting. 

We've tested usability. We're confident in the function and everything is working as expected. Now we’re going to look at playtesting. 

Playtesting is the moment where the designer and the players kind of start agreeing on what is fun in the game. Playtesting still heavily involves people playing our game, but we're trying to capture attitudinal data on top of behavioural data.

Behavioural, being what players do. And attitudinal, what players say. And often, these two things can be very related to each other, or they can be completely different. 

And to capture such things as attitudinal data, we're going to use questionnaires, surveys, and even interviews. Screening for the right people, and interviewing, is particularly important during this period of testing. We want to understand those players better, and their gaming habits.  

It's a good opportunity also to continue to check our player motivations that we're crafting. Like Emma talked about before, by having them fill out that Quantic Foundry Motivation profile. This provides more context around their experience and also understands what is fun for them.  

Cool. This is a fun photo in the lab. It looks like it's from the 80s or something.

[Laughter]

This is, like, a room that's next to where we sit. We set it up for a playtesting session.  

Here, we have four PCs, kind of divided off from one another. And we're going to have players play through this game. 

This is about six or seven hours of playing, so a day of it. We have four of those. We do that for three days, so 12 people. That's quite a bit of data. So that’s good enough to go on for now. 

And, yeah, that's Agent Intercept. It's on the Steam store...sold it.

[Laughter]

All right. So, here's a video of one of our playtests.

You might notice, it is kind of like the GIF example I showed before. But it's no longer grey boxed because we're in the production phase. Things are getting more polished and built-out.

I wonder if this will work? Let's try...there's meant to be audio here. We can just watch it for now.

So, as we're going through this process here. The user's talking about their experience as they're playing, and what they're encountering in the game…

[Technology not co-operating]

[Jono gives up]

So, I will tell you what they were saying. They were going “this is fun!”

[Laughter]

They said something along the lines of

“It's fun, I like the fact that this level is bigger, and there's more options, and there's more things to get involved in”

So that's kind of good for us. Having these choices and options is one of the experience pillars we're going for in this game. We want players to explore the levels and try different ways to navigate through it.

There was a moment there that was hard without the audio to see. As they came into this room here, there is a moment which is a highlight during the level, when a zombie jumps through a two way mirror. It’s kind of a jump scare. 

The music completely missed that, as they were picking up an item in front of them, and had a menu on the screen. So they completely missed that great moment.

This was an insight that we found and delivered to the team. They removed that item pickup there, so that wouldn't interfere with that experience.  

But, yeah, during this phase of playtesting. Things we're asking is

“Do players enjoy this?”
“Is this getting the emotional reaction we expected?”
“Are they using certain adjectives we’re looking for?” Certain descriptions about the game we want them to say
“Does each section work for the player?”

So, how do we tie this back to fun? 

Well, after conducting our research. We were able to collect all this data and show elements of where we think we're hitting the mark, and things where we might not be, or things we might not even know.

Things that we might be hitting the mark on are, you know, what we call expected fun. Things the designers intended to be fun, that the players are saying are fun. Great. Everyone’s doing an awesome job, and we can all go home early.

There also might be things the designers intended not to be fun, that a player is saying is not fun. This could be something like going through menus, or going along the map screen. If we asked if we investigated “Hey, was navigating those menus a fun experience?” and they might say no. That’s fine. It wasn't meant to be a fun experience.  

Where things start to get interesting, is unexpected but fun.

These are the things we didn't necessarily intend to be fun in the game, but players are saying are fun.

We’ve seen some examples of this in one of our recent games.

When you kill an enemy in the game, it did this animation of a spinny, twirly death. And players were acting really positive and responded well to that, which was unintended.  We can decide to lean into that more, and move forward more with that. Or maybe we might be like, um…that's not really the experience we're trying to create. We might go away from that.

But where we spent a lot of our time, and is ideal in research, is here. Things the designers intended it to be fun, are not what the players are saying are fun. This isn't ideal obviously. In this case, the player isn’t receiving the intended experience, and not having fun.  

And the word "fun" here can easily be swapped out with something else: difficulty, pacing, progression. Our job, in research, is to identify that, and isolate the issue, and give information to the team on how we may be able to improve that.  

Cool.  And I'll finish my part, here, with a nice quote from John Hopson, another game researcher. He says

“Asking Is it Fun?, isn’t valuable in most circumstances.
Instead, by ignoring the overall fun and choosing to perfect the individual moments and mechanic, we produce something more than the fun of its parts”

Back to Emma.  

[Emma] So, what we find in game dev is that finding the Fun is surprisingly elusive. But it comes together with a combination of big picture thinking about the overall experience, right up front. And then having the focus and intention on understanding how the individual parts come together.

An idea we'd love to leave you with is

“Could understanding more about what's intrinsically fun for your customers, influence the way you develop and test your software?”

Would love to hear what you think. Thanks.

[Applause]