Above: KPIs.
Question: Fundamental question — how do you measure fun? Any design process that requires iteration has a feedback mechanism. But if you use time in game as a proxy, that can just mean I’m incredibly frustrated. Maybe I’ve played a level nine or 10 times, and I spend a little money to get past it. In the short term, you’ve made money, but in the long term, you’ve upset me because I’m not having fun. How do you tell if I’m having fun?
Telfer: One simple way I like to see how a user base is having fun is just chatting with players in the live chat. Looking at high-level KPIs, you can’t necessarily say if a game is going to be fun over the long term. You can look at symptoms of what fun is. You can look at engagement in low-level gameplay — battles, rounds played — or in something like a match-three game you can look at how many moves it takes to finish a level, how close players get to completing the puzzle each time. That can give a sense of how fun that level is. We can look at the life span of a user, how long they stay in a game. But until we get into things like biometrics, it’s very difficult to measure.
GamesBeat: The fine line between fun and frustration is important. Are there particular KPIs that game designers should be aware of?
Telfer: One big caveat that I always find with metrics is segmenting your user base by your user-acquisition strategies. The source of a user is very important. When you’re looking at averages across all of your data, your user base includes a whole bunch of different types of players. But just segmenting by user acquisition can give you a sense of bright spots in the market, which source is giving you your best users. That should be the ultimate goal of your product marketing — not just buying enough users to keep the game going, but getting users of higher quality. We’ve seen 20 percentage point differences in things like retention just because we’re using a different marketing source.
Levy: You should also segment by concrete platform. Android and iOS will probably be different. Organics versus someone you paid six dollars for will probably be different. Sometimes, you’ll have to group. You can’t get every single country because some countries will have very few users to let you get significant data, but making sure that you’re considering these differences is important. You might be optimizing for an average that doesn’t speak to the true appeal of the game.
Kassulke: When you’re talking to Apple or Google, for example, just one platform, they’ll want to see their retention rates. If you want to have any promotion, check that out. If it’s not correct, they’ll probably not feature you, and that’s very cheap marketing. As you say, it’s important to check out different target groups. Concentrate on tier-one markets and then take care of others.
Karstaedt: Talking about platforms and what they want to see and combining that with issues around soft launch — one really important thing is finding stability. When you load up the game and interact with it, does it work for everyone? Even if on average one percent of users is affected, on some platforms that might be a large group of users. The propensity of people to leave you a one-star review if a game crashes is so high, and then, Apple will come at you and say, “Well, you have 3.5 stars. Your game isn’t a good experience for a lot of people. We’re not going to feature you.” Not dealing with that is simply shooting yourself in the foot. Monetization data is also important, but fundamentally, the most important thing is getting your game stable. Make sure it loads quick. Make sure you eliminate those simple frustrations. Then, you can move on to other things.
I totally agree about segmenting — and segmenting very specifically. When you’re doing that, think about what you want to do with the game. You can’t design something that appeals to everyone. If you want to design something that appeals to people who like short sessions and then you see that people are playing your game for a long time, that might not be a good thing. The data can help make those decisions. First, understand what you want to give the market and then optimize toward that.
Above: The Walking Dead.
Question: You can soft launch for a short time, but that will only give you data on the initial phase of your game. How do you stress test something like economy design? If that breaks down in the long term, how can you test for that?
Telfer: It’s a very tough problem to solve. Do effective play testing internally. Make sure you have ways to model your economy and project something like a year or two years down the line. Play your game as if it was two years later. Implement a way to load it up on a device that plays the game as if it’s been running into the long term.
Levy: We have models that stress test different situations. You’ll know where players are at seven days. From there, you can develop models that look at what players are spending and how fast they’ll get through levels. If players aren’t spending quite that fast, will they get through levels at a different rate? You need to get into your economy design and test it, see how it pans out when you change key variables. Does it end up where you thought it would?
Kassulke: It’s one of the hardest things to understand, that kind of long-term social behavior among players. From the beginning, it’s important to make sure that there’s a good way for new players to reach out to old players. You can run simulations of player behavior, though it’s hard to do — run a game very quickly with 10,000 simulated agents that interact and form guilds. If you see a point where they stop interacting, that might be a problem. It might simply be easier to trust to your instinct — “I’m going to do everything I can sensibly and hope for the best.” If a game is active for two years, after all, that’s already a win. I’d take that to the bank.
Levy: Once you’re live, you can add more things to your game. Your game is going to be very different after two years of life in ways that you can’t necessarily predict. This is really a place where you need to trust your design intuition to make the right choices. As he was saying, if you’ve kept players for two years, you already have a fantastic game.
Above: Call of Duty: Infinite Warfare publisher Activision gave fans a chance to play the game early.
Karstaedt: I would simply say you can’t do it. That’s my personal opinion. We have a lot of titles that are out in the market. The oldest one that is still regularly played is 10 years old. People are still playing it. But that has to do with bringing new content in. We let the consumer choose some things. But can we predict that? No.
We’ve released a lot of games. All of our titles are normally built for three years. That’s a minimum. Then we go on and ask if it’s worth it to invest another year or two into the title — part two, part three, whatever. But really, planning for that is impossible. If there’s someone who can do that, good luck to them.
GamesBeat: I’ll leave you guys with one thought. If you look at The Walking Dead game from Telltale, they force you to make decisions — who you’re going to save, who you let die. Then, at the end of the game, you see percentage choices that users make. Maybe 55 percent let a certain character die, and 45 percent saved their life. That’s fun for me.
I think of Call of Duty as well, where they use heat maps of where people get shot in multiplayer to figure out the design of a particular level. Game designers use that data, but then, they also release it after a match, so you can go and look at where you died and become a better player as a result. There are ways, I would say, that data can be fun.
Disclosure: The organizers of Casual Connect paid my way to Berlin. Our coverage remains objective.
