Data-driven versus intuition-driven game design

When it comes to game design, we’ll find out one of these days if artificial intelligence can beat a human being at creativity. But today, the matter is still up for debate.

Some game designers believe that intuition is the best tool for being creative while others think that data-driven design rules, particularly in an age of digital games — mobile or online — where you can collect so much data about what players want.

What’s the right balance between science and art when it comes to game design?

We discussed this topic at Casual Connect Europe in Berlin on a panel that I moderated. The panelists included Philipp Karstaedt, general manager for Europe at Gree International; Tammy Levy, director of product for mobile at GameStop’s Kongregate; Adam Telfer, product lead at Wooga; and Christopher Kassulke, CEO and owner of HandyGames.

Here’s an edited transcript of our panel.

GamesBeat: What should game developers, at a minimum, know about data-driven game-making?

Philipp Karstaedt: There’s a place for data driven and a place for design driven. The only thing that can go wrong is when you say “this is the only way.” It depends on what you want you want to do, what stage you’re at, whether you’re a publisher or developer. There’s no right path. Everyone is still figuring this out.

Tammy Levy: It’s fine to think about both sides, data and design. If you’re going at it from the data side, get informed developers who can make design decisions. If you’re coming from the other side, learn about data. Understand the basics so you can make informed decisions. It depends on context.

Christopher Kassulke: As game makers, we’re all more coming from design in the beginning. But then, data takes over. We’ve seen that in the games and in the stores. A lot of games are becoming data driven. There’s some sense that what we’re missing is that the data guys aren’t really talking to the design guys, and the design guys don’t understand the data guys. They speak totally different languages. Most of the data guys aren’t gamers. Without any game experience, data is worthless.

Adam Telfer: My role is part game design and part data design. I’m at the meeting between the two. I see data as a tool, just like any other tool in game design’s toolbox. I go back to when I started in game design, when I worked in play testing. Data, all it is is effectively play testing your entire user base, using it as a high-level view of what’s going on.

If you’re just getting started, there are free tools available out there for collecting data, looking at things like retention and engagement numbers. That will help you take that first step toward making better decisions in game design.

GamesBeat: I sense a lot of emotion around the divide between data and design. As far back as I can remember, Zynga has been a lightning rod for this. They would talk about doing A/B tests on what color a button should be. That was a bit more of a controversy about how to design monetization into games that might be able to proceed without monetization, though. It’s not the whole topic.

Kassulke: Perhaps it’s not, but at the end of the day, ever since we’ve had free-to-play we’ve had data-driven games. Before, we’d just say, “This is a great game,” and you’d buy it at the store for $60. We had a design in our head, and we believed that was a good way of making a game. Maybe it’d be tested in small groups, but that’s all. Keep in mind that it’s always about what happens in testing.

Karstaedt: Zynga was really just the first company that used data for monetization. In the end, data is about way more than that. On the other hand, it’s also true that data has almost become too powerful in some cases. People have said, “This is my data hammer,” and suddenly, they see data nails everywhere. The art of game-making goes far beyond that. Understanding game design and how people interact with it, that’s data as well. It’s about retention. It’s not always monetization. People are unhappy about the monetization side for various reasons. But it’s a very powerful tool. Without it, it wouldn’t be possible to emulate some of the successes we’ve seen. It’s self-defeating to ignore data just because you don’t like monetization.

Levy: It’s funny that you mention Zynga because I was there at a point when it was overly optimized at times. It’s important to be aware of how to use data as a tool but not let it dictate all the decisions you’re making. Understand that it has its place and time. Not everything can be optimized to the last percentage point, the last extra cent out of each player.

It’s also important to talk about scale when you talk about data optimization. A lot of times, you don’t have enough players to optimize certain features based on data. Then, as a designer, you have to ask yourself what your instincts tell you as far as how to optimize the game and make it better at that point. If you only have 100 or maybe even 10 players getting to that point in the game, there’s no significance in your data. It’s important to understand at what levels you can use data and at what levels it can hurt you.

GamesBeat: Do you run into a lot of game designers who will not pay attention to data? I saw Warren Spector, who’s a pretty famous game developer in the U.S., put up a Facebook post after the U.S. presidential election. All the analysts were wrong about who would win, and so, he said, “If anybody comes to me and says I have to do a data-driven game design, I’m just going to say ‘Trump.’” Have you ever had to convince someone to look at data that way?

Kassulke: It’s not only Trump. It’s also Brexit, to give another example, here in Europe. Everyone said that Brexit couldn’t happen, and it did. Life is not that simple. You can’t quantify everything. Games are entertainment, and everyone is entertained in different ways. I always ask people, “What was your favorite Indiana Jones movie?” It’s always the first one. Nobody likes the last one because the marketing and analytics took over. It was just boring. Games that are only data driven will be boring.

Consumers are not stupid. You need to have some soul in your game. Data is good, but you need to make the first steps into a game as a game designer. “That’s my vision. That’s the way it will be.” Then, you can check that against data and perhaps change it step by step. But that’s a different topic.

Levy: As a publisher, we see both ends. We see developers coming to us where they’re fully design driven, and when we show them data they say, “No, I’m not interested in listening to that.” And we see the other end of the spectrum, games without a soul. “It has good retention because it has this feature that these other games have as well.” It’s just a compilation of pieces of other games that are supposed to work well, jammed together into one Frankengame.

For developers that don’t want to listen to data, one thing that I’ve found is very powerful is benchmarking, talking through the impact of making changes based on things we see out in the market. We have a lot of games out there to use as examples. “You think you’re going to be the outlier, but of 10 games out there exactly like this, nine have behaved in a particular way in this area.” Talking through the impact is important; “If you make this change, this is what could potentially happen in terms of these metrics.”

Telfer: I don’t want to talk about the Trump stuff too much, but when you think about that situation, it wasn’t just the data that said Hillary was going to win. It was also a lot of intuition from a lot of experts. I don’t think you can just assume that the data created that issue. Also, there was a lot of soft data from polls that had potential bias, which points to a problem in the way that data is gathered. In the future, we’ll hopefully have better models to gather and use polling data.

Going back to Warren Spector, he’s one of the few game designers right now who have that luxury of not being affected by data at all. Most of us don’t have that luxury, especially when you’re talking about free-to-play games, high-volume games, games that are trying to find a broad appeal. In those cases, data is very important.

When I work with designers who are resistant toward using data, it usually comes down to the idea that as soon as we start using data, that means we’re going to entirely focus on quick wins and what you said about Frankenfeatures. That’s not really true. It’s two separate things. You can look at data and use it to follow an easy path, or you can use it to try and build a more engaging part of your game.

Let’s say you want to increase retention and engagement. One way is to add a daily reward system, something that gives players a little reward every time they come in. That’s the usual way. Another way to approach it is to look at our game design loops and try to make them more engaging, so it’s inherently fun to come back. You can start both of those paths from data that says players aren’t coming back, and you need to build higher engagement.

Levy: Where we see that work best is when we point out a scenario or an improvement, and then let the designers come up with a creative way of approaching that problem, instead of dictating a way of doing it to them. It’s a hard thing to do. Say you’re testing a website, and you have trouble finding a button. The obvious solution is to make the button bigger. But it might be the color or the way the layout leads you to it. You need to let the designer fix the real problem. A lot of data people make the mistake of simply saying, “We need to increase retention, so you should add a log-in bonus.” But that’s not the problem. You need to be able to say, “I’m seeing this gap. Can you come up with a solution to fix that?”