Data-driven versus intuition-driven game design

GamesBeat: On soft launch, this is a fairly new phenomenon, but nowadays, people will put a game into soft launch for quite a long time. Games are going into soft launch for as much as 18 months or two years. Supercell puts games into soft launch and then kills them. What are we supposed to learn from soft launch? Why do some games take such a long time?

Telfer: For Dawn of Titans, they were in soft launch for 21 months. They were looking for a two-point increase in their 30-day retention, something like that? That’s how important that was. If you increase your retention by two points, you’re looking at a pretty big difference in your overall LTE, how long that game lasts. To me, you should soft launch as long as you want. When you launch, that’s the one chance you have at feature traffic driving a critical mass into your game. Why not take the [time] and soft launch with data so you can properly evaluate your design?

GamesBeat: If it doesn’t make the charts, though, is that all just a waste of time? Dawn of Titans still tanked.

Telfer: It’s hard to say because I don’t know how much money is made in soft launch. I assume that if they hadn’t done that, it would have tanked faster. Is it worth 21 months of production? That’s pretty hard to say. But I know from looking at soft launches that it’s a very tough decision to stop a game in soft launch. It’s a game that designers have put their heart and soul into for a number of years. Are you going to tell those people to kill their game because they’re off by two percentage points on one number?

Karstaedt: That’s the epitome of games as a business. But in the end, it’s true. If you wait for almost two years, the market moves a lot. A lot of games that have been successful have excellent retention and monetization, but they also hit the taste of the market at a particular moment. Coming back to this question of design driven versus customer-insight market driven, there needs to be a balance. You can have the best game for two years ago, but at this point, people don’t want to play it anymore.

You can’t get it right by making just one decision. It’s one decision after another, and at some point in there, you’re going to have to get lucky. Data can help maximize your chances. But even Michael Jordan didn’t hit every one of his shots.

Levy: It’s also important to set out your goals for soft launch and understand what you can and cannot do with it. We’re talking about a company that had the luxury of keeping a game in soft launch for almost two years to maximize D30. Let’s say their D30 was three percent and aiming for five. How many players are going to come in over and over again to validate that D30? It’s a huge amount, thousands. That costs money. Indie developers don’t have the marketing dollars to do this over and over again.

You have to get smarter about what you’re trying to validate during soft launch. Sometimes, a small team won’t even have enough resources to validate an ARPDAU. You’re talking about one to two percent of players. You also have to get smart about the data you decide to analyze during soft launch. I hear over and over again from developers who want to do two A/B tests or five A/B tests running at the same time. I’m supposed to be the data person, and I’m fighting them on this. You need good test design, and if you want to validate all the hypotheses you have, you’re going to need a million-dollar marketing budget that you don’t have. So, get smart about how to approach this. What’s the key question you want to answer during soft launch? Go after that.

GamesBeat: Chris, are you doing any soft launches?

Kassulke: No, we’re no longer doing soft launches. The question is, is it worth two years of spending and investing time and changing the game, when something can happen out there and suddenly your concept is no longer interesting? Or your competitors catch up and pass you because you took two extra years? I know too many companies that are good at checking out what others are doing in soft launch.

GamesBeat: That happened to the Flappy Bird guy. They were getting ready to launch Ninja Spinki in December, and somehow, everybody found out, and clones started flooding the market.

Kassulke: Exactly. We absolutely check out what our competitors are doing. That’s very interesting for analytics and data because they do the homework. It’s easier for us nowadays to simply release games. A game that works in Portugal isn’t automatically going to work in Brazil. A game that tests well in Austria might not work in Germany. Depending on the way you go about your soft launch, you may not get the right data.

That’s why we’ve decided to just go out and release our titles. Maybe we won’t get a feature on Apple or Google the first day, but we can bring our consumers automatically to our game, which can mean one million downloads right off. We save a lot of money, because CPIs are crazy nowadays. If you want to do test markets and all the data, you have to invest a lot. As you say, do you have a marketing budget of $1 million? That’s nothing nowadays. With that much, you’re not even in the big league. You’re playing down there with the indies.

GamesBeat: Are you significantly changing games after launch?

Kassulke: We do that later on because then, we can get real data. Everything else before that is fake in a way. The title isn’t ready. You’re changing things. You do a lot of testing while competitors are coming out. Two whole years of testing a game just to get two points higher, so Apple likes you more, or you get better investment of marketing money? It’s still a bet.

GamesBeat: I get the feeling Tammy disagrees with you.

Levy: I still feel like there’s value in test markets. But it all depends on your goals. We have games that we test market for three weeks and games that we test market for four months. We try to be conscious about this and not do an excessively long test market.

You can do simple things like testing the stability of your game. You can release in one country, get some users, make sure the game isn’t breaking. When you release it globally, at least that won’t disappoint those first players early on. You can go to some countries in Europe, spend a few thousand dollars, and still get some big wins in terms of validation.

Again, this is focusing on things you can actually change. I’m not going to focus on my D30 retention, but maybe I can focus on my tutorial and make sure that people get to see the whole game and learn the game. I can make sure that’s working, in broad strokes, during a short test market.

GamesBeat: A lot of this makes me think of Mark Skaggs, the co-creator of FarmVille and a lot of other big titles at Zynga. I recall that he developed a philosophy of game designers swallowing their pride, understanding that they have to rely not only on their intuition, that they have to realize what they want isn’t necessarily what the market wants. In Zynga’s case, early on, you had a lot of younger men designing games for an audience that turned out to largely consist of older women. They had no instinctive idea of what that audience really wanted.

Levy: My favorite quote from Mark Skaggs goes, “Measure five times, cut once.” Whenever there was an argument, that was something he kept saying over and over again. Create a plan but be informed about what you’re going to execute on before you do it.

GamesBeat: I also remember Andrew Sheppard saying that they’d changed from the days of launching minimum viable products, toward launching more finished products. This is a bit like what you’re saying. You’re not just putting something out there, so users can finish your design for you. You complete what you’re envisioning and then put it out.

Kassulke: We need information for every game. What we do is essentially test groups. They’re sitting at HandyGames playing the game with the game developers, and then, we can ask the right questions. We know what they see in the game and what they don’t. Are they checking out advertisements? There’s technology out there to really test-case your games. We can even tell what they don’t use and why they don’t use it. Then, you can do another test group and find out if the yellow button works better than the blue button or the red button or whatever. You can already do that.

We’ve checked out places like the Philippines as a test market. Money works much differently than in Europe. The user behaviors are different. When I see a game that’s half-ready and the developer says that they’re kicking it out of their portfolio because it’s not going through the funnel, well, of course not. That’s normal. Why would people play the game? I wouldn’t even put something like that on the market as a test case.

I often joke that I’d love to get all the titles that make it into the last third of the funnel from Supercell, from Zynga, from you guys. Those games could make a lot of money if you just did things a little differently, if you didn’t spend millions on a single title. Maybe you don’t need that. Maybe you just need to advertise a little differently. I’m still waiting on getting the discards from that end of the funnel, but I’d do it.

Karstaedt: I often hear people talking about games as if they’re any other kind of product, which I think misses the mark completely. A consumer product makes people’s lives easier. A game is not a tool. It’s a story we tell. If you put out something where you’re not convinced about it yourself, people can sense that. They don’t want to play something that’s designed by committee. They don’t want to be the person who has to help you improve this. Certainly, the Philippines and all these other soft-launch territories are getting more expensive. Australia is one of the territories where you would say the players are fairly similar, or Canada, but everyone’s doing that now.

Soft launching can help your game better, but it’s more about launching it and continuing to develop it, adding features along the way, which you should be doing with a mobile game anyway. Adding content to keep the game fresh and keep the players engaged. You can do that based on indications of how people interact with the game and what they like most about it, but that’s extra. That’s how you keep a game alive. It’s not how you bring a game back from the dead. You can’t fix a bad game with data.