How to stop losing revenues and players to cyberattacks

The video game market has grown to $116 billion worldwide, and that makes it a big target for cyber criminals.

Sony felt the sting of cyberattacks in 2011 when hacktivist group Anonymous took down its PlayStation Network. That forced Sony’s CEO to apologize to all of the company’s game customers. But that was hardly the only attack. Video game publishers lose up to 40 percent of their in-game revenue and microtransactions to fraud each year, and ransomware has taken off as well. In 2016, Valve disclosed that Steam Stealer malware was compromising 77,000 accounts a month.

That resulted in a huge loss of revenues and players as well. I recently joined a webinar with a couple of security experts at game companies about this problem. The panelists included Ryan Safarian, vice president of engineering at Lucktastic publisher JumpRamp Games, and Arash Haghighi, manager of infrastructure at Smilegate West, publisher of online games such as Crossfire, one of the largest first-person shooter titles in the world.

Here’s an edited transcript of our interview, including questions from our audience. The session was sponsored by Akamai. (Here’s an audio version of the session).

GamesBeat: How do we detect a cyber attack, and what are the first steps in dealing with it? 

Ryan Safarian: There are a lot of healthy steps that any business can take toward threat detection. The first recommendation would be getting your baselines on a couple of following topics. The first thing I would recommend is your ingress and your egress. You want to get a solid understanding what your activity cycle looks like over a set period of time, whatever that ends up being. How is that time frame defined by your business? Whether it be a session a day, a week, a month. But just getting an understanding of your peaks, your valleys, what things look like during the day.

Another healthy step — you can do very primitive stuff. Status calls. Just get a solid understanding, simply put — how many 200s, 4xxes, 5xxes are popping up in that similar activity cycle? Whether it be programmatically or something like that. You can build some kind of logic, or not even get that far advanced. You can start having some predictions around what activity is going to look like. If you do a burst campaign, or a special promotion, you’ll be able to identify what looks normal and what doesn’t.

What kind of preventative code do you have? For all the engineers out there specifically, follow through the UML. Go through every single use case. Try to understand that preventative code and the nature — the notifications you can build around the most sensitive and critical parts of your system. Not everything needs a notification. Whether you’re doing pager duty or a Slack notification or email, you’ll drive yourself crazy. You’ll end up essentially desensitizing yourself to a lot of information. You just want to focus on the most critical systems first and start working around that.

You don’t know what you don’t know. The more detail that you have around the transactions that are going in and out of your system, the more educated and more prepared you’re going to be to detect these attacks. When you’re in the middle of an attack, you want to isolate that attack surface area. Is your application modular enough where you can temporarily disable the service, or could you try to quickly get a sense of the effect on the application as a whole? Is this bringing you down completely? Is this a game that has completely hampered all of your end users?

You want to quickly get an assessment of that. The engineer needs to report all of this information for your dev ops team. They need to bubble this information up. Your CTO, CFO, CEO are going to need this information to make those critical decisions and pull the proper levers. You guys are on the front line. You need to quickly understand what kind of hurdles you can throw up to dissuade the attackers.

Attackers are going to keep coming. You need to treat it like triage. There will be times where some really ugly code needs to make its way into the system. There have been times where, at four in the morning, I threw in some stuff I’m not proud of. But you can always refactor that and put in something a bit more elegant. You just need to stop the bleeding.

GamesBeat: Arash, do you also have an answer on this?

Arash Haghighi: To detect any attack, you’ll need different kinds of sources and tools. We can recommend to any organization, any company developing games, you need to use a different kind of monitoring tools to make sure what’s going into your network, into your servers, your throughput and bandwidth. You have to come with the tools and make sure there’s no unauthorized activity in your network and your servers. Going through the event log checking, going through servers, using IPS and IDS and different kinds of firewalls.

Regarding human behaviors and the tools, you have to monitor things like pop-ups, or any weird emails. Any weird password activities. Network bandwidth monitoring. Any kind of drive usage. Any kind of abnormal CPU or memory usage. Check the transactions, check the events. You have to have a team, or strategy and plans to always monitor your competitors. Always find out, at firms everywhere, how people are talking about your game or service. That kind of information will help you predict or detect an attack.

You have to come up with schedule change policies. You have to be careful not to share too much information. If you’re going to have a guideline of some kind for your players, you have to make sure not to put in too much info that’s related to support activities. If you have strategy plans, initial plans, you can deal with any kind of detections.

GamesBeat: What are the main risks to the business when a cyberattack occurs?

Haghighi: It depends on the service, or the region, or even the business goals involved. If you’re under attack or under threat from hackers, you have to think about what kind of risk that represents for your business. Of course you’ll have players and customers complaining in forums, even your own game forum, and that can be dangerous for your business. You have to make sure you satisfy everyone, and that’s hard.

On the other hand, if you get attacked, the company ranking may be hit, and then you have to make sure of your place in the market. Game revenue as well can be impacted by any attack. So it depends on what you can figure out about the mitigation of an attack. If it comes from competitors, maybe users will go to them. You have to make sure you keep your users as much as you can. If you’re losing IDs, passwords, account information, that’s very hard to do. You have to be able to show improvement or have compensation plans to recover lost market.

You have to make sure you can save your critical information against any leaks. Of course that kind of leak can be a disaster. Hackers might even be able to open a case against you to get more information. It’s important to be able to keep your business safe as much as you can.

Safarian: To Arash’s point, the revenue impact is going to be the biggest risk to any business, but there are other impacts to your core user base. If you start looking into your retention and your user experience, we all get an understanding of the cost-effectiveness of retaining rather than acquiring new users. If your hardcore players, application users, the ones that are dedicated, the ones whose trust you’ve earned, who are constantly coming back to your application and being a touch point — you really want to make sure that their session and their flow is completely unmolested. You want them to have a very steady and consistent experience.

The third part of that, after revenue and retention, is going to be new user acquisition. You’re going to have to constantly pull in new users. Not everyone has that viral app that’s widespread. You have to purchase new users, acquire new users through different pipelines. Think about the revenue impact as far as that’s concerned.

If there is any issue on your game, you have to do one of two things. Either completely stop the new user acquisition funnel you have, which is going to throw off all the third-party analytics and data that they’re doing, if you’re acquiring from Google or Facebook or something like that — they have algorithms around these things. If for whatever reason your application is at a standstill, now you’re triggering a series of events that will take a long time to recover from. Google will all of a sudden see the conversion rates dip, which will put you down lower on the charts, and now you have to fight an uphill battle to get back to the numbers you had originally.

You might have a healthy user acquisition funnel, but for whatever reason, if that comes to a halt, now you’re relying on your business executives and pull a bunch of levers to get out of the rut. You’re spinning your wheels in mud at that point.

Those are the three most important components as far as how it relates to the end user, but there are also deeper business impacts as far as third-party relationships. At JumpRamp we have relationships with Hasbro and MLBPA. There are efforts that we need to secure around personal identifiable information, around data storage, all that needs to be considered, because if there’s any kind of misstep, all of a sudden that relationship dissolves and it’s a major detriment to that contract. Or any kind of promotion we’re giving out within the application. We need to be considerate of that. There’s going to be an end-user impact that will directly relate to the deeper business impact.

GamesBeat: Another question is: Can we prevent cyberattacks, and if so, what are the steps to take?

Safarian: It’s a loaded question. I’m going to flat-out say no. You can’t prevent it. There’s always going to be a bad actor. If you have, for example, any end point, any API, you are available for an attack. There’s a liability there. There’s always going to be a bad actor out there that’s going to try to attack your system. You can’t prevent that, in my opinion.

All you can do is lessen your attack surface area. Use best practices. Build those robust guardrails around your most critical systems. If anybody is working on a cloud or a private VPN, you want to make sure you have your ports locked down, your security groups in line. At the most basic level, are there any mid servers you need to have on any of these nodes? What are the monitoring and detections like? Are any back doors exposed?

There are best practice white papers all over the place. Do yourself a favor. The business will always force you to develop and iterate quickly. It’s your responsibility, as a developer or a dev ops engineer, to pump the brakes a bit and fight back. “I understand we have this feature, this next thing we need to get out there, but we have a serious risk sitting here that can put a stop to everything we’re working so hard for.” Your voice needs to be important in those conversations. Again, you’re on the front lines. You’re going to bubble this information up for those critical strategic details that need to be acted on.

You can be proactive. If there’s a known phishing threat, for example, or a list that’s been exposed — user emails and passwords — how is that applicable to you? Nine out of 10 times, if there’s a user that’s been exposed before, they’re going to be compromised in your system as well. It’s simple things. People generally use the same password across the board. We’ve increased our IQ around that a little bit and people are being more aware, but there is a large chunk of the population across the globe that still does not adhere to any of those practices. As soon as a password is compromised in one area, a bad actor is going to take those exact credentials, run a script, and bombard your system. You need to figure out a way to deter that. How can you build a system, or rules, around helping your users get out of that rut?

There’s a bit of education you need to do as well. As a company we need to help educate our users and lead them. Whether it’s stronger password protection, whether it’s encrypting or hashing certain data — when you’re storing this data, make sure it’s not easily accessible.

Haghighi: Always, security is not 100 percent. Even if you try to protect everything as much as you can, there’s always a new way, a new technology, new areas that can leak. If you’re using an API, for example, you always have to have some concerns about that.

As far as prevention, you have to analyze your work flow first. You have to know your game architecture, your service architecture, and your organization architecture, all those working data flows. You have to come up with security plans around your work as much as you can. The hacker is trying to turn that around and find any way to come into your system.

According the last set of statistics and reports I got from our security company for 2017, attackers have even been trying to use apps they’ve published themselves on the Google Play store. Users download these apps, install them, and they act as zombies or weak links for attackers. Hackers can use these resources to attack a game through DDoS attacks, web traffic attacks, brute force attacks, and others. We have to build an educational plan, for employees and for users.

Part of prevention, in the beginning, is using monitoring tools. Don’t allow any weird attachments into your development team, your dev ops team, your customer service teams. Everyone should be trained about that. Never use free software. You have to think about why software and tools are free. Monitor abnormal traffic. You have to have defensive tools and a defensive strategy around things like firewalls and IPS. Meet regularly to think of any possible breaching areas and try to fix that with your developers and dev ops team.

According to reports, maybe 86 percent of attacks these days target the gaming industry. We have to think about many kinds of attacks, including DDoS attacks and web attacks. You need a solution for that. If you’re using the cloud, you have to come up with cloud mitigation solutions. If you run your own data center or co-location, you have to think about other kinds of mitigation services. Upgrade knowledge, use new technology, and do all the training you can.

Once your employees and staff are trained, that can keep hacking activity from even coming to your network. Everyone knows about social hacking and social engineering. That can come to your organization without any activity on your network. You have to think about training and talk with your employees and users about what not to trust. There are many ways you can find malware emails and the like. You have to do training with your employees.

GamesBeat: I see frequent news about cyberattacks of all kinds. We all saw the non-game industry attacks that happened during the U.S. presidential election. That’s raised awareness among people around the world that this is a serious problem that hasn’t been solved. Every now and then you see something more catastrophic. A serious case last year involved Valve’s Steam service, where 77,000 of its accounts were being hacked every month, and it uncovered this script-kiddie malware, SteamStealer, that made it easy to prey upon people with lousy passwords.

Haghighi: I have a few statistics. According to reports I’m getting, in the second and third quarter of 2017, DDoS attacks in the gaming industry increased four percent. 86 percent of DDoS attacks are happening in the gaming industry. Several other industries — telecom, finance, software, education, retail, media, entertainment — all that together is less than five percent. The UDP fragment attack is the most popular, still, but hackers are also using DNS, NTP, and other vectors of attack. A company should make preparations against these kinds of attacks.

For DDoS attacks, they fall into two different categories: infrastructure attacks and application layer attacks. For applications, attackers are using GET, PUT, and POST vectors that you have to make sure of. Regarding web application attacks, in the third quarter of this year, we’re still seeing SQLi attacks and LFi attacks on the top. Any developer or publisher has to be careful of these kinds of injection attacks and others.

Depending on the region you’re in, and other criteria, different kinds of attacks are going to happen. You have to come prepared with different kinds of solutions. It’s very normal, in the gaming industry — for example, we’re closing in on Christmas this year. Depending on the game and the area you’ve launched that game, you have to expect some kind of attacks, but you can prepare for specific ones.

Safarian: The fact of the matter is, as a business, you’re working hard to create a positive user experience, some kind of value-add, some kind of evergreen application. We’re all fighting for that prime real estate, as application developers or game developers. We’re fighting for that place in the user’s daily regimen. The average user is going to maybe engage with four, five, six applications. We’re all fighting for that real estate.

Now imagine that you’ve worked so hard to become one of those evergreen applications and now everything just falls apart, because of those bad actors that we keep talking about here. The IQ and the attack preparation around these hits, they’re only increasing. Every week there’s new tech and more vulnerabilities being exposed. At some points these vulnerabilities are coming a lot faster than the solutions, the stability and the security. Yes, these things are the things that keep me awake at night, because again, you don’t know what you don’t know. A lot of new things are coming up every day that people are trying to use to destroy the stuff you’ve worked so hard for, to try to create this positive experience for your users.

People who say that they’re concerned, but not worried, I’m happy. I’m glad the anxiety hasn’t crippled them to the point where they’re awake at night. My question back to the would be, how critical are these things in your application or your game? If there are systems that are not super critical, that you don’t mind — what’s the cost-effectiveness of getting hacked? At some point you will have to evaluate that. “It’s not worth it for us to prevent an attack here.” That’s the ugly truth sometimes, speaking from a business standpoint.

Things are improving every single day on the security side of things. However, I just don’t feel like it’s as fast as the vulnerabilities that are being exposed.

GamesBeat: When an attack happens, how do we deal with it and help avoid reputational damage? We’ve covered a lot of that already, but the reputational side of things is a new topic here.

Haghighi: It comes in two different parts. First, let’s say, once the attack happens, what are you going to do? It doesn’t necessarily matter what kind of attack in this case. If you google anywhere, you can find plenty of reasons to say, “Don’t panic.” Of course, you’ll get a lot of complaints once an attack happens — from your CEO, from your founders, from players, from the news media, everywhere. But first, you can’t panic or give in to stress. You need to be calm. You need to think about the scope of the attack, and stay calm.

In my previous experiences, in use case scenarios and live scenarios in this industry and other industries, at first you may know what happened. Users can’t log in, maybe, or your administrator password has changed, or your services are down, but you don’t know why. You should never try to turn off anything right away — servers, routers, service. You have to think about that. You have to start to investigate that first.

Definitely, you need initial plans. You have to come up with team management plans. You have to think about who can help you. You have to have a strategy coming from your IT department or security department, and then you have to go with the recommendations of that plan.

The first thing should be to measure the scope of impact. Let’s say, if the service is totally gone, you have to think about how you can turn it back on. You need initial backup plans for that — backup service, disaster plans — to be able to start the service running again. Then you can start tracing the activity in your network. Maybe the hack has left some logs, or your monitoring tools can help you with that part. You need to investigate as much as you can, find any logs you can, and find the leaking area.

You need to monitor inbound and outbound traffic, but that should be in a totally transparent mode. You have to think about that. If the hacker recognizes that you’re tracing them, they’ll stop activity. You have to be transparent, but you have to monitor to find out what they’re doing. Or if this is a DDoS attack — let’s say it’s saturated your bandwidth, saturated your service in total. You have to find a way to connect to your service again and relaunch it.

For all of this you have to have initial plans. You could reroute traffic, for example. Different companies have honey pot services, so you can route your traffic there and then start the rest of your plans. You can block malicious activities, start the server network cleanup, and restore data. It depends on the attack, but you may have to restore some data. Restoration should be a shared experience with a trusted party. In some cases you have to bring in a consulting company, a third party, to help you, if your team on your company’s side can’t help with the issue.

These are the technical parts that you have to deal with if an attack happens. Regarding reputation, of course, the first point is your users. They may be aware of an issue, but they don’t know what’s happening. You don’t need to share that you’re hacked, or that you have an issue, but you have to have a prepared web space, prepared services, so you can redirect traffic to a maintenance mode and then say, “This is the issue and we have to investigate.”

You have to make sure you have a good compensation plan and package. It depends on the service, but since we’re talking about games, we have some in-game services, in-game events. We have to have satisfaction plans and packages that we can come up with, so we can talk about that with users. It all depends on the scope of impact. If it’s only affecting a few users’ information, you can deal with them directly without any public announcement. If it’s the entire game, the entire service impacted, you have to come up with a plan to publish that news.

You can potentially consult with third parties, especially if it’s happened before to similar companies, or if it’s a part of a global phenomenon like ransomware issues. But you need to be calm, and then you can find a lot of ways to avoid damage to your reputation.

Safarian: To highlight and expand slightly on some of the things Arash mentioned — let’s just put ourselves in that situation. The attack is present at the moment. You need to, first and foremost, take a deep breath. Keep composure. The stress that it’s going to bring you—if you’re level-headed about these things you can make better decisions. You want to avoid making any kind of drastic or destructive decisions in the application.

There’s internal and external. I want to divide this up. First thing is, just evaluate the effect here. What is the net impact? How serious is this attack that’s happening today? If it’s somebody who is scamming tens of thousands in virtual currency in your system, you need to evaluate that. Is this good or bad for the system? If this is someone who’s gone in and is changing personal information for other users, you need to stop that. You need to evaluate that. How much of a concern is this? Keep your stress levels intact.

The next thing you want to discuss with your team internally is the exposure of that hack. There are times where bad actors will go out there and they like to brag about it. You want to check social media. Simple things. Even Google keywords — bots can go out there and capture these keywords so you can understand, okay, did somebody brag about a hack on a gaming system? How easily is that re-created? How much exposure is there for copycat attacks? How much information is out there? You want to manage the information going out. You want to control that message.

Let’s say the dust starts to settle after you’ve started to handle the attack. You want to understand, okay, what are the after-action tickets? There’s going to be a lot, across the board. Whether it’s something at your infrastructure layer, something in your application layer — again, that triage we mentioned previously. Is there something you need to follow up on as a fast follow? Do you need to go back and rework something, re-factor something?

Are there some things that were brought to light that start to take your focus in a different direction? You can say, “Okay, let me look at my application or my infrastructure from a different direction here.” A lot of times developers and engineers would be coding for sunny scenarios, when they’re pressured to get business functionality up and out. Taking that step back allows you to reevaluate, to put on a different pair of glasses and say, “What does my application look like? What does my infrastructure end up looking like?” Building on these after-action tickets after the dust settles.

Finally, what I would really strongly recommend is everyone give yourselves a strong customer success message and team. The team that’s sitting there and dealing directly with the customer or end user, that message needs to be clear, concise, and to the point. Obviously you want to control that message. You want to make sure the end user understands that there was an issue. It’s being resolved or has been resolved. This is what we want to do to make it up to you. Thank you for being such a strong user for us. We want to continue your business. This is what we’re doing to ensure this doesn’t happen again.

You want to control that message. You want to make sure your customer success team is out there on those front lines and doing whatever they need to do to calm the nerves.

Question: To prepare for cyberattacks, what tools or strategies do you recommend in pen testing your own systems? That’s penetration testing, where you’re basically attacking yourself to test your systems.

Haghighi: As far as strategies, as I mentioned, first of all you have your resources and then you have your staff developers, network engineers, operational engineers, systems engineers, and the team you have around them. Everyone around your company, everyone in your IT team, they should know the game architecture. Before launching any game, any service, you have to know the data flow. You have to come up with systems where you know exactly — if not A, you have to talk with B. If, under normal throughput, A talks with B, but you have more than normal throughput, that’s an alarm. You have to think about that. There could be an issue between A and B.

You have to come up with methods of testing. We’re doing tons of penetration tests, and not limited to our websites. We’re doing port scanning. We’re doing protocol scanning. We’re using hacking tools provided by a vendor we’ve been very satisfied with for testing features. We’ve done consulting. We’re participating in security events, seminars. Get as much information as you can that you can use in yoru testing area.

Safarian: One thing I would recommend right off the bat is building yourself some synthetic testing tools. You can do in-house stuff, but there’s definitely other options out there, plenty of software that we could mention. You can do simple things like CloudWatch in the AWS stack, stuff like New Relic, which is essentially an agent that’s going to sit on top of your entire application architecture and be continually monitoring. That’s on the web server side of things.

On the front end, mobile side of things, you have things like Crashlytics Fabric and so on. These things that are tuned to identify any kind of poor performance. On the back end, ELK stack, Logstash, Kibana — set yourself up so you can have the detail monitoring. You’re not going to be able to cover everything, but you can prepare yourself.

Let me go back to synthetic testing again. Set yourself up with some simple tests, some simple scripts. Unit testing. When you’re about to publish a feature, make sure you run it through your unit tests. Make sure you follow the best practices and standards. You have a service that’s sitting there and synthetically pinging your system for common, known use cases to get through your normal regression.

If you have a solid QA team, you can help train them. It’s really a learning and growing process here. We have a great QA staff, and they’re constantly reimagining their process and their regression. They’re trying to say, “Okay, here’s my script for all the sunny scenarios. Here’s my script for the bad actor scenarios.” Things we’ve taken from the past and the IQ we have from previous attacks, and then we say, “Okay, let’s roll these into our regression testing.”

These systems are all big machines. Sometimes any change you make will affect something down the line. It’s more like chemistry than physics. If there’s some kind of implementation you’ve built in system A, there might be some effect or exposure in system B. Really building that out as a growing unit — whenever you build out any kind of feature or version of your framework and your system, as it goes out just have that diligent practice to regroup and say, “Hey, what are the Q tests that surround this? What are the things we need to do? What are my scripts that I need to create or run?”

Build that strong relationship with your QA team. They are the shield for you. These are the guys that are going to sit there and do the dirty work that some developers — dirty secret — actually don’t do. Work together with them. Get these things out there. Use the tools that are available. Again, New Relic has a cool synthetic testing piece. I mentioned Crashlytics Fabric. Having an ELK stack set up on your back so you can keep that — it’s all about that instrumentation.

On your system, from an architecture standpoint, modular design. Sit down where you can limit the effects. Any time you make one change, make sure that doesn’t affect a lot of other things in that modular design pattern.

Question: I’ve seen a lot of news about ransomware. Is that something I should be concerned about? Should I consider working with any firms or vendors ?

Safarian: With ransomware, again, these are some of the questions you need to ask yourself. I go back to the audience poll, the “concerned, but not worried” people. I wonder what their take is on that. You need to evaluate what’s critical in your system, what’s available there that could affect your end user, and what’s there that could bring your end-of-day revenue to a halt. You need to prepare yourself for a lot of these things.

Answer these questions at an executive level first. Try to realize, what is the effect on the business for any of these types of attacks? What are some of the safety nets and guardrails we can put in place to avoid this. If any of this information does get out, if my system has been completely compromised, what are the things we can do? If you bring in any talented dev ops engineer, he’s going to help you put your stack into a different environment right away. “Let’s get out of here. Let’s get off this and move somewhere else.” Or you’ll be able to throw your database layer pretty deep to where you can put more security around it and avoid some of these things.

Again, best practices. I would preach that until I’m blue in the face. There’s a lot of white paper out there, a lot of great articles out there. If you’re on specific stacks, like AWS, they have a lot of — I don’t want to call their documentation amazing. It’s not the prettiest. But it’s very in-depth and very robust. You can get in there, roll up your sleeves, do a little research, do a little digging. This is your hard work that you’re putting out there into the universe.

GamesBeat: The thing about ransomware that’s interesting that it’s skyrocketed in the past year. The problem before that picking up a ransom was very hard. With the arrival of things like Bitcoin it’s become a lot harder to catch that person.

Haghighi: You guys have covered everything, but again, ransomware is a type of malicious software. It’s built around an encryption component. The only thing we can recommend is that you have to update your antivirus, update your antimalware, your Windows firewalls and patches, or Linux patches. It doesn’t matter which operating system you’re using. Make sure to never, ever trust any information you’re getting from third parties or emails. It’s easy to attach any kind of malware, even in a simple JPEG file.