When Phoenix Labs launched the co-op monster-hunting game Dauntless, it wasn’t alone. It had the virtually unlimited server capacity of Google Cloud behind it.
That was important because Dauntless was a free-to-play game, one that invited any and all gamers around the world, on game platforms as diverse as the Windows PC, PlayStation 4, and Xbox One. When it went live in May, it had to accommodate all players. And when it went live on the Epic Games Store on May 21, it scored four million players in a week. Now it has scaled to more than 16 million.
In an interview with GamesBeat, Phoenix Labs vice president of technology Simon Beaumont and Sunil Rayan, managing director of Google Cloud for games, said that the key to scaling it up to millions of players in a matter of days was containerization, or standardizing the game so it could run on any servers using Google’s Kubernetes engine. Kubernetes allows Phoenix Labs to encapsulate what is supposed to happen in a game when four players download the game and play it together over online connections. And that four-player experience is then repeated over and over across millions of players, all of whom have a unique gaming experience.
As the demand spikes hit and Phoenix Labs was totally surprised at the influx of players, the cloud did its job. The role-playing game wasn’t a lightweight title. It was a triple-A quality game, with good graphics that immersed players in the fantasy of hunting down beasts known as Behemoths in small groups. With each beast killed, the players earned new weapons and armor. And that kept them coming back.
June 5th: The AI Audit in NYC
Join us next week in NYC to engage with top executive leaders, delving into strategies for auditing AI models to ensure fairness, optimal performance, and ethical compliance across diverse organizations. Secure your attendance for this exclusive invite-only event.
Containerization
Above: Styxians in Dauntless.
This kind of game is increasingly common, but each job is demanding, Rayan said. The game developers offload the multiplayer server operations to Google and its vast data centers, and the developers concentrate on making the game fun and as bug-free as possible.
“We had plenty of things that we wanted to develop and add” to the game, which is a kind of living experience that can be changed and improved day by day, Beaumont said.
Both Beaumont and Rayan said it is impossible to predict if a game will go viral. But the teams had to plan for a number of scenarios without knowing exactly how many players would show up at any given time. They had to guess at the server capacity required, and the underlying cloud-based service had to work in an uninterrupted way, handling the real-time combat. If the game hiccuped, players would know because they would fail to get into a match or face a slowdown in gameplay — something that could easily get them killed and very angry.
To stay agile, Phoenix Labs runs all its game servers in containers on the Google Cloud Platform (GCP). The studio has a custom Google Kubernetes Engine (GKE) cluster in each region where Dauntless is available, across five continents (North America, Australia, Europe and Asia).
When a player loads the game, Dauntless matches him or her with up to three other players, forming a virtual team that is taken to a neighboring island to hunt a Behemoth monster together. Each “group hunt” runs on an ephemeral pod on GKE, lasting for about 15 minutes before the players complete their assignment and return to their headquarters, Ramsgate, to polish their weapons and prepare for the next battle.
“We partnered with Google well before the launch and they’ve been on this journey from the start,” Beaumont said. “We used Kubernetes as a managed service, for Oauth, and it worked out very well. We had a very small team with a few platform engineers. And they allowed us to scale. It turned out really well.”
Scaling up
Above: Dauntless has come to the Switch.
Four million players in a week is a lot, even for Google. Phoenix Labs, meanwhile, had to closely monitor Reddit and Twitter feedback from players.
The team had the benefit of working out the bugs ahead of the launch, through closed beta tests and open betas, but the mission was always to get “one Dauntless,” so players could play with their friends, wherever they were, Beaumont said.
The company’s lean team worked side by side with Google Cloud Professional Services to execute over 1,700 deployments to its production platform during the week of the launch alone.
“Our customer engineers were in the same room with all the Phoenix Labs engineers during the launch, making sure that they paid attention because, as you know, they far exceeded their expectations,” Rayan said. “We know that the launch is super important and we had our customer engineers in the same war rooms.”
Beaumont said the teams got together before the launch and hashed out a plan and looked over the design. They had to make sure the game software was ready. The Google engineers knew the technology and weren’t coming into the process blind, Beaumont said.
It wasn’t all easy. Some things that were lower priority came “back to bite us,” Beaumont said.
On December 10, Phoenix Labs flipped the switch on the Nintendo Switch hybrid game console, and it invited even more players to join in the fray.
Above: Ramsgate hub in Dauntless
“It was very exciting for us,” Beaumont said. “It’s a great place to be, but you’re going to have bumps along the road. Issues are going to arise. It was great to be working with a team that understood the infrastructure and helped us through the issues we saw. We were pushing Kubernetes in a ways Google hadn’t seen before.”
Expectations got bigger and the teams had to set the bar higher, Rayan said.
“This is at the upper end of what we’ll see for a successful game,” Rayan said.
And in its quest to go worldwide, Phoenix Labs is preparing to expand to new regions such as Russia and Poland (they recently launched in Japan) and take advantage of other capabilities across Google.
Normally, such complicated games are patched together “closer to the metal,” rather than with Kubernetes containerization, Rayan said.
“They pushed the envelope on containerizing all the elements. They have a cloud-native game that leverages the power of the network,” he said. “They pushed us as a service to what Kubernetes could do. Generally, you will try to matchmake players from the same region for game servers. They truly have built a game that can span boundaries.”
For example, by leveraging Google Ads and YouTube as part of its digital strategy for Dauntless, 5 million new gamers were onboarded in the first week of launch; using YouTube Masthead ads also increased exposure to its audience. Phoenix Labs has migrated to Google Cloud’s data warehouse BigQuery for its ease of use and speed, returning queries in seconds based on trillions of rows of data. They’re also beginning to use the Google Sheets data connector for BigQuery to simplify reporting and ensure every decision is data informed.
There are other choices, such as Amazon Web Services, and there are sub-services from other vendors that can be integrated into the Google solution. But Beaumont said the close interaction with the Google engineers was key.
“There were more people at Google wanting to help than we had problems to solve,” Beaumont said. “When you are looking for a cloud provider, look more than what’s on paper. That is part of your decision. Make sure they can support you in whatever journey you are going on.”
