PlayStation 5 lead system architect Mark Cerny revealed more details on the Tempest Engine for making sound appear to come from multiple directions in Sony’s upcoming game console.
In a video presentation today, Cerny took a deep dive into the technical designs for the console, which is expect to debut in late 2020. With more processing power, Sony is investing heavily in 3D audio via a dedicated Tempest Engine with custom designs to make sound appear more realistic and situational.
The challenge is that players can use many different ways to listen to their games today, including a variety of headphones, TV speakers, or surround sound systems with up to seven sound speakers. Sony has optimized for headphones, and it is working on the TV and surround sound systems still.
With the PlayStation 4, audio typically gets a very small chunk of available processing power, or even less than the SPU (synergistic processor unit) cores of the Sony PlayStation 3’s Cell processor. By comparison, the PSVR virtual reality headset had 50 different sound sources available. The Tempest Engine has enough processing ability (with a kind of processing called single instruction, multiple data, or SIMD) as all of the PS4’s eight-core processing capability combined — just for 3D audio.
June 5th: The AI Audit in NYC
Join us next week in NYC to engage with top executive leaders, delving into strategies for auditing AI models to ensure fairness, optimal performance, and ethical compliance across diverse organizations. Secure your attendance for this exclusive invite-only event.
But the PS5’s new Tempest Engine will support hundreds of sound sources, meaning it will appear that sound could be coming from hundreds of different directions as you hear it in a game. When you have something like raindrops, you’ll hear them coming from many different directions. That will help you feel a sense of presence, as if you were transported to some other world.
Above: An example of how one person hears sound, known as HRTF.
You can also more precisely track locality, or where objects are located. If an enemy is sneaking up on you, you’ll be better able to tell that. Such sound effects are common in games today, but they are often muddled. I remember playing a round of Fortnite and hearing footsteps in a house. But I didn’t know where those sounds came from, and I wound up getting ambushed as I searched the house.
One of the big challenges is that we all hear sound differently. Our ear canals are all different, and so the path a sound takes into our ear drums is different.
Audio researchers have referred to how we hear as a combination of sound frequency and direction, which they can measure in something called head-related transfer function, or HRTF. You can see an example of that in the image above. Each person has a different HRTF, which you can measure in a special lab. Some people may not be able to discern the spatial audio at all.
Sony can’t do that, so it has to come up with a default HRTF and see how it can be modified for each person.
“We wanted 3D audio for all,” Cerny said.
Above: Mark Cerny is the lead system architect for PlayStation 5.
And rather than license existing technologies such as Dolby Atmos, Sony decided to do the work on 3D audio on its own. But it created the Tempest Engine as a custom processing resource in the PS5 to make sure that 3D audio got its fair share of overall processing in the system. That’s expensive, but Sony hopes that bet will pay off.
“There are a lot of variations, so it’s nice to have computational resources of the Tempest Engine,” he said.
Using HRTF in games is complicated, as sound may appear to come from one direction, or from a general area in real life. You can think of the sound of a footstep versus an explosion from far away.
To deal with the variety of sound devices, such as TVs and surround sound speakers, Sony is working on something it calls Virtual Surround Sound, which gives you a sweet spot on your couch for listening to the sounds. The team is trying to improve it so that the area where you can sit is wider, much how engineers tried to enable better sweet spots for 3D viewing on TVs some years ago.
But Cerny said there’s a big different between the sound of the PS5 and the older PS4.
“I listened, and wow, I could feel a difference. The improvement is obvious,” he said. “It seemed like sound was coming from the real world, rather than the game.”
Cerny said it would take a few years to get to great 3D audio realism for a lot of games.
“This is a journey,” he said.
