The founders at Survios are true believers in virtual reality, and they’ve poured a lot of effort into the hit VR games Raw Data and Sprint Vector. Now they’re switching from games to something more like an immersive music experience with Electronauts.
The VR app will enable music fans to remix, compose, and perform their own music, riffing on works by artists such as The Chainsmokers, Odesza, Steve Aoki & Boehm, and many other bands that I am intimately familiar with (not).
At the core of Electronauts is the Music Reality Engine, which lets anyone perform and produce their own versions of the hits. It doesn’t skip a beat thanks to a technology called quantization. I spoke with Nathan Burba, president and cofounder of Survios in Los Angeles, about the new technology and the creativity that it brings to VR music.
“You can take a song by the Chainsmokers, ‘Roses,’ and determine when the different elements in the song will play,” Burba said. “It’s like you are playing inside a song.”
He said you get a sensation of playing the song at the right tempo, thanks to the quantization and a little mind trick that helps you deal with latencies in music headsets.
Electronauts debuts on August 7. It will be available on Steam and Oculus Home for HTC Vive and Oculus Rift at $20, and PlayStation Store for PS VR at $18. It’s also launching in VR arcades across 38 countries worldwide.
Here’s an edited transcript of our conversation.
Above: Nathan Burba is CEO of Survios.
GamesBeat: You were excited about the technology behind Electronauts. What makes it work? Can you explain that?
Nathan Burba: The project started with us creating quantized instruments. What that means is that you can perform an action in VR, and with a certain amount of latency that’s added, we then play a sound. The reason we do that is because we can make the sound happen at the correct tempo. That way you don’t have to worry about the tempo yourself. You’re not pounding your hands in perfect time like a drummer. You just perform your actions in the game and it sounds like you’re in time.
The best way to describe it is, it’s similar to the trick of how the HMDs work themselves. Some of the people working on Electronauts are former hardware engineers that also worked on our hardware back in the day. The latency trick we’re doing fools the brain into thinking it’s played a note at the right tempo, even though it hasn’t, because we delay it slightly. But that delay isn’t long enough for your brain to pick up on it. Certain tempos allow for that delay to be under 30 milliseconds, and with that timing, your brain says, “Did that happen in time? Sure, why not?” It plays along.
That creates the sensation of playing music in the right tempo, even you aren’t necessarily. It’s an amazing experience, to think you’re making this music. That’s at the core of the quantized instruments, in addition to the fact that they’re always in the right key. They always sound good, no matter what’s going on with the rest of the audio sync. Then we started layering other pieces on top of it, playing a bunch of samples in real time.
The way the game works, we request the stems for a song from a musician. These stems are the entire songs they’ve constructed, and then what we do is we put those into the game and allow you to play around with them through various interfaces, as much as you want. You can completely change the mix of the song at any given time. You can take out the vocals and play a solo, or bring everything down to a simpler version of the track. You can build up to a drop. All those kinds of arranging that a composer normally does — you can rearrange a song at will. You can take a song that never even had a drop and add one that sounds correct and relevant to the song, even though it was never there to begin with. You can play with all these pieces like they’re Legos.
It’s a revolutionary technology that has applications in a lot of places outside VR, but it synced up really well with VR, and we’re a VR company. It ended up working out well in Electronauts.
GamesBeat: If you’re mixing your own sounds that you’re creating into sounds that you’re hearing — are you making a song together with the game?
Burba: There are sounds that are taken from the song itself. Let’s say we take a popular song, like “Roses” by the Chainsmokers. That song has various elements. It has vocals. It has guitars playing in different keys. It has a drum kit. We’re letting the user determine when those different elements play, or whether they play or don’t play.
Some of those are turned on and off in real time and some of them aren’t. It depends on the kind of instrument. With the vocals, you queue them up and they play on time, and you can kind of remix them. With a guitar, you can play the actual notes yourself and sequence those notes to create melodies. It’s a big sonic playground. You play inside of the song.
It’s different from remixing a song using, say, a typical DJ turntable, because that’s designed to mix two different songs, alternating between one or the other, and all you have is the final mix. It’s like a cake. We’re not just giving you cake and frosting. We’re giving you the eggs, milk, and flour, everything, before it’s all put together. You get to decide how much or how little of those elements it has, and when it has those elements.
It’s the first time musicians have ever worked with a company with the raw stems, as they’re called, to allow this to occur. They did this a bit in Rock Band and Guitar Hero, but it wasn’t quite to the depth that we’re seeing with this project.
GamesBeat: You can be inside VR and then touch something with your hands, then, to turn on or turn off something like the vocals?
Burba: It’s a combination of interfaces that you touch — or bang on, so to speak — and interfaces where you turn buttons on and off. There’s what we call the orb kit. You always have two drumsticks, one in each hand, and you bang on the orbs. The orbs themselves are designed to be incredibly juicy, like how it feels to play a real drum. There’s a few instruments like that. There’s a laser harp. There’s a pinball, a kind of electric ball you can play around with.
Then there’s various buttons you can turn on and off. There’s a backing track, which has six different tracks. Usually they’re part of how the song itself is arranged. There’s an intro, a build, a drop, all the way to an outro. You play those in order. Then there are stems that are pieces of those tracks, and you can turn those on and off any time you want. You can make the entire track silent if you want.
A good example of why this is useful — have you played a game with the sub packs, the vibrating backpacks you can buy to go with VR games?
GamesBeat: Not really, no.
Burba: Imagine just having a giant rumble pack on your back. It adds a nice element to a game. But with Electronauts, because you’re in full control of the audio, you can control exactly how that thing massages your back. You can turn the bass drum on and start doing these giant pulses in certain parts of your back.
It gives you full control over the audio spectrum in a very easy way that’s intuitive for anyone. A kid can go in and immediately know what they’re doing. It’s that level of control, but then everything sounds really good. It’s a way to democratize music, so anyone can feel like a musician or DJ without needing any equipment or experience.
Above: Electronauts lets you mix your own songs.
GamesBeat: Getting back to some of the core innovation there, why do people care about the tempo matching in real time?
Burba: There are two primary difficulties with music. There’s what note you play, and then what time you play it. People who are really good at music know how to do both of those really well. You can put them down at a piano and they’ll start playing something that sounds good.
You remember The Jerk, with Steve Martin? In the first 10 minutes you see him with his whole adoptive family, and he has terrible rhythm. He’s trying to beat time with the music that everyone else is playing and he can’t quite do it. He can never be on time. This is designed for people like that. If you try to drum or sing or just — a lot of people out there just have awful rhythm. They have a hard time staying in time with a song, but it’s frustrating. They might love music, but they just can’t get the hang of that.
This essentially takes that away. You’re always in time, with certain restrictions. You can’t play everything, but within that window of possibilities, you’ll feel like you have rhythm. And then the other side of it is us choosing the right samples from the song, being very cognizant of what parts of the audio spectrum are being used, so that everything you play, the notes that you hit, they’re always in the right key. We’ll have seven notes that all sound good, and you can play them in any order or at any speed. They’ll still sound good to anyone’s ear. That takes away the need for you to know which notes are the correct notes and which are not.
Imagine sitting down at a piano and you can just immediately start playing. That’s what the music reality engine does. It fools your brain into thinking that you’re a great musician, just like the VR headset itself fools you into thinking that you’re somewhere else.
