GamesBeat: When you call it the “music reality engine,” what is that, really — the technical core of that?
Burba: It’s an engine, by which I mean it’s a script that runs continuously, every X amount of frames. It’s an engine we built out of Pure Data. That project started with a primary electrical engineer, back when we were doing hardware. When we stopped doing hardware, we had to figure out something else to do with that.
That engineer’s other related passion was music. He always wanted to build a music engine, and he wanted to use Pure Data, because it mimics the signal flow style of how you do electrical engineering, these continuous very low-level electrical processes in Pure Data. The project created a software synthesizer in Pure Data that fits inside of Electronauts.
Above: HTC Vive Pro headset.
He started off writing these Pure Data scripts. Pure Data is a visual scripting language, similar to Unreal, where you just connect different pieces. It’s also similar to Max/MSP, which is used by a lot of DJs. He started building a system where, primarily, you could play simple quantized notes, quantized .WAV file samples. That was inspired by Plink, which was a quantized music game, a little web app made by an advertising company that was stuck in my head for a year. I was obsessed with it, because I could see the possibilities.
We started making that, a simple little quantized sampler. It was outside of VR. You just played it with a game controller. We made that quantized sampler in Pure Data, and that by itself was incredibly fun to play with. We kept building pieces on to that, adding more tracks, adding more instruments, letting you turn tracks on and off, the vocal tracks. We built a synthesizer and added that as well. It’s the ability to play all of these different things in one song package, play them in real time, and play them procedurally in a way that a modern computer can run it. There’s enough variance that you can actually make a real song and put in there. You have your vocals and percussion and all the elements that a song typically has.
That ability to play all the different pieces in real time with incredibly low delay — you have to be able to touch an orb in VR and then the sound has to happen virtually immediately. We optimize the delay, but that’s basically what the engine is. We also have a way to compile C libraries into Pure Data, something experimental. So we have this engine we’re running, and that sits separately from the game itself. Our game connects to it through what we call Open Sound Control. It processes inputs and outputs through the music reality to play all of the sounds. Because it sits separately, we could make a mobile app with it or hook it up to Unreal — it’s an extensible engine that could be used in many ways. It’s just an engine that lets you play components of a song in real time if you have the stems for that song.
The engine also controls the music visualizer, which is a bunch of different visual events that are synced up to different musical events — when the drum happens, when the clap happens. That goes back to the engine to say, “Animate now.” It’s also a visualization driver.
GamesBeat: What’s the main gameplay mechanic?
Burba: It’s pretty simple and open-ended. You go in, choose a song, and have fun making music. We made the gameplay a bit more directed for the arcade version. The arcade version plays the backing track for six tracks, running through them one at a time. When it goes through the entire song, which takes a few minutes, then the select pops up, more arcade style, and you choose another song and go into that one.
Beyond that, the challenge of the game is just to be the best DJ and make music that sounds really good. We wanted to keep it fairly open-ended, not as game-ey, so we could encourage people to make music for music’s sake. It’s fun. People get lost in there for 30 or 40 minutes at a time, just jamming for themselves.
GamesBeat: Are you recording that music as well? Can you share it with other people?
Burba: We wanted to do that, originally. As we started working with the record labels — there’s a legal side of this that’s very compelling and innovative. We can get all these big record labels and music producers to work together. They want to send us their stems, which are like their babies. But one of the stipulations that we agreed to was to not allow recording functionality, because they don’t want people pirating their songs through this mechanism.
So from the standpoint of the features in the game, there’s not a recording functionality. That said, there’s always what we call the analog loophole. We can’t stop anyone from Twitch-streaming this, or recording it with many kinds of recording software. You could record it and share. What we want to do eventually with the game is have .MOD support, so people can take their own songs that they’ve made and put them in the game. They could basically use the game as a DJ tool and perform their own music. In fact, we DJ’d the Unity party at GDC with this game, and our E3 party as well. We still don’t have that yet, though. Honestly, we ran out of time to get it in for shipping.
Above: Electronauts turns you into a music creator.
GamesBeat: Within the game, can you play back something you just performed?
Burba: What we do have available is the arrangement system. The music itself is broken down into loops. You can make little chunks of a song, where each chunk has a different backing track and an instrument sequence it plays using a certain set of stems. You can set up to 40 of those and make an entire arrangement for a song. You could make an arrangement that lasts up to about an hour, and that’s your custom mix of the song. That can play by itself. You can also play along with that. It’s a full song creation kit in the game.
GamesBeat: What drove you guys to design it this way, with the different components you’ve talked about?
Burba: It started with the quantized instruments, the ability to play the instruments. Then we saw that playing an instrument without a track in the background kind of sucked, so we added the backing track system. It started off very basic. But as we were experimenting — we put homemade music in there to test things out, but really, what piqued our interest was taking stems from pre-existing songs and seeing what they sounded like.
One of the earliest ones — Trent Reznor released the full masters, the full stems, for two Nine Inch Nails albums. I think it was The Slip and Ghosts. He wanted people to remix those songs, and then he put together a remix album that came out afterward with some of the best fan remixes. No one had ever done this before. So these masters are just sitting there online, incredibly high quality, and the songs are great. We took some of the songs from Ghosts to test them out, and the result we got was incredible. It opened our eyes to understanding the other interfaces we needed to make.
Taking popular songs in general, looking at all the songs that are out there, pop songs and rap songs and EDM — we looked at all that and said, “What other interfaces do we need to represent a full song properly and still make this user-friendly?” That’s where the vocal tool came in. That’s where what we call the “la la looper” came in. We needed this looper tool, because otherwise this certain design pattern that’s common in songs, we couldn’t replicate it in Electronauts.
We built all of those pieces, and that allowed us to have the variance we needed. Now, when we go to artists to put their songs in the game, we can faithfully represent them and still have that dynamic interface. The songs guided us in the interface design process.
