Visual Language of Electronauts: An Immersive Sound Based VR Experience

[Music] so my name is Justin Corey I've been working at Serbia's for about four years as a lead artist joined the company in 2014 worked on raw data and then once we wrapped that up moved to this awesome music experience called electrons so electro knots is not a game it's not a tool it's an experience it's a music creation experience and it's one of those things that you can't quite put your finger on it what you're going to do with it because it's just it's the art is driven by the engineering and okay quick shout out to raw data first vr game to generate 1 million in less than a month and the first viewer exclusive game the top number one on steams global sales charts and we just released Graham Factor recently where we patented the fluid locomotion technology and again top number one on Steam and

oculus so let's talk about the visual language of Elektra knots so what you'll see here is our electro knot he's pretty funky he's pretty awesome he's an astronaut who plays music and let's go over the story electro Universal in the future humanity harnesses the power of the harmonic wavelength known as the electro harmonix RIT while rippling through the depths of the universe a great disruption in the harmonic fields causes the electro nuts become separated lost in four different quadrants of space cosmic technology alien and psychic you must groove through each zone and use your melodic resonance fears to make the universe dance and unite it's time to get the band back together and ok first first term one of these terms was used often when describing electro nonce and something that we adhere to which is synesthesia it's a perceptual phenomenon in which the stimulation of one sensory or cognitive pathway leads to automatic and

voluntary experiences in a second sensory or cognitive pathway best way to describe this is you hear music you dance it's one of these things that you just can't control and uh yeah it plays a big part in the game here and when I first heard this term it was by a man named Ted Simmons Iguchi if you guys work in games and you don't know his name for shame she's been around for quite a while worked at Sega responsible for the Sega Rally Series res and a space channel 5 so he says the majority of music games fall into the category of rhythm based or time-based Mizoguchi says that's not where my interest is rather it's more about how the layering of music with visuals changes the world that you're in and that becomes an experience in and of itself so very cool it's not just about the game he's more interested in just getting your attention so let's talk about the music reality engine this is something that Serbia's developed it's architect core technology electronics ok

so you might say what is the music reality engine it's essentially technology that allows a person to be a performer even if they are not the most musically inclined it's a beautiful piece of technology because he's for me you know I've tried to play instruments all my life and never fully succeeded and then you know admit and I touched this this game or this experience it just changes everything and you can have fun with it you get lost in it forever so moving on to our inspirations and again Tetsuya Mizuguchi space channel 5 was again one of the first games that we discussed when we talked about kind of the fun nature and the language of the game um it's one of those things that as you play the experience grows that the environment moves more of the color

shift more it really rewards you for performing well and it's just something that we kind of inherently built into the electro naughts game yeah I'm also a very big influence on our game the colors the strobing patterns the embodiment of nature and science and math lots of geometric shapes lots of movement you'll see these heavily used in our visualizers space another big inspiration especially in terms of color palette you'll see a lot of these nebulas share very similar in almost cosmic light colors that inherently just kind of happened to be included in our game and yeah it's just such a beautiful thing to look at and really just again you'll see it everywhere when you're just crooning through the the space so let's move on to the prodigy one progerin is essentially the hub it's the

place where you start any song that you're about to play it's it's essentially a radio station in space you're broadcasting yourself out and you see this tunnel you get launched through it every time you go to your next track so this concept was originally done by Brian Higgins a great guy very very awesome and talented person so when I joined the company or I mean the project more like it there wasn't really a very clear direction in terms of where things were going to go we were going very hard sci-fi and while it's cool didn't exactly reflect the kind of experience we wanted to bring so really without trying to shake up things too much or change things too much I kind of tried to make a few quick adjustments to the visual style of the game and it's really just about bringing the color palette to something that matched the mood so we kind of moved into this direction incorporating more music into the

environment turning the look of the sci-fi to something that you find more in a like you know audio equipment so looking at the materials that you'd find in audio equipment was like a big inspiration for the kinds of shapes that we would use for the environment you can see at the bottom right corner that is your launch tunnel and we wanted to decorate to look like a stage like if you're performing at a concert for me to think that was the biggest selling point is that because people are about to go off into these zones you want to feel like they're about to you know enter a stage so that was a big big selling point for me and these are the concepts again these are a company of some of the modeling block outs that I did in in conjunction with paint overs that brinded so here is a the in-game shot of a very epic sequence where you're flying from space you enter into this ship through a large tunnel and then you land to this hangar which is there we'll launch out to your next song

so this is the prodigy one interior in real time and here's our launch tunnel in real time this environment was very very intimidating to build the scale of it is just absolutely massive when you see a character inside of this environment in relation to everything else it's just it's dwarfing and the trick is to find the right balance of scale and the balance of detail and also get it done on time which is not much time to do but you'll see we've got you know a lot of the colors that the nebula is represented all represented here I'm very unified and when you're in there everything's just bumping musics going speakers are moving lights are strobing it's very very awesome so here's the project 1 exterior so remember when I told you we fly into this ship well this is the ship that you fly into again another awesome concept by Brian Higgins so my my goal here was to try and recreate this as best as

possible ball so making it perform it not the easiest task so let's look at the literal translation of this into 3d so here's my take on it this is where we move we move away from concept and we start getting some details in there and this is the polished piece that is now the proud g1 the railgun on the front is kind of like really powerful with an image to look at because it just makes you feel you know you're just getting shot straight out of there it's very clear it's very direct it's it's just incredibly interesting shapes and again attaching speakers to everything was just a big motif wanted to beat your head beat you in the head with it here's a back shot so you know one of my difficulties yes I'll get concept but then I don't necessarily know exactly all the details that need to go into it right so again trying to look at references of audio equipment just general turntables everything that I could get my hands on I start

incorporating that into into the ship get a little closer here again and this your scale these things the biggest trick is balancing the small details to the large details and you'll see here to your right is the back end of the ship which needed to get probably the majority of the love because that's what you see the most and I'm very happy with the end result and you know I think ultimately really served its purpose and once you fly into that that back end it's pretty epic so moving on then we have the Waverider the Waverider is again this is what you're always flying through or flying inside of as you're going through your visualizers it's it's a very fun looking ship originally I wanted it to be like a Cadillac like a space Cadillac because it kind of like this motif that I've always thought about for quite a while especially when this kid when this game popped up but ultimately we kind of settled on something that's a bit more unified for the look of the entire environment but yeah still very cool got

equalizers speakers everywhere I think the the idea is that everything is propelled forward by sound audio waves and then hears shots at the Waverider in game pretty pretty good translation not you know not exactly ideal or identical but still very happy with the look of it but yeah yeah it's a it's a very cool place to be on in the inside as well unfortunate I don't have a shot of the actual interior of that but you'll hopefully get a chance to go down and play it today because it sounds on the on the show floor okay so let's talk about our Electra not so we had a wonderful cut character concept artist named Keegan he came on board middle of the project and was really just spending a lot of time just kind of fleshing out the design trying to take it a little little more so a little more fun direction just straight sci-fi and

astronaut and ultimately we ended up moving into this direction very very colorful very vibrant matches the environment matches the ships just very fun and wacky this character model was done by Dammam woods he's our lead lead character artist at the company he's a busy man he does a lot of character work for us and here's a look at some of our visualizers that we design these were done by Gabriel Valdivia gabe is also a tech artist at the company and very talented guy very creative guy really really really enjoyed the inspirations that he put into this game and also you can see in this image is a couple examples of our instruments the spheres you see in front of you are the harmonic spheres these are the initial instruments that you used to generate your audio very very intuitive very simple and very fun disco balls were a big big inspiration for these guys because initially the the

visual look of the game was not dictated by the art direction but more by the technology right and what the designers put in so it was all very very prototype but once you started using it you didn't want to change it you didn't want the arts then take over the experience so we had to kind of accommodate those things the things below you are your decks you'll see these these nice hovering or these nice holographic patterns coming from the bottom and these cubes that are inserted into each one are your different instruments and your different ways to interface with the whole experience on our Left we have our back tracks which you can cue up and set up and then we also have filter cubes to the right of that those gray cubes and then to the right of the center you'll see that there's a few other colored cubes those are your other instruments and there's some fun stuff there to play with them so another great visualizer by Gabriel I believe part of the tech zone

it also has one of our other instruments called the lava loops so this was also again a very very gray box kind of thing was just a cylinder with the cylinder on top of it and you know you don't really know what to do with something like that because again it's an interface and you don't want to change the way we work so I just said okay well you know why not make it a lava lamp or something like that because you know as a kid I remember having those in high school and you know they're kind of a part of the whole psychedelia they can be totally related and everybody recognizes these in some fashion so yeah that was there I think it turned out pretty well I like what they did with it and again once you see what the interaction is with these things and the music and the sounds that they produce it's pretty it's pretty awesome and then to my actually one of my most favorite is this synth orb right here it's kind of like one of those plasma globes that you get when you were a kid where you touch it with your finger and the electricity sparks to it

that was like one of the initial things as we were playing with it and they had a few iterations but of course it was like what when you touch it like kind of didn't know what you were getting out of it and once we turned it into this plasma plasma type of fact it became very clear how you interact with it it has all kinds of layers - it's a 3d it's a 3d interaction so you can move inside outside up and down on to create some very interesting noises and then we have a harp that's kind of like sick it's a very fun fun one to play with was also kind of a tough one for the designers to really to really believe in but I think they kind of figured out how the interaction should work because plucking something in VR is actually not that simple but they solved that pretty nicely let's see here's one more visualizer and then okay actually this is also cool so these tracks on our right we just recently incorporated audio I mean a voice work so before it was all just bad just instruments right

and then getting actual voices into the game was a tough challenge how do we get how do we get vocals to work so one of one of the most interesting one of the last things that kind of got into the game was this vocal track system and again please do try it when you get in there it's a ton of stuff to play with and lastly looks like we have our audio grenades so these are things that are kind of like one offs that you can throw into the air and just have fun with and they explode and make a cool noise and again I would say when it comes to the visualizers I can't speak for Gabe and his inspirations but fruit sure this one you see right here is one of my favorite and I don't know if you guys have ever played outrun but it's an old Sega arcade game essentially as you're playing and hitting the instruments you're going to see these kind of waves fly by that are like cars on a track and I think in general a lot

of the look of these visualizers were heavily inspired by the 80s in the synth wave which i think is just a very cool thing to see come back yeah the 80s the 80s had had a big part in the overall visual direction of this game kind of gaudy kind of Aegeus very colorful very neon but ultimately it really sells the idea of technology the future and then the type of music that we're all interested to which is electronic music and then that's pretty much it so thank thank you guys for letting me come out I really appreciate you taking the time to listen and I hope you guys get a chance to get on the show floor and play the game [Applause] how're we doing on time by the way plenty okay push ins yes like how much

how much time did it take to create each one um it really range depending on like from concept to execution I think anywhere between two to three weeks luckily a lot of the elements that make up those visualizers are relatively simple objects but you know it's more about how do you tie all that together and they're also very reactive to what you're doing so they're not just these things that are looping behind you but colors are shifting scales of things are changing this the rate of these things are all are all being affected by her performance thanks Dave Gable love you for saying that he worked very hard on these on these visualizers and another guy James Little John he's one of our VFX and tech artists a lot of the effects that you see all all the instrumentation every little detail that's not readily apparent is all done by him as well and those guys work very

closely together to help create that look alright so I had a quick question sure for like folks who naturally play instruments and stuff like that they obviously have a leg up but those who are new to it what kind of visual language did you use to kind of cue them as to when would be a good time to hit a crescendo or stay on the beat or did you guys problematically just I'm gonna push it to stay on beat I think at the end of the day like you know because of the music reality engine and the the what we're doing on under the hood no matter when they decide to hit that orb it's going to sound right on beat okay um as somebody who is more musically inclined than others you're going to have a much better time at performing right but for those that are just they've never really picked up an instrument they don't know how to dance or they don't want to keep a beat it sawed it you know it's going to make you sound as good as you can and it does pretty well I mean you can go pretty crazy on that stuff and it's just what ends up sounding really great but yeah for those that are inclined and and have some experience it's going to

only enhance that that much more I mean what I show you in those images does not really explain the amount of depth that that thing has it's it's it's just absolutely amazing how much you can actually do so there's seat you can queue up sequences you can you can have all kinds of modulation going on it's just it's it becomes just an experience and then becomes more of like a production environment if you want it to be so the layering is just absolutely crazy I remember just when it first started it was just the orbs right and it was just that one was enough to keep me going for a long time and I mean anybody who gets in there wins it playing for 45 minutes to an hour straight and you have to pull them out because they just won't leave it which is pretty much speaks for itself and how awesome it can be I think inherently everybody wants to be a performer everybody wants to be able to you know be the cool guy who's playing and DJing at the club and it really gives you that opportunity to do that I mean we've got you know awesome thing

we got to be at South by Southwest and we were performing on stage there and it was it was a great feeling to see that it was really proud for the guys and another thing to keep in mind is a lot of the audio tracks that you hear in the game are not just musicians but the engineers that made the game so these guys are you know they love music they really believe in music they take it very seriously even though it's a very fun and wacky game but they spend an insane amount of time building the technology and all I did was come in and kind of try to help make sense of what they bill and give it some some visuals to it right I certainly didn't join the project as an art director I'm just lead artist who is there to help get things done and help move things forward and since I did come in so late it was kind of like we just knew we had to enhance everything a bit more because I mean I always felt it could stand on its own to be perfectly honest I was just happy that I was allowed to come in and you know add some add some extra speckle to it you know thanks mm-hmm I you mentioned that the

engineering was the one that influenced the art direction yeah um was there a point during the development that it was the other way around like the art direction influenced how things would be made or the design of the game I would say absolutely no I mean when it comes to when it comes to something like this my job is just to enhance and help right I I have no interest in letting my art dictate how you interact with things and just how you were supposed to experience everything I just want to help drive what they've built and make it you know something pretty to look at and something fun to look at again if there was really nothing else but making the music I still think it would just be amazing so it's hard to say like the only the only major influence I had overall was just the this introduction sequence that many people have not gotten to see yet so thanks so visually

speaking I noticed that you changed the design of the the first area ships were to match the color scheme like how was this process of balancing like saying the visualizers color schemes with the overall color scheme of the game because I feel like it can get pretty distracting if you have like a bunch of like yellows and oranges with a bunch of like blues and purples well it's it's one of those things I mean if you go to these festivals or you go to these concerts right it's all about the distraction it's just it's just an eye festival it's high candy everywhere and part of that is just the culture of it right it's it's again psyckadeli it's all about trying to overload your sensory your sensors with whatever you can't beat the music and then the visuals clear from playing the game um I I certainly think of anything the player is always just focused on what's in front of them you know what I mean it's very rare that you're taking your eyes

away from your your instrument right everything else around you is there for the everybody else to observe and enjoy it's not necessarily for you as a performer so really I mean as people are playing their music you want these really bombastic visuals going on in the background you do want these strobing elements I mean it's just the nature of the electronic music scene be able to stay focused oh yeah oh yeah okay and even if you can't focus you're just like oh my god I can't do this like you just you're smacking the orbs and everything sounds great and you're having a blast and people around you are having a good time hello great talk um I'm curious about the iteration process is there is there a point where as the music comes in the visuals change or vice versa like is there like how does the synergy work between that has an iteration process but so the the visuals themselves like the visualizers are we're completely

dictated by you know Gabriel and the audio itself is completely separated from that experience right the only thing I can say is the way that they tie these things together is just through the technology of what they built in their engine and then how do we transition from one song to the next in one room to the next is also its own interesting experience we kind of they've done an amazing job of being able to merge the worlds together and blend it together you're not just loading in and then loading out and then loading in you actually feel like you're traversing and kind of just ripping through space and ripping into these zones and just the language of it all just was handled very nicely I mean it's kind of it's one of those things that's so abstract you can't necessarily describe these things right it's like the game is the game is a very very it's very out there it's just like everything about it is just it's you can't put it's hard to put words to I mean if you saw half of those visualizers how would you even begin to describe what you saw really right cool thanks you're welcome

amen how much time was spent on the concept in stage concept didn't ah that for me is impossible to say all I can say is when I joined cuz again I was finishing up raw data and as soon as I finished it I was moved on to this because it was like they needed somebody to help come and tie these things together and then just put polish down on it and they had already done I mean I could show you all even from the visualizers I mean I've dug through everything an insane amount of conceptualization went and from the instrumentation to the visualizations and then of course when we got to the ship itself I mean that was I think one of the toughest times so quite a bit of iteration I mean for me to collect all that stuff to display would have been quite a bit and I was already feeling I'm quite surprised how fast I kind of just talked through the whole thing but they're just an insane amount of stuff that they did so it's just if at some point we'll probably have these these things available online it would be great for us to collect all of our art and just kind of share with everybody because you would be shocked how much has changed it was a mostly 2d

painting type stuff for it I kept well it's a combination of the boat right it's a start with all the geometric patterns and all these really hard surface shapes these very precise elements have to be you know it's best to do them in 3d and then add everything on top of it thanks everybody cool all right well thank you very very very much [Applause]