Porting Your Title to Oculus Quest

[Music] i'm alex silken i'm a co-founder and cto at cervius and uh today i'll be talking about porting uh your vr game to the oculus quest in the context of creed so the key takeaways of this talk i'm going to talk about some kind of common pitfalls that you'll probably uh fall into like we did uh some recommended workflows uh for getting that thing squeezed onto the quest and um just some examples of the tricks we had to do to get it get it running on the quest and uh just a bit of an intro about the game um are these videos playing i'm not even are they actually playing okay um sorry i don't usually use max um so um it's a boxing game uh it's based on the creed movie uh it was originally launched in uh last fall on the pc and playstation platforms um

it was our third unreal legend game uh playstation was the lead sku uh for this for this game um this is because you really want to have your most constrained platform as your lead skew we learned this the hard way with raw data our first unreal engine title that came out on the playstation playstation vr came out while raw data was in early access so we weren't really planning to initially release it on the platform and we just learned the hard way that trying to downport an experience is just really really tough so i know this talk is titled porting your title to the quest but i guess my first advice to you is don't do it just develop for the quest from the beginning um and unfortunately we had to do this again for creed but that's because docu's quest came out almost a year after crete came out but it was such an amazing opportunity for us because we got the ship uh creed as a launch title for the quest so we just couldn't pass it um but uh it was it was not easy um so um

okay were the videos playing okay all right never mind um so you should really understand what you're developing for so let's uh let's talk about what what's what what a quest is and what's special about it what's different about it than developing for a traditional uh console pc so the quest architecture is a little different than what you're probably used to it has a tile based uh render also some called binning based renderer essentially uh pc and psvr use immediate mode rendering the entire screen is rendered together but these mobile gpus what they do is they divide up the screen into tiles the tiles are much smaller than what's demonstrated here on the screen this is just for demonstrated purposes and your draw calls are submitted as normal you're not submitting them on a per tile basis you're still submitting them normally for the entire screen but when the gpu goes through and divides up the tiles

this is called the binning process it takes all the triangles it sorts them and it figures out which which which tiles the triangles belong to um and then the gpu processes these tiles sequentially so unlike the once again traditional graphics cards that you're probably used to uh there's no vram there's no dedicated ram uh that the graphics card uses it's it's actually a shared memory between cpu and gpu um this does mean that the gpu doesn't does not require as much memory because it's going through these small tiles so it's much more efficient so it's great for battery life etc but it does this is an important consideration for what things are just inefficient on this on this platform so the rough target you should aim for um is three hundred thousand verts um for your entire scene characters etc uh creed shipped with around two hundred thousand um you should aim for around 100 to 150 draw calls creed shipped with around 70. we were

pretty aggressive with our environment optimization merging a lot of things down to get those dracos down and uh something that you can't really change is oculus quest renders at 72 frames per second that's the refresh rate of the screen that is roughly equal to 13.8 milliseconds for a frame this is how much this is the time you have for your gpu and your cpu all the fret threads to do all the work for the frame and submit it to the to the system to render and this is non-negotiable and you have to understand because uh the system is uh you know it's like vertically uh as with vsync you're really locked to this refresh rate so if you miss like i like to call it like a train if you miss this train you have to wait for the next train so you have to wait at the train station waiting for the next train to arrive so even if you if your frame took like uh 14 milliseconds uh 15 milliseconds you have to wait for the next time refresh rate uh so essentially if you keep missing that

budget by even a little bit now you're essentially each one of your frames is actually taking like 27 milliseconds you're just having your frame rate so you're essentially now your your game is only running at 36 fps which is pretty vomit inducing in vr so you have to be really careful and strict about these budgets um so the tools in the debug environment you have ovr metrics tool that's pretty useful uh for a heads-up display that gives you some stats particularly fps so if you're running a shipping build you don't have a lot of your tools available this is a nice handy way to just get some data when you're validating one of those finer shipping bills render dock is really vital not it will it will really let you inspect what is happening uh in your scene so if you use using unreal using a material editor like a graphical editor uh for materials it's a graphical tool

you you connect some notes together and then that outputs some shader code well you don't really you don't really know what it's outputting unless you really look at it and sometimes the results can surprise you and they can be less than efficient um and you can also look at your draw call costs so you have to be really mindful of the fact that unlike direct normal you know traditional architectures your draw calls are impacted by this gpu binning process because your draw calls don't immediately go and render they have to go through the bidding process which means all your drop hold costs are a little uh grow a little bit grow a little bit but you do get a relative cost through draw calls to understand what's expensive because not a drill costs are equal you have a budget of 100 150 but some draw calls are more expensive than other draw calls so it's just a rough target uh snapdragon is good for uh just seeing uh you know how all your threads are lined up so you can see if you have scheduling issues if you're stalling on

things then you have android studio so we really even with playstation we really advocate try to develop as much as possible on the pc the iteration time is so much faster you just click play and you're there you know with a quest you have to deploy to the target platform it's a long process um but every once in a while you do have to debunk on the target platform so most of the time you're running gameplay code if there's a bug in your gameplay code it's not really platform specific but every once in a while there's something so that shows up so we advocated used to use android studio over visual studio because with visual studio we saw that sometimes even stepping a line of code could take like 20 seconds it was just way slower and on the on the topic of you know using pc to iterate um so i'm sorry if uh everything is kind of unreal context because we're in real developers some of these things is still applicable to other engines but with unreal uh you can basically launch the engine with a these console parameters

that essentially emulate the hardware rendering capabilities of the quest or at least the rendering hardware interface so that you get a more accurate representation of how the scene will look on your pc of course not as good once again as testing on a target platform but really try to use as many uh tools as possible to test on the pc rather than the quest which might sound counter-intuitive but um yeah so we finally you know got the thing to compile and deployed a device and very disappointedly the things just crashed on startup so so that's not fun right so what's going on with us so we're running out of memory so most likely it's going to happen to you too if you're developing for a much uh you know a much more powerful platform with a lot of memory uh you're most likely gonna start running out of memory that's your first uh kind of uh obstacle that you have to jump over so quest only has 2.75 gigabytes of ram uh for your application so that's pretty small in comparison to like

you know desktops that have 16 gigabytes of ram nowadays easily uh so on unreal you can use a console command report full that will just give you a nice memory report of everything that's loaded uh in your scene all the textures etc static meshes assets you know game game object actor instances and for us textures were easily the thing that was blowing us way past the budget um so as a quick hack we just like uh globally clamped all the textures to a really small size just to let us launch the game uh on the device just to start playing around with it i mean you can just clamp it to like 32 by 32 everything will look like doom from the 90s but at least a launch and definitely use astc compression so this will shrink your texture sizes which is great obviously for once again managing the ram uh applications but it's also great for perf because now the gpu doesn't have to uh load you know you're constrained by

the bandwidth essentially loading the textures in the gpu so smaller textures will mean your gpu can do its work faster uh so after we got our mem report um you know we found that a lot of our assets were loaded that we didn't need to this was the result of us being kind of sloppy because we could get away within a playstation vr and we weren't quite hitting that memory limit so in unreal there's a console command that essentially allows you to get the reference paths to uh any asset so we essentially did a dive through found textures that were questionable like why is this ui texture loaded in the middle of the game that's normally from the main menu things things of that nature uh so for us um this is actually been still i mean to this day often a uh issue uh uh umg slash ui is the usual copyright of a lot of stuff being loaded that shouldn't be loaded um a lot of different umg widgets uh umg is the framework for ui and

unreal just referencing a bunch of things that it shouldn't reference um referencing characters or reference from different game modes etc so and ui textures also do not stream uh so that's an additional kind of ish you know gotcha in unreal so that really increases the problem um and then we add a lot of you know unnecessary hard references to assets so for example we'll have like a data table for a system that refers to say character assets or audio for some system for every single map or for every single character well in a in creed you're only ever in one ring and you're only ever fighting one character so it's instead of instead of having those be as hard references there's really soft references so dynamically loading them um and uh somewhat unreal specific but uh just in general you should try to avoid uh blueprint dependencies um blueprints are you know the essentially the archetypes templates of an actor um that is spawned

or like a prefab if you use unity so sometimes people don't realize that casting a blueprint class to another class is a hard reference because you essentially need to access its functions and properties et cetera um so it's better to use an interface potentially to break that up if you want to make a blueprint interface or just move the reference function or variable to c plus so you can just cast to a c plus plus base class uh because loading accessing the c plus plus class doesn't require you to load the blueprint class reference um so uh anyway so let's move on to uh optimization time so uh allegedly this is where the fun begins um so let's let's find the bottleneck neck you know with optimization uh you should really do it in a data driven uh manner figure out what's actually costing you uh you're you know wasting your cycles and target that uh and this is kind of potentially not a very scientific approach of trial and error uh so in unreal there's a stat unit command it's pretty pretty standard to

like get a your frame timing and see how much your threads are taking but it's a little useless because uh you know as i mentioned you're basically locked to 72 frames per second so your frame time is always going to be like 14 milliseconds or whatever and then your game thread is potentially going to be waiting on the draw thread to catch up so really you want to be trying to isolate what it is that's causing you to go over the budget so the trial and error here is if you think it's the cpu thread just pause and unpause the game that would just pause all your gameplay logic and if your frame rate suddenly stabilizes then you know that's your bottleneck if you think it's your fill rate so it's fill rate being uh the cost of shading every single pixel will just decrease the screen resolution to like 10 and if suddenly your frame rate stabilizes then you know that's your bottleneck shaders in unreal is a show materials console command which essentially just turns off all the shaders everything's just uh shaded with like a standard gray shader and uh draw calls

there's constant commands to to toggle specific objects like show static meshes show skeletal meshes once again if you if you just hide all your static meshes and your frame stabilizes you know that's the thing that's bringing you over your budget so once you kind of start isolating your general bottleneck that's when you want to start drilling into the stats sunreel has a pretty complex stat system essentially and you like type in stat space and the stat group name so stat rhi stat scene rendering those are standard those are some good examples of stats that give you an idea what's going on um stat rhi for example will tell you how many drop goals you're doing like how much memory you have loaded for your gpu stat scene rendering lets you further drill drill into like what's your uh render thread doing is doing so you might see that uh shadow or culling is taking a really long time so one thing you have to keep in mind is uh that you know this stat system is

really complex and unreal uh but it does which means it also really affects per perf uh it distorts and distorts the numbers it reports um i don't remember like the numbers on the quest but just on playstation just turning on any one of these stats just suddenly adds two milliseconds to your cpu which can easily just kick you over the uh your budget so now all of a sudden you're waiting for an extra frame your stats get distorted so that's kind of useless and on the other hand kind of hilariously the stat rendering is using this really antiquated uh ui rendering system uh called canvas so most of unreal now uses slate and those like each character is like adding like a drawcall or something so then once again you use this system and it's just distorting your numbers so as a kind of quick workaround of course it's better to use render dock if you really want to get do get have a deep dive through but it's convenient to be able to just have a glance at your stats so you can have qa running through all your maps

and kind of record any bottlenecks any any hot areas we created a lightweight slate widget um that has uh some stats that we we decided that are kind of useful uh that we show there um we there it's rendered using slate so it doesn't impact performance uh and it's composed on the screen using this debug draw service um it shows the average and max values so that's useful because like i caught our uh world tick time just the average was good but the the tick time was like 10 milliseconds higher and i realized like just from there like then i did a profile and i realized like every like few milliseconds we had something taking 10 milliseconds so we had these spikes causing uh irregular frame rate um and we have ioni defining the budget for text color coding so green is good yellow is probably danger zone red is definitely bad so it's convenient as well because we define different budgets for different platforms if you have a different frame rate or et cetera and uh so there's some existing cached

engine values that are already there so we can already read them ourselves we also modified the engine for some additional values specifically world tick time because like i said the game thread time that's reported by stat unit is kind of useless because that's affected by waiting on the rendering thread etc most of the time really interested in what your world take time is which is where all your gameplay logic is that's the thing you're going to be optimizing most of the time um so so cpu in unreal uh it's pretty standard we stat star files dab star files it's a console command you use session frontend so starting in 423 unreal insights is coming it's a whole new profiler it's way more powerful it's it also does profiling for memory loading etc so that's going to make your life way easier but um it's really standard across really the playstation etc what we found is moving components is usually one of the most expensive things especially if you have like these really deep attachment hierarchies

um so the simplest is just detach your hidden and disabled scene components like audio components effects components especially have like designers working with you sometimes they can be pretty sloppy and they attach all these things that aren't even being used right now and it's just costing you cycles i know it sounds kind of mundane but honestly when you're on a really uh constrained platform you really can't get away with just being sloppy like that um all right that's a typo i don't know it says overlays uh that's supposed to say overlaps my bad so overlaps uh so it's when you detect that one trigger box is within another trigger box um so those are really expensive and unreal um every time you move a component then they update the overlaps on top of that it's kind of sloppy the way they wrote them is even if you don't have a single component in the attachment chain that updates an overlap it will still go through this entire like update overlap chain down the children so once again if you have a long attachment chain it will

go through that so as a optimization we just kind of deleted that because we didn't want to use them and that saved us a lot of cycles since we weren't even using overlaps we actually back in starting raw data we run our own async overlap system that updates overlaps on a worker thread and that doesn't block your thread every time you move an object um obviously it's deferred but we use we use events for when overlaps happen anyway so um that's been a huge uh kind of optimization for us i know it's easy to like say hey my hand is close to this bottle i'm just gonna stick like one trigger box in the hand one trigger box in the bottle uh you can't be sloppy like that on these platforms especially once you start having thousands of bottles and two hands and touching everything so um ticking once again very simple just disable tick when not needed so often designers just have components and actors just tick when they're not even they just have a flag to disable certain behavior but they're still ticking so tick interval is another thing often thing if the things are not significant they don't even need

to take every frame once something you have to watch out for is they're not real at least the tick interval you can they added tick interval so you can say like okay this tick ticks like once every like point one seconds or something but there's no control of which frame you take so now if you have a frame rate hike or whatever now all of a sudden things get just get pushed into one frame so you have like one frame where a bunch of things tick the next time nothing ticks so you really want to load balance so we've modified the engine once again to really help us uh load balance and tick things uh you know one frame versus the other frame uh i mean the easiest is just to say okay like half the enemy is ticking this frame how the enemy is ticking the other frame now all of a sudden technically you just did a 50 optimization good job uh so uh blueprint logic uh just it's really slow try not to have any blueprint logic in tick uh try to nativize as much as possible throw in c plus if you can active pulling just spawning things uh dynamically is pretty damn expensive so we developed

our own active pulling system um which then automatically disables like physics collision and a bunch of other things so we don't have to do it manually every time if you're using umg ui you can't get away without using invalidation boxes um so let's move on to the draw calls make sure to use multiview so multiview allows you to do one draw code per eye makes no sense not to use it unless you really want to double your draw calls for some reason uh make sure try to merge your geometry as much as possible you can do it in maya or unreal has a merger actors feature uh we actually modified unreal to like make an unmerged actors feature that we serialize like the previous state it's great for iteration but um make sure when you merge things you actually minimize the material count because even if you merge a whole bunch of actors together but they're all using different materials still that's still each individual call per material so now you actually just made things worse to be honest uh because now things don't call so minimize your material account instance as much as you can if you like instant static meshes um then it's

just one drop hole that's even better uh and like i said not all draw calls are equal um some draw calls are more expensive switching materials expensive switching meshes is even more expensive and actually the the complexity of the mesh actually affects the the draw call cost uh this is so this is the capture of render dock of um uh part of a part of the scene in creed uh this is actually a stadium piece you can see how we merged uh this one stadium piece all the way from the floor to the ceiling uh this we divided the stadium between five to seven pieces and we used three shared texture atlases um in this in this one capture you can see the only two textures that attaches drawcall is essentially the diffuse map and the shadow map um and uh so we use three texture atlases across the entire stadium so um on to gpu um disable or z-pass that's an optimization in unreal um so a lazy pass is actually redundant

in the tile based renderer so the the binning process actually pre-sorts all the triangles uh so that uh they're already it avoids overdraw by only rendering the triangle that's closest to the screen so the z-pass in unreal actually counteracts that um also don't use alpha test like so if you have like a texture with a that tells the pixel like this is transparent or not actually don't use that uh i know that's like a common optimization but that actually counteracts once again this early sorting in the that happens on the hardware level and the gpu because the gpu already pre-sorts of the triangles so this actually forces it to invalidate that whole calculation so get rid of specular specular added about 100 instructions for us per pixel i know that's like an entire hard pill to swallow because things are usually shiny in the world but for us like uh the environment took up most of the screen and just

removing that specular just gave us two milliseconds back on the gpu so that was pretty massive uh gains we still kept it on the characters because you're boxing the characters are sweaty we want to simulate that shininess of their sweat um and uh post-processing so that's an even tougher pill to swallow because post-processing just is such an easy way to just really drastically change the appearance of the entire scene the problem is it really does not play well with the architecture of the quest because with post-processing essentially you have to take you have to do a whole new render pass you have to take the rendered image and you have to pull it back into the the binning process and operate on that again so it's just it's really expensive something we're talking about in the future is just modifying our shaders for some just a global uh shader effects so that you can potentially like apply uh things on a you know per object basis so it's not a screen space effect but simulate those things

and also um stay away from dynamic lights um you know big as much as you can it's kind of obvious i guess but uh we actually faked lighting on a per shader per object bases where we need it uh so you know speaking of about faking post-processing and bluing and um and dynamic lights so i don't know does that video is that video playing it's not sorry uh okay so i probably should have tested this but uh essentially it's it's a ring there's spotlights uh floating around um and um the spotlights hit the ring and you can actually see a circle so for the bloom around the spotlights instead of using post-processing bloom just attach a sprite uh way cheaper and for the actual circle on the ground what we did is we actually projected instead of using a dynamic light to calculate you know how the the spotlight affected we just projected a circle circular patterns on on on on on the floor there um and that's way cheaper i only edit

about 20 20 instructions uh but uh we didn't want to add the thing is that's 20 instructions for every single pixel so we didn't want that active during the actual fight because the spotlights were only floating on the flight uh during the this intro sequence um and then during the fight we would have particle systems and all these other things spawning and using up more draw calls so uh we put in a kind of global if statement to disable that uh one thing you have to watch out for if using unreal is uh if statements in the material graph still actually evaluate both conditions and then they choose which one so this is something we picked up in render dock so instead we wrote we wrote this just in the usf file we wrote the actual shader code with the if statement in there and we called that using a custom node and unreal you can use a custom node to call into any shader code by name um so mentioned uh texture atlases make sure to uh combine all your textures as much as possible like i said switching between textures

is a pretty expensive uh drawcall operation so you wanna batch things as much as possible so we used about two three for the for each environment make sure you use astc uh compression um so you can use render dock to confirm that or memory port to confirm that your textures were actually uh successfully converted to stc um sometimes textures failed because of some weird things that the artist did um similarly actually you know speaking about texture dimensions make sure they're power of two just make sure the textures are clean because if they're not powerful then then they won't map properly and they're not gonna stream properly uh and map mapping is is great for caching too because then it's not gonna use the highest res texture when it doesn't need to when it's far away um and on the topic of dependent versus independent actually so dependent here refers to you're using one textures uh you're looking up in one texture and using that result to actually look into another texture so we found that the cost here is was

actually about the same so if you can you can then potentially bake some expensive calculation you're doing a gpu at runtime just bake that into a texture and use that to look up into another texture um msa it's actually cheaper on tile-based renderers so and honestly nvr you really should be using that we ship with four forex msaa shader hitches was another issue we had basically anytime a particle spawned for the first time in the game you would get a hitch so you in unreal there's console command r dot safe shader cache uh which basically uh we had to run through the entire game make as thoroughly as possible use this console command i'll put the binary file with all the shader permutations and you can package with that with that in 421 they have this pso caching feature which i hear is making that easier we have to ship in 420. um the exfoliated rendering uh if you don't know what that is is essentially

rendering the screen at different resolutions depending on if it's the center of the image or it's the edges of the image since it doesn't since if it's in your peripheral it doesn't need to be as uh high-res so you can actually control the resolutions of those regions i mean worst case scenario you might even have to downgrade as even the center image if you really feel right bound and try to avoid long thin triangles because those can really not play nice with the whole binning process because then the the bounding box i'm still not you know reading online i'm still confused whether they have dealt with this or not but essentially as far as understand the bounding box can make the long thing strips fall into multiple tiles where they're not even really contributing any pixels and you're just making all those tiles more expensive um memory i talk about the fact that shared bandwidth so that does mean that if you're doing some expensive stuff on the cpu with the memory it can actually affect the gpu and vice versa which once again is useful to like be able to toggle one of the processes on and off to figure out which one's actually causing the bottlenecks um some memory stomp tracking uh

is uh so essentially if you if you if you have some somewhere writing or reading into memory that it doesn't actually own some de-allocated memory blocks or unreal has a memory stomp allocator uh that you can compile with but we found out that it caused the game to crash really fast because it really pads all the memory um so like i said ideally you test on the pc but we had this case where our game was crashing like five times a day uh randomly in like random regions so we knew it was a we knew it was a memory stomp and it was only happening on the creed uh on the quest build so is a pretty uh kind of nasty situation so we found out that you can use essentially the built-in android memory stop allocator here and we caught uh this stomp happening deep inside of the ovr compositor specifically related to the splash page so essentially in their like load screen pop push functionality they were accidentally

writing over some memory that they didn't own so we help them fix that crash in their system and yeah so other general tips they you know just like playstation vr there's some trcs related to how fast you know your game loads how long you spend before you show the load screen so what we've been doing basically for our titles is we usually load into an empty room first and then because if you have uh if you're if you're trying to load into your regular map first it will try to load that and you just will not be able to push the load screen fast enough uh the game will just be stalling uh but even there uh there's often issues with just global assets being referenced like global managers referencing some assets with hard references where they're not needed etc so you're going to need to break those reference chains if you want to have a quick load time audio the unreal the audio the default on

android audio does not actually spatialize audio uh at least it didn't when we shipped uh and so you can use android audio mixer android or other third-party plug-ins for spatialization it actually messed up all our mixing and we had to ship so unfortunately we shipped without specialization because it was either the audio team was going to somehow i don't know try to redo all our audio for the entire game on our chip so yeah um so that's that's about it uh from all the tips does anyone have any questions sorry that was pretty i don't know if you guys were ready for such a technical deep dive or whatever i don't know hopefully that was useful to people for thinking about targeting the quest yeah hey that's really interesting and useful because i'm in the same boat thinking about porting i have a more general question on like once you've done it and you've got it working on the quest yeah and you but you primarily

develop on pc like how do you keep your integration working and keep you know because it's so such a pain like trying to test it again and again and again i'm curious so how do you manage that making sure your developers continue to keep it working once you've done the original ports yeah well so what that's why i don't like the word port it's more uh parallel development right so you choose your you choose your target sku um and you you have ideally qa that hammer on this build uh every day uh you have continuous integration so jenkins or something that basically outputs these builds on a daily or hourly basis and then daily basis depending on you know if it's a full package whatever and you have qa actually testing these things on a regular basis that they can they can catch issues uh really as fast as possible so it's definitely a lot of process um and um i'm not saying it's perfect like it's it's a it's a pro it's a constant battle of like you getting in fights with other people because they break things

so yeah hi thank you yeah uh first off are your slides going to be available do you know that uh yeah i cannot blow them yeah okay and in the when you were mentioning some console commands except for the few that were adb uh was that the unreal console you're talking yeah there's it's all in there and real unreal so yeah all right thank you yeah hi um i was wondering for new for people starting to develop a new experience would you recommend that they uh start by developing it on rift and then port to quest or start on quest and then um add more stuff to take advantage of the higher performance on rift well like you would you would always be kind of developing alongside because all your testing is probably go most of your most of your iteration is going to be on pc but if you're going to try to launch in the quest like i said avoid porting try to launch that thing on the quest as soon as possible so you can at least periodically see that when you add things you didn't just blow yourself way past the budget because now

you added 20 things and they're all past the budget now you have to determine like what what needs to be cut and what what doesn't so it's always easier to add things than remove things you know removing things is killing your babies adding things is making more babies i guess that's interesting i hear a lot of people say the opposite actually what do you should just focus on developing the experience first without worrying about performance and then do that after the fact it's i guess it i don't know a question of philosophy all right thank you yeah hi there thanks that was really interesting um can you tell us anything at all or commented all about the submission process and how quickly they're turning approvals and communication at all i on one i i just i don't know if i can really give an accurate really estimation of that because a i'm i'm on the software side really so i don't i'm not part of the schmoozing process fair enough yeah