Taming Technologies Behind Astro Bot Rescue Mission

[Music] hello everyone thank you for coming my name is yuki miyamai from si japan studio i was the technical leader of astrobot rescue mission today i'm going to talk about what the most important technical goals were for this game and how we achieved those goals for those who don't know astrobot rescue mission is a vr platformer which was released in october 2018. it uses various features of playstation vr and the dualshock 4 controller there are over 50 levels with many different visual themes the game was developed by 26 people in 18 months this is a photo of the team asobi in japan studio during the development these were the

three biggest technical goals our target frame rate is 60 and it's scaled to 120 by reprojection as primary drops are not allowed in vr games we have to keep 60 fps at all times and second because it's vr visuals are physically close to users so they should be comfortable on your eyes and at the same time convincing the third one is smooth game flow which means we have to pay much attention to game flow or game stage transitions because they are repeated many times if they are not implemented well users will easily get tired in vr the purpose of this talk is to share some of the techniques we used to work toward achieving these goals so the first topic is rendering optimization this has the biggest impact

on the frame rate i want to talk about some of the techniques which were effective to keep 60 fps at the first optimization we kept the scene compact we knew that optimization was really tough for vr games so we started optimization at the game design level we divided levels to small sections and each section is roughly 30 meters by 30 meters and depending on the player's position some sections are not visible from the player so we manually set the section visibilities players go almost straight from a start point to a goal so manual control was manageable for our game in the product we had a maximum 2500 charcoals at a time including shuttles and stereo rendering i think this is small enough

our renderer spec is as follows we used forward rendering with h sample msaa as a small optimization here we pack the depth value to the alpha channel of the color buffer by doing so we can skip the depth resolve and when sampling background color buffer we can get the depth value for free we had some special effects shaders but for most of the objects we used basic single layer pbr materials the number of lights is also really small we have max two blended ibl's one directional and a 16 point and 16 spotlights all of these lights are evaluated in every material with no tile or cluster calling because it's much faster for this number of ways so we kept the pipeline simple as much as possible for the gpu performance

as another gpu optimization we implemented dynamic resolution system we measure the gpu time of the previous frame and decide the resolution of the current frame the good thing is dynamic resolution can handle emergency cases for example diving into fires or water those kinds of extreme cases and more importantly it increases the resolution in most cases our resolution setting is shown in this table 1.0 scale is the display resolution of psvr and we have scales from 0.85 to 1.5 we have only four settings but it greatly improves the image quality compared to just one setting here is how it works it's simple if the total gpu time of the previous frame is over 16.6 milliseconds we immediately

decrease the resolution when we scale it up it's more conservative to estimate the gpu cost of the next resolution we scale the stereo rendering time by the pixel count ratio and if it's below 16.6 milliseconds for 30 frames we increase the resolution some gpu jobs are not related to the resolution such as shadows so it's important to exclude those times from the estimation and because the gpu time is not proportional to the pixel count for example it depends on whether it's vertex heavy or pixel heavy we added the scale parameter s and 0.85 worked well for our case i played some levels and took statistics on ps4 the resolution was over 1.25

scale on average before implementing dynamic resolution we had been using 1.0 by default so it's a big improvement on ps4 pro 1.5 was used almost always although the quality was improved sometimes we noticed resolution popping because we had only four settings we prepared the four pipelines in advance and just switched them but now i think free dynamic approach which has finite resolution granularity for example per two pixels will be better as it produces smoother result there is another problem in heavy scenes we don't see any frame rate drops even in heavy scenes but the resolution decreases to the lowest quietly it happens in many levels our qa team is very strict about frame rate but not about resolution degrees so it

might be missed in the final qa process another problem is that even though the resolution is the lowest the frame rate is keeping 60. so it's difficult to know how much how many milliseconds we should optimize to get higher resolution this is an example we show a performance indicator in the center of the view when the frame rate drops so everybody can see it this is the zoom up we show frame rate and gpu time in milliseconds without the dynamic resolution system the frame rate drops like this qa team reports this issue and also it's easy to see how many milliseconds we should cut to keep the frame rate but if you turn on the dynamic resolution there is no frame rate drops but the resolution is too low we want to

get a higher resolution but the gpu time information here is useless because it's impossible to know how many milliseconds we should cut in the next resolution because we are rendering in the lower resolution so to control the quality with dynamic resolution system we decided target resolution and emergency resolution the target means we should achieve it at minimum and the emergency resolution is only allowed in emergence cases not everywhere with this consensus we didn't use the emergency resolutions until the very end of the production we intentionally made the frame rate drops and encouraged the team to optimize the levels i think this worked well if we look at the graph again you can see that the lowest resolution was really used

in the product it's less than 0.1 percent okay then next is a cpu side optimization topic stereo rendering method we consider three methods for stereo rendering the first one is to run the rendering pipeline twice this is the easiest implementation but has lots of cpu waste the second one is resubmitting a command buffer this saves the cpu cost to generate a command buffer and it requires no shared modifications but draw call count is still big the last one is to use instance raw to double the geometry in a single path you draw the objects with double instance count and switch the render target with slice index in vertical shaders

this can have the draw called count so it's good for cpu and gpu however this requires vertical shader and some pipeline modifications so we chose resubmitting a command buffer because we also had to support non-stereo rendering pipeline and we wanted to keep the code simple on ps4 command buffers we built a processed directory on gpu so resubmission has almost no cpu overhead but because the command buffer is processed as it is we need to care about synchronizations ourselves for example when we make gp wait for rendering of the previous stage we usually use labels allocated from command buffer and wait wait until the label is set by gpu to a certain value but when the command buffer is processed second time labels are already filled so you have to

clear the labels after the first commands are finished to do this we allocated the memory block for the labels and put a memory copy command a clear command at the end of the commands so the shared command buffer is created as follows first set up the left view constant buffer by cpu view id is also included in this buffer push scene draw calls as usual which are for materials and post effects push output draw call that selects the render target there are several ways here we put two draw calls one is for left and the other is for right and called one by geometry shader depending on the view id or if render target array is available you can just use slice index in vertex

shaders finally push memory copy commands at the end of the command buffer which fills the view constant buffer with right eye and clear synchronization labels once you've created this command buffer you can simply call submit twice in this way we can save much cpu time okay now i want to move to the next topic effects in vr we implemented some effects and spend time to make it look good in vr so i want to share what went well i picked up three effects water clouds and 3d texts the first effect is water we have some water levels and as a gameplay the player can go into the water

in vr players viewpoint is free so we couldn't avoid to draw the boundary of the water on air i think the unique point of our water rendering is that we seriously render the boundary rather than hiding it in addition the surface should be moving as a game play requirement in this image you can see two components of the water rendering water surface and a fog effect applied to the underwater area i'll explain how to render the each component the surface is just a fine mesh no dynamic tessellation because pre-tessellated geometry was faster for our level size we move the vertices in vertical shader with three sine waves which are tweaked by material attributes and 16 wave sources controlled by

a game called the shading is not so special we just calculate scattering and absorption using the distance to the background but an interesting effect we added was surface tension effect we bend normals at intersections to the background it's exaggerated but we feel we feel we felt it's realistic we also bend normals at the camera intersection which makes the boundary look nice underwater 4 is just a plus effect but as shown in this figure we needed to detect the under altruism in the screen and apply the fog only to that area actually you use stencil buffer the steps to create the stencil or first clear the stencil buffer with zero draw the scene objects without stencil

operations and draw water area box which covers the bottom of the water and sets the stem survival to 1. and next draw the water surfaced with stencil operations this is a bit complicated when depth test passes we set three and when depth test fails use decrement for front face and increment for backwards then only the underwater area has stencil value 1 so we apply the fog effect where the stencil is 1. the reason why that step 4 is complicated is to handle any viewpoints like these figures above the water inside the waves and completely underwater this might be hard to understand now but if you take time to think later you will be able to see this rework

here is the final result you can see that the player can dive into the water smoothly the water drops on the screen adjust distortion particles this is another thing you can see that the fog is only applied to the underwater area and also you can see the boundary okay that next effect is clouds clouds were used in many levels of the game so the cloud rendering was important for us our class are close to the player and artists want to control the shape freely so we can't use the cloud layer approach which was used in horizon zero dawn suited for situations where players look up at the sky also

they need to be lightweight so we have to avoid lots of transparency and of course they need to be moving so we took a mesh approach again we draw the polygons sample a 3d noise texture and use it for the shading we also use the noise for the vertex displacement the noise texture is similar to the horizon method r channel contains low frequency noise and gba channels have higher frequency noises the low frequency noise is eroded by the high frequency noises the detail of how to use the noise was explained by the shima engine team at siggraph 2017 so i'll skip it here for shadows because dynamic shadows on the clouds need many sample points to hide the polygon feeling we just used baked shadows on the clouds

including cell shadows so here is the shading steps we take five samples along the v-ray and accumulate the sun scattering precisely at each sampling point you also need to sample the noise in the sun direction several times to get occlusions but we can't afford to sample many points so instead we just sample one point towards the sun direction from the last sampling point we subtract the noise value a from b and use it as a multiplier this produces detailed sound occlusions and baked shadow map covers the static large scale occlusion and finally we add ambient by evaluating ibl diffuse with the surface normal and the accumulated noise value this is a very rough approximation

but it can make the clouds fit the scene and it's easy to implement so i show the components by images clouds are made with fine meshes like this with five samples baked shadow map and ibl ambient you get this because five samples is too small we don't get any volumetric feeling yet so we move the vertices to y direction according to the noise values by adding one sample to the sun direction you get more details like this like this we wanted more details on the intersections so we used the noise as alpha values when it's close to background we reuse the noise so there is no additional sampling but we use different frequency weights usually higher frequency than

the shading like in this code sample with some magic numbers and a step function we got nice details on the intersections this is the in-game result for this scene at the half of 1080p it took around 1.2 milliseconds per eye on ps4 it looks a little bit cartoony due to the vertex displacement but we think we've got interesting movements okay the last effect is 3d text this is just one slide but what's worth mentioning is our texts are all made with polygons most of our texts have small extrusion and that gives better 3d feeling than texture planes in vr another good thing is that it's resolution free we tweaked text positions many times

but we tweaked resolution setting only a few times one more good thing is we didn't have to make special shaders such as distance field shaders we simply we used effect shaders to generate the polygons from text we built an automated system using type node in maya and we finally supported 23 languages okay this is the last topic smooth game flow this might sound like a game design topic but i'd like i'd like to talk about techniques behind it as you can imagine unsmooth game flow in vr creates discomfort for users for example long loading time makes users tired device tracking interruption while

displaying something leads to sickness and if a black screen is displayed for a long time the game loses immersion so we do the opposite we try to minimize the loading time we keep the device tracking all the time and we avoid black screens and show something that users can interact with so this is our main game flow we have a word select menu a pause menu on levels when a user selects a level in the word select the menu closes and the user moves to the level and when the user clears the level they go back to the word select users have to clear the levels one by one and they always go back to the menu every time and the pulse menu can be opened anytime this is the spec

and it's a very simple design to achieve this game design and make it smooth we thought about memory management first to minimize the loading we load the word select and the pulse menu as resident resources once we launch the game and never unload them then when a user selects a level we lose the level resources everything fits in memory in other words we optimized every level to fit in memory when the user clears the level we just unload the resources we repeat this process when the user moves to the next level load the resources and unload them once the level is cleared so with this strategy because unloading

is so fast the only path that takes time is the transition from the world select to the level so i will focus on how we manage this transition first i'd like to show the video of our final smooth result this is the word select you select the level astro comes to the controller and you can interact with an effect bar blue the loading has finished you trigger a word revealing effect [Music] and you can start the level this figure explains what happened the top row is what users see and the bottom row is the loading process when the user selects a

level we start an animation to close the board select at the same time we start loading meshes and animations while loading those resources we keep the rendering of the in-game controller also we show an effect ball that the user can interact with using the controller so you feel you are still in the vr world even in loading time once the loading is finished we start loading textures at this point the user can start the level and the revealing effect creates the word the point is we delay the texture loading and allow users to start levels before finishing the texture loading this shortened the loading time and the time users think of as loading was less than 5 seconds so how did we implement the delayed texture loading

the implementation is so easy when the when we load meshes we assign constant color textures and we load texture files in order of distance from the player and switch the temporary texture with the actual texture once it's loaded we rarely see any popping thanks to the large revealing effect we also investigated other texture streaming techniques but considering our level size and game design i think this was the fastest way to achieve our goal so to sum up minimizing loading time keeping device tracking and keeping word interaction these are so important especially in vr okay we covered these topics today you might think there are no advanced technology or complicated techniques and we always

choose the easy path it's partly true we pay much attention to implementation cost we choose easier ways if possible so that we can focus on other important things i think this way of thinking is important especially for small teams with limited amount of time so i hope this talk will help you to cut implementation time and achieve great vr experiences thank you i think we have almost no time for q a but i will move to the wrap-up room so if you have any questions please come with me or just email me thank you very much