Beyond the Remake of 'Shadow of the Colossus': A Technical Perspective

I would like to thank everybody for attending this talk. Uh there are a lot of great opportunities out there and I'm flattered that you're here. I'll let you know that uh there are a couple of bonus slides at the end of this presentation that I won't have time to cover. However, this talk and its slides will be available on the GDC Vault. And uh or you can just reach out to me directly if you're interested. And uh also, please don't forget to fill out your evaluation forms at the end. Um I'm pretty sure I need the feedback. So. Now that you are here, whether intentional or you just realized that you're in the wrong room, you're most likely wondering who the hell is this guy? Well, my name is Peter Dalton and I am the technical director for Bluepoint Games. As you can tell from the gray in my beard, I've been around for a little while. I've been working in the games industry for over 18 years and I will never leave. I love my job. I love the insanely hard problems and I love the

talented people that I'm surrounded by. Over the 11 plus titles that I have shipped, my specialties revolve around core engine development, streaming, memory, performance. I like to believe that it is my job to basically make happen. So. Bluepoint Games. Perhaps you've heard of us, the masters of the remaster. Uh I have to thank Digital Foundry for that title. Seriously, the Digital Foundry guys are amazing. Their technical reviews are spot-on and Bluepoint is tired of me ranting about how our frame rate and frame pacing must be perfect or Digital Foundry will call us out. Bluepoint has made a name for itself by remastering games, many of which I'm hoping you've heard of. We've been blessed to work with titles that we absolutely love and from my perspective, it is insanely awesome to that we get that I get to see exactly how these games were made. At Bluepoint, there are two primary

ideals that I try to promote. With number one being quality is the foundation of success. I believe that it is absolutely true for our industry. If you want long-term sustained success, you must have quality to feed it. And number two, every project that we release must be better than our previous release. And hopefully you can see this by looking at the titles at our past projects. The depth at which we remaster projects has grown exponentially. Starting with the God of War collection, which involved texture cleanup, to the Uncharted collection, which involved all aspects of the game being modified. In fact, for Shadow, we don't call it a remaster. We coin it a remake given the complexity of the project. And moving forward to our next project, we coin it a re-envisioning given that it goes well beyond what we even thought was possible in Shadow. Now, I want to take a minute to show you a video from PSX 2017

showing a comparison between the original Shadow and the PS4 remake to give you a better sense of what we do.

I feel that it is important that people understand a couple of our overarching goals that we take into each remastering project. Number one, we have to be true to the design of the original game. Number two, we have to respect the decisions of the original game team. And number three, we must not lose the magic of the original game. And if you say to yourself, I can't find the magic in the original, then what the hell are you doing? And number four, we have to acknowledge that it is rarely a black and white answer for what we do and as a result, there will be lots of internal debates. These debates are good and healthy and most importantly, they show passion. To help explain our thought process, I like to use an analogy. As a kid growing up, I used to love watching cartoons. Who didn't? I'm sure if you think about it, you had your favorites, certain shows that stand out in your memories. Well, in my case, I loved ThunderCats

and have fond memories. However, if I go back and watch them now or try to show them to my kids, they aren't anything like I remember. What we are trying to do is bring those memories back to life and enable others to have similar experiences. We're trying to recreate the game the original game developer would have released if they had the technology that we have today. So, let's take a technical look at the development process that we use at Bluepoint when remastering a title. Besides determining whether or not it makes business sense to remaster a game, there are several key factors that must be considered. Number one, we absolutely must have the access to the to the released package. For Shadow, this was the final package that was delivered to Sony for distribution. This should include all patches, back-end servers. Basically, we need to be able to run the retail project in-house. And number two, we require all the final source code required to rebuild the

game's final binaries. Now, there are exceptions. We can work around miscellaneous systems. For example, if we are missing an audio library, chances are we're going to replace the entire system, so no biggie. However, if we are missing gameplay code, it quickly becomes cost-prohibitive to reverse engineer those libraries. Others examples include server code if required is a bonus. However, we can work around not having it. Tool pipeline code uh is a bonus. However, we really We usually don't build our processes around legacy tools and primarily only use them for reference. And finally, any source assets that were used to build the original final distribution package is a bonus, but not mandatory. So, looking at the development process that we go through, the goal for the first month is to obtain all the required pieces and to rebuild the the game's binaries for the original target platform. We then send this build to our internal QA and run a whole suite of

parity tests. I should call out that it is critical to the success of the project that we get these first steps correct. Small errors here will be magnified as we get into full development, leading to wasted time and incorrect decisions being made. Here we feel that taking time to ensure correctness is preparing for success. Moving forward, once we have parity, we upgrade the original title to its latest SDK to make life easier, then rerun all parity tests. Basically, we make incremental and verify parity through all steps. Take Shadow for example. Very early on, we decided that we were going to use the remastered PS3 version as our base. At that point, QA ran extensive parity tests between the PS2 and the PS3 version, combed old bug databases, searched the web for user feedback, etc. Concurrently, engineering got the game up and running on the PS3, converted to the latest SDKs, we localized all Japanese code comments, uh

rearranged code libraries to make us happy, and then rerun all parity tests, and we fixed the issues before moving forward. At Bluepoint, every developer at their desk has full access to our target platform, which for Shadow was a PS4 and the original game, which in this case was a PS3 dev kit. Programmers are expected to debug and play the game on the PS3 dev kit to determine original code intent and purpose and to diagnose parity issues. We do the same thing for all other departments. Art always has full access to both platforms and are expected to be familiar with both. And in the case of Shadow, QA went the extra mile and threw the PS2 into the mix. Only after parity is verified do we start porting the code to the target platform. Po- porting is usually a one-to-two man project that takes approximately two to three weeks before we have the code compiling. During this phase, we are more

interested in just getting everything to compile on the new platform, not porting it. Basically, we try not to lie to ourselves and pretend that we know how everything works, and instead we're using this time to familiarize ourselves with code base. We create pound defines to disable whole systems, mark up changes we have made to the original code with the most important goals being number one, to get the code to compile and link, which is harder than it sounds, and number two, make it trivial to identify where we have modified original code. So, congratulations. After about two months, you have a binary that you can launch on launch on your target platform. Not only did you get the game to compile, you got it to link, which was a huge pain in the ass, but you can't help but want to play the game, so you hit run, and boom. Everything blows up. In fact, the breakpoint that you put at main never got hit. You blew up during pre-minute.

The call stack doesn't make any sense, and everything feels broken. Memory is being requested. You have 32 to 64-bit compatibility issues, plenty of code assumptions that a pointer fits in a uint32. And resource files are being requested. In fact, where the hell is the file system? And you have Indian issues everywhere, and how do you deal with memory-mapped files that assume a 32-bit pointer? At this point, you can't help but wonder what you've gotten yourself into, and does it make sense to assign this task to another engineer? At this point, I must tell myself to stop being a little and just keep pushing forward. So, given that the first step is just to get the game to compile on the target platform, the next step is to get the game to main, and then to the main loop. The main loop will be pretty much commented out at this point. However, our goal is to get the game running displaying nothing but a black screen. Getting to a black screen is a critical

milestone for us. It is really the point at which the entire team of engineers can really start piling onto the project and looking at their respective domains. It is the point at which momentum momentum really starts to pick up. So, I want to stop for a second, and I want to share a little bit of Bluepoint history. With all the projects that we've worked on before Shadow of the Colossus, we would work within the realm of the original code base. We would rewrite low-level platform code to work on the new target platform. We would build a new rendering system to handle the needs of the game. Tools and processes would be adapted to closely fit those of the original development team. The problem was that with every new project, it felt like we were completely starting over. Code would end up being very tied to a specific title and thus not moved from project to project. The team, art, engineering, including QA, only needed to re-re-relearn processes as we moved from game to game.

Basically, it started to conflict with our ideal that each project needs to be better than the previous. So, for Shadow of the Colossus, we completely revamped how we work. We made a point of building reusable technology and processes that evolve from project to project rather than being recreated. This requires a deeper commitment to processes and long-term planning rather than just solely focusing on immediate goals. So, we took our Bluepoint engine, dusted it off, and started to make a major investment. For reference, the Bluepoint engine is a proprietary engine and has been around for quite a while. In fact, it has been licensed and used to ship several titles. However, it was time for it to evolve. Now, getting back to the goal of getting the game running. We achieved this goal by merging the original code with the Bluepoint engine. Basically, we structure the code libraries so that the original game is built on top of the Bluepoint engine. This allows the Bluepoint engine to provide all platform-centric systems such as memory,

threading, file IO, rendering, etc. For us, it is key to keep in mind which original game systems we want to port to the target platforms and which ones will be replaced by Bluepoint systems. Basically, we don't want to waste time porting a system that will just be replaced in the future. With that said, I think it is important to make one clarification, okay? When it comes to gameplay systems, AI, uh character logic, etc., we want to keep all of the original systems intact. While we will fully replace core systems, with gameplay we take a much more surgical approach to fixing and enhancing. Next, I want to take a minute to discuss Bluepoint's approach to game assets. While it is ideal that we get the source assets from the original game team, and we do to varying levels of success, we do not depend upon them. Rather, we only use them for reference. The reason that we don't use source assets as a starting

point is that they're often wrong or incomplete. Teams are notorious for not checking everything into source control. Often, local file changes or P4 shelf changes are used to build the final package, creating very difficult to find parity issues. This is exaggerated when dealing with patches as the likelihood of a one-off local change increases. To eliminate this concern, we spend around two months extracting all the data within the original distribution package to BPE compatible formats. We also assume that before we ship, we will have the need to edit every type of data, not just key file formats. While this is time-consuming, it has several key benefits. Uh number one, we know that the data that we have is exactly the data that shipped. And number two, we actually learn a lot about how the game is constructed, allowing us to make smarter decisions moving forward.

Understanding the content greatly helps us understand the code, enables us to make decisions that work with the original code as opposed to fighting it. And number three, once we have extracted the key file formats and converted them to BPE formats, such as models, animations, skeletons, textures, collision data, these assets are immediately available within our BPE tool set. And finally, because we are taking full own- ownership of the data, we are no longer dependent upon the content pipeline of the original team. We don't need their tools, and we don't need to follow unfamiliar practices. It is actually quite liberating that after you the extraction is finished, we follow Bluepoint processes rather than the processes of a remote team that we don't understand. Now, to be completely completely honest, there are some game-specific file formats that we don't always extract and will take shortcuts. We do, however, regenerate these files addressing like

Indianness and 64-bit compatibility issues, and we spend the time to understand and document their purpose. For example, in Shadow, we did not fully convert all of the padding data to BPE formats because we figured that we would never change this data. This came back to bite me in the ass, and I'll tell you more about it later. So, let's take a look at the final results that we achieved in Shadow in relationship to how we integrate two distinct engines. I like to call this the dance, okay? Without proper planning and coordination, chaos would take over. However, with each engine assuming specific responsibilities, we can create a harmony of technology. What this diagram is trying to illustrate is the responsibilities of each engine. The The original Shadow engine maintains sole responsibility for the majority of all gameplay. If there was a gameplay bug or a behavior that we wanted to modify,

chances are that we were going to modify the original Shadow code base. On the other side, the Bluepoint engine handles all core system responsibilities. Memory, threading, platform services, along with anything visual. The Bluepoint engine handles the management of the world, the static geometry in the world, the particle systems placed, the lighting, etc. It remains the responsibility of the Shadow engine to create all dynamic game objects such as Agro, Wander, and Colossi within the Bluepoint engine in order to build a complete scene, which in turn feeds the render. Take for example Wander, the main character in the game. The original game has a concept, an object representing Wander, and also a link to a BP engine representation of Wander. The original game provides all the simulation logic and pushes matrices and the required state information over to the Bluepoint representation. In turn, the Bluepoint

representation can add additional functionality such as head tracking, and drives the render to ensure that everything shows up in the final scene. So, let's take a look at the game and break down a scene starring Colossus 6. So, who is responsible for what? Shadow is responsible for all dynamic characters in the scene. In this case, Wander and the Colossus. Shadow is responsible for AI behaviors. When QA would report behavior problems, we addressed it within the original code base. Shadow is responsible for collision. In retrospect, this was perhaps a mistake and should have been brought over into BPE in order to make it easier to work with. And Shadow is responsible for building the animation blend trees and building the final pose. Shadow has these responsibilities because the simulation is dependent upon the final pose. It would have been better for us to use BPE systems for animation. However, at the time intertwining shadow and BPE

processes was very difficult. The eye state of the Colossus was determined by shadow code passed to BPE code where it was managed used to determine the correct eye color and drive dynamic uh shader parameters. It is the shadow code that handles the simulation of the bones that hang around Wander's waist. These were never lifted into BPE given that the original coder did such a great job. So, that is about where the original shadow code ends and the BPE engine takes over. BPE handles all rendering and scene management. The entire environment and atmospherics that you see are all handled by BPE. The physics simulation of Wander's poncho is handled by BPE BPE. It is uh purely visual and needed improvement and thus all aspects were moved from the shadow code and moved into Bluepoint. BPE's responsible for the dynamic rings attached to the Colossi. In the original, the rings are static and misaligned. They are now dynamic.

And finally, BPE is responsible for all audio and particles within a scene. One way to think about our approach is that we are taking the original game and overlaying visuals and enhanced gameplay. Here's another scene with a path nodes debug display enabled. The blue lines simply show available connections between nodes while the purple line shows the navigation path that Agro, your horse, will take to get to you. When we started the game, we figured that we would not change the layout of the environments enough that we would have to modify collision. This game This became perhaps the biggest lie that we told ourselves. Before shipping, all collision within the game was completely rebuilt. We also told ourselves that because we were not significantly changing the layout of the environments, there would be no need for us to modify path information. This was also a lie. As the environment started to be finalized, QA started flagging issues where Agro could no longer path to you because someone placed a large tree on

top of a path node. In certain areas, Wander could not find lizards because they were now hidden under hills. In other areas, the hawks and the birds in the game would start flying through mountains that were now in their paths. After a a bit of digging, it became obvious that it was not an acceptable solution to require art to go back and fix the geometry to match the constraints of the original. Instead, within the last 2 months before shipping, we wrote a tool that allowed for path nodes to be visualized and edited directly in the game and serialized back out. Before this, the pathing logic was the last piece of untouched code and data. Well, we didn't necessarily set out to change every single piece of data loaded by the game, it became a necessity. So, I guess the moral of the story is to assume that if something can bite you in the ass, it will. I think that's a pretty safe mantra for anyone in engineering. So, Shadow released a year ago this February. I explicitly don't want this talk to feel like a postmortem, but more of a

sharing of our approach and processes. My goal is to be completely transparent and share what worked and where we fell short. Well, on Shadow, we had a fairly sophisticated worker job system. We were not able to retrofit enough code to take full advantage of it. We had this notion of the shadow code completing its full simulation before moving on to handling BPE game objects. The CPU frame was broken up so that Shadow took approximately 70% of the game frame and the Bluepoint engine took about the other 30%. We could find areas to optimize shadow code by jobifying. However, this basically just moved code from one core to another and really without really increasing the CPU saturation. As As a result, we did did not we did not get much parallelization within the game code. Within the Bluepoint engine, we represent a scene as a tree of game objects where each game object contains components that exposes functionality.

To update a scene, we would walk the game object tree and update all components. The overhead of walking the tree quickly became time con- time consuming. To make things worse, we would actually walk the game object tree a second time to build a list of items to render. This happened during the sync point between the game and the rendering threads creating a major bottleneck. Well, toward the end of Shadow's development, we implemented a couple of Hail Marys to address the problem. They were band-aids at best. Moving forward to our next project, we knew that we needed to re-architect how we coordinate the game loops between the two engines. Basically, it was time to take what we had learned and evolve the dance. The major breakthrough for us came when we decided to stop treating the original code base as special, but rather as just another think process that happens at a specific point in the frame. You can think of the entire original Shadow game

code as a component on a game object exposing functionality. Unfortunately, this is easier said than done. To evolve to where we are today beyond Shadow, we had to make the following changes. Number one, the first thing that we did is we eliminated the need to walk game objects to determine what needs to be rendered. Rather, we make components responsible for adding and removing persistent render items and we support a command buffer style interface for changing render item parameters to keep everything thread safe. Number two, we no longer walk the game object tree to update components. Rather, components register think request delegates. Think requests can be added or removed at any point from any thread. A single object can submit as many think requests as de- as desired based upon demand. This makes it very easy to create objects that only need to think for one frame or under very certain or under conditions. Think requests can also be invoked at any point. If not explicitly invoked,

they will be invoked when their bucket is processed. However, it is trivial to be in the middle of a think request, start an async job that you need to wait for, and invoke other think requests to ensure that we never stall. With these changes and looking back at Shadow, now rather than merging the Shadow game loop with the Bluepoint game loop, we would simply have the Shadow code register a think request within the correct bucket eliminating the distinction between the Shadow update and the BPE update. When the original Shadow code needed to wait for async jobs, it would simply start processing other think requests rather than stalling. This type of behavior is proving to be critical on our next project. In fact, this last week, I spent a day rearranging think request dependencies to fix stalls saving approximately 2 milliseconds within our game loop. I didn't optimize any code. Rather, I simply fixed scheduling issues to increase CPU saturation. Having the flexibility to easily rearrange a

frame's work to fill dead areas and remove contention is proving to be awesome. So, here's a diagram showing our current running layout. It is much easier to use and a lot more flexible. At this point, my only regret is that we didn't implement this system earlier. So, rather than looking at Shadow, let's take a look at how we have evolved and what's next. So, sorry. I hope you didn't think that I was going to expose our next project. I know this isn't fun. Well, maybe it is a little bit to me. Uh well, I'm super excited about our next project and the tech that we are building. I'm pretty sure Bluepoint would make me walk home all the way to Texas just to shoot me if I screw this up. So, next, I want to talk about what we have learned from past projects and how it affects our systems. How we go about creating a harmony of technology by doing a deep dive into our memory

system. How it is designed with flexibility and what steers those decisions. Before I do so, here's a quick disclaimer. All of the examples that I'm going to share are based upon my personal experiences and memories. Uh and my examples are based on outdated technology from the companies in question and don't represent their current technology or processes. And finally, I apologize in advance if I misrepresent anyone's work. So, memory. Who needs it? I'm pretty sure I do. I've been working or dealing with memory systems since I started my career. Over this time, I have tried numerous approaches and have shifted the way I view memory based upon the games that I have shipped. In fact, I wrote an article for Game Programming Gems years ago. As I look back, I still embrace the ideas presented, but I'm a bit embarrassed that the imple- implementations were so short-sighted. I actually got my first job in the games

industry working for Beyond Games creating a Hot Wheels game. The key to landing this job was a BSP collision system that I had written that required less than half the memory footprint of the current system basically cutting memory requirements from six to two megs on a PS2. If there's one lesson that I learned then that is just as true today, it is that proper memory management is critical to achieving performance and shipping titles. So, let's take a look at Bluepoint's hardest title to date. Shipping Titanfall on the Xbox 360. What an amazing game Respawn created. And if you buy me a drink, I have a lot of stories. If you don't recall, the original Titanfall was developed by Respawn and released on the PC and Xbox One. Bluepoint released the Xbox 360 version approximately 2 months later. Simple summary, if I ignore the performance aspect, it is not easy to

ship a 5 gig Xbox One game on an Xbox 360 with 512 megs. So, how did we do it? A lot of hard work or as we like to call it, Bluepoint magic. The original game used a fixed-size bucket allocator for allocations less than 32 bytes and DL malloc for everything else. Where all memory allocations were funneled into a single allocator and dispatched from there, creating a choke point where thread contention became a major issue. I know that a lot of people love DL malloc and argue that it is sufficient. While it has a lot of great features, uh in practice on Titanfall, we found that number one, the version of DL malloc we used did not have any concept of virtual allocations and thus a custom system was required. Number two, DL malloc was not ideal for large page al- page-aligned allocations. DL malloc works by placing tags at the beginning of allocations. If you need a texture with a 128-byte alignment, the tag can

create memory waste. And number three, due to DL malloc not being ideal for small allocations, virtual allocs, or page-size allocations, we needed DL malloc to release unused memory back to other memory systems. In practice, we found that DL malloc held on to more memory than we thought was ideal. The key to uh shipping Titanfall from a memory point of view was memory tracking, knowing exactly where all memory was at all times, tracking fragmentation issues, and tracking memory by category so that the peaks could be compared from build to build to determine trends. From there, we were able to optimize data formats in nearly every system to cut memory usage. We reworked the memory system to we used a fixed-block size allocator for small allocations less than 512 bytes. We used DL malloc for medium-sized allocations and several changes were made to DL malloc to get it to aggressively release memory. And we

used large a large page-based allocator used primarily for textures and large vertex buffers. And we added a concept of a single-frame allocation where small allocations would be placed within unused DXT mipmap memory. This memory was only valid for a single frame and coordinated with the texture streaming system. Basically, every byte was used. By the end, we had spreadsheets backed by more spreadsheets for every level showing max memory usage, how much memory was available for texture streaming, etc. While I'm still in shock that we pulled it off, looking back, the whole system was perhaps a bit more complicated than it needed to be. So, Uncharted. What a great project. Going from a PS3 target to a PS4 target, how could there be any memory considerations? Well, there weren't any real memory considerations provided we kept streaming textures reasonable. However, there are a couple of key things that we

learned. Number one, all memory was allocated at startup by the core memory system. From here, all memory was assigned to specific allocators. There was a hardcoded table within the code dictating how much memory each allocator was allotted. The downside is that when an allocator ran out of memory, you would play the Russian roulette game to push memory around until the problem went away, often over-allocating. There was no virtual memory support and fragmentation issues were evident. There was code to handle the shuffling of memory and pointers when defragging. However, it came across as error-prone and touched numerous systems. The later Uncharted games adopted a rule that memory could not be allocated before hitting main. I loved this ideal given that it was straightforward when to create your memory allocator and at the end of main, you could easily check to ensure that there were no memory leaks. And finally, Uncharted introduced the idea of a tagged heap allocator, which is very similar to a single-frame

allocator that we implemented for Titanfall, just evolved. So, what were some of the philosophies that we felt were important when designing the memory system for the Bluepoint engine? Number one, we should never run out of memory. Never expect null from an allocation request. In fact, we explicitly halt the game if this happens. Number two, we need smart allocation schemes to allow for a wide variety of allocation patterns. Basically, encourage small custom allocators rather than an uber allocator. Number three, allocators should help to eliminate thread contention. And four, ensure debuggability across all allocators without a lot of custom work. To us, debuggability in this case means the ability to track and categorize all memory allocations and tools to help detect and diagnose common memory issues, with the most common being memory stomps. With these basic philosophies in place, we built a list of our goals and I'm only going to touch on a couple of the

key points. Number one, we determined that all allocators should be platform agnostic. We didn't want to write custom allocators for each platform. An allocator should deal with memory patterns, not the specifics of where the memory came from. Number two, allocators should be memory agnostic. Basically, every allocator should work with both CPU or GPU memory. While this seems trivial, it does have implications. During Shadow our per-frame GPU memory allocators benefited greatly by removing tags and markers that were being written directly into the memory blocks for memory tracking. Assume that memory will be allocated at any point. Programmers need power and flexibility to create great systems. And number four, we wanted to eliminate the need for fixed-size allocators. Basically, allocators should work with fixed memory blocks, but also should support growing and shrinking as required. We want to

avoid over-allocating and the memory Russian roulette game. To get started, let's look at how we ensure all memory allocations are routed through our memory systems. We start by overriding system memory routines, which is pretty straightforward. And second, we create BPE macros to wrap all memory requests. We choose to use macros for a couple of reasons. Number one, they're simple, straightforward, and require no additional steps. Number two, some sort of redirection is required to support malloc and free. And virtual alloc will also require custom wrappers to be platform agnostic, so it's nice to sort of standardize everything. And then, macros also allow us to provide additional features such as location tracking and passing additional parameters directly to memory requests such as alignment requirements and requesting specific allocators. Converting to using macros is not a big undertaking. I've done it about three times in my career and each time it takes about a week. If you do not

redirect and control all allocations within your code, I highly recommend that you uh start and using macros is the cleanest method that I've found. And even if you forget to wrap a new or a delete, it doesn't matter. It'll still get picked up to overriding the system memory routines. And finally, to ensure we redirect all memory routines into our system, we simply control all memory. On consoles, we allocate all physical memory upon initialization. As a result, if we miss an allocation due to a third-party library, there simply isn't any memory for it to request. Talking about third-party libraries, whether audio, physics, uh video playback, or anything else, I think there are a couple of mandates that I believe that we should all insist upon. Number one being, if a library does not provide direct control for how it gets its memory, don't use it. If you can't control the threading behavior of a library, don't use it.

And if you can't control how it loads files, don't use it. There is one third-party library that I would like to call out that we use extensively, the EASTL, which is a replacement for the standard containers. I hope that anyone that has been in game development for any amount of time has uh some at least an opinion on the debate of whether or not to use standard containers. For us, we don't. However, we have found an awesome replacement that provides a lot of really cool features that, while not always safe, allow for smarter code. If you haven't checked out the EASTL, I strongly suggest you do. So, let's take a look at a diagram illustrating the flow of memory requests. As you can see, it all starts with overriding system allocation routines, which are redirected into our memory coordinator. From there, memory requests are funneled into appropriate allocators. These allocators manage their own memory pools to service memory requests and fall back to requesting memory pages

from the platform page allocator. Note that the only system that is platform specific is the platform page allocator. This helps to ensure allocators only deal with memory usage patterns and ensure that we get consistency across all platforms. So, the MC contains some of the hardest-hit code in the entire code base. Its primary purpose is to redirect memory requests to the appropriate allocator, all without creating thread contention. Thus, it must avoid locking mechanisms. We are able to achieve this goal by ensuring that the MC is basically stateless and that the few stack-based variables that it forwards into the allocators are all stored using thread local storage. There's one caveat to keep in mind. Our threaded job system is constructed using fibers that we switch in and out. Because we are using thread local storage, we need to prevent fibers from picking up the wrong set of variables when switched. So,

within our job system, if we perform a context switch and switch out a fiber, we also create a copy of the memory local thread variables. Then when the job system switches back to the previously suspended fiber, we we uh restore the local thread variables. Thus keeping everyone happy and consistent. By using thread local variables, not only do we allow memory requests to be redirected to any specific allocator, but it is also at the heart of how we categorize memory. In our engine, every single allocation is categorized, even if it is just categorized as a general allocation. Our strategy is once a category starts to account for too much memory, we simply start splitting the category and refining our tracking. And finally, the MC gives us a great single location to track all out of memory issues, eliminating the need to spread that code throughout the entire code base or throughout the allocators. One thing that you can see from the code snippet is that we create a special function

that is never inlined to handle out of memory issues. This has actually been really great. If QA ever encounters a crash, it is trivial to determine if it was an out of memory issue by directly looking at the extracted call stack. No need to look at the log or try to interpret line numbers. The allocator, or as I like to think about it, the heart of the system. This snippet shows a portion of the interface that all allocators must provide. The primary idea here that I want to call your attention to is the fact that we declare all memory routines as protected. Basically, we want to prevent code from directly bypassing the memory coordinator. This helps to ensure that all allocators follow consistent conventions and are properly registered with the memory coordinator. I mentioned earlier, anytime an allocator needs to grow its memory pool to service a request, it requests new memory pages from our platform page allocator. For Shadow on

the PS4, we determined that 64K pages were the ideal size. We also align all requested pages to 1 meg boundaries. We wanted smaller pages to minimize waste and fragmentation, but also minimize the TLB issues. For us, 64K pages was the sweet spot. I also want to point out that when coding allocators, we strictly avoid the use of mutexes. While allocators need to deal with threading issues, we stick to using atomic locks. These are significantly faster and the restriction of not being re-entrant is easy to work around. So, let's take a look at a few of the different types of allocators that we support. The dispatch allocator is the default allocator that the majority of all allocations get redirected to. It is a very simple redirector that basically looks at the alignment and size requirements of a request and forwards them to the appropriate allocator. We use a bucket allocator for small

allocations. We use a heap allocator, which is similar to DL malloc. It uses an intrusive red-black tree to store free blocks and uses a heuristic of uh preferring recently freed memory followed by best fit, then adds headers and post headers to the allocations to provide tracking information. And to round it off, we use a page allocator for large memory requests that fit nicely into our 64K pages. We also have several special use allocators. A simple ASCII allocator that is only used by tool code. There is a virtual page allocator to handle virtual memory requests. And there is a frame allocator, which is great for packing multiple requests together to ensure memory coherency and creating small blocks of reusable memory. And let's not forget about the tagged frame allocator, which is awesome. Our tagged frame allocator is based upon an allocator described by Christian

Gerling in his uh 2015 GDC talk. The basic idea behind the tagged frame allocator is that we are creating an extremely fast allocators where allocations are only valid for a short period time. These are basically temporary memory allocations that are valid for n number of frames where a frame is arbitrarily defined for each uh allocator. The fact that we don't need to track the memory to or call free to release memory is a huge win. Not only is the code requesting the memory easier to write, the tagged frame allocator can take a bunch of shortcuts knowing that individual free calls uh do not need to be supported. It is also impossible to create memory leaks with this allocator. As we discussed earlier, one of the main one of the primary goals behind allocators is that they should help to minimize thread contention. This is achieved using two strategies. Number one, we use atomic locks rather than mutexes.

And number two, uh by using thread local storage. Several of our allocators use the pattern illustrated in the code snippet. For the tagged frame allocator, this allows it to service memory requests for multiple threads simultaneously without ever locking. If you've ever had to deal with multi-threaded performance, this should jump out immediately as a huge win. Within the Bluepoint engine, there are more tagged frame allocators than any other type of allocator. Each tagged frame allocator is built on top of a tagged framed arena, which allows all tagged framed allocators to share a common memory pool, eliminating over allocation issues. The tagged frame allocator or the tagged frame arena will also grow and shrink upon demand. Thus all pool sizes are dynamic. If you don't have anything similar to this in your code base, I would highly recommend that you make it happen. And finally, the platform page allocator

or the brains of the operation. As we have discussed, this class is responsible for abstracting away the details of how 64K pages are managed. In practice, we restrict anyone from accessing this class directly. The only customers of this class are the memory allocators. On Windows, this is a very simple class and is built using virtual alloc. The PS4 version is more complex due to platform considerations. At the heart of both implementations is the concept of virtual memory and mapping physical to virtual addresses on demand. If you're not familiar with the differences between virtual and physical memory, you might want to investigate. So, looking at the platform page allocator, there's one last problem that I would like to discuss, which is a problem in every non-uber memory system that I've worked with. Given that we are avoiding an uber allocator and embracing numerous allocation patterns, we need to be able to take any random memory

address and determine which allocator it belongs to. This is required to properly direct free calls. While BPE macros allow us to optionally specify which allocator owns memory when calling BPE free, this is not ideal. Instead, we need to support the get memory owner routine. So, how did we solve this problem? We threw memory at it. We break up the problem by realizing that we only need to determine which allocator owns the memory page that contains the memory address in question. Within the platform page allocator, we use a 1 meg block of memory to store a direct lookup table, which maps 1 terabyte of virtual memory to its owning allocator. If you have a better solution, I'd love to hear it. So, what do you get when you put this all together? Hopefully, a game without performance issues and runs within retail memory. Okay, that isn't going to happen. But hopefully, you have the flexibility and tools necessary to get

there from a memory perspective. Here's a screenshot showing our memory statistics. You can see the platform page allocator at the type the top and each of the various allocators below. Here's another screenshot that shows memory category tracking. I want to call your attention to the outline black boxes that I've marked up. Most consoles have a concept of development memory. Basically, the development kit has more memory than the retail kits. We make it very easy at all times to see where memory usage is in regards to a retail kit. From the screenshot, you can see that the closest that we came to running out of memory on a retail kit was 130 megs or the equivalent of the max memory of four PS2 games. There's no way we could run out of memory. So, to finish up, I want to share one additional feature in Shadow of the Colossus is related to memory and made a significant impact on development. Texture streaming. I'm sure many of you

have texture streaming solutions. We use GPU-based feedback to determine what to stream and specify a fixed streaming texture memory budget. For Shadow, we initially set this budget to 1 gig and after some tweaking, most of the game looked great. Some areas required a higher texture budget to facilitate its needs. And some areas would simply crash due to running out of memory before even hitting the 1 gig budget. It was about halfway through the development that the team began to stress about memory and started getting flashbacks of previous projects. So, how did we solve the problem? We set the texture streaming budget to unlimited. Problem solved and now everyone is happy. Instead of relying upon a fixed memory budget, the texture manager monitors the total available memory by querying the platform page allocator and acting appropriately. If there is extra memory, stream in more textures. If

no more textures are required, then there's nothing to do. And if memory is low, start releasing textures. I should note that this does require a bit of balancing to avoid uh ping-pong effects and to determine proper memory thresholds. However, these thresholds are game specific, not area specific, so they only needed to be calculated once. This was really a major turning point for stability as out of memory issues basically disappeared. Of course, we still had to deal with memory usage issues where memory was scarce. However, these areas only needed to be pushed to an acceptable level and the texture manager would auto calibrate. Moving forward, we are pushing our texture streaming solution beyond what was achievable in Shadow by prioritizing loading even higher resolution textures when memory allows. So, here's a screenshot showing texture streaming stats at the top of the screen.

Here you can see that we have over 2 gigs of textures loaded. However, the scene only actually requires about 680 megs of texture data as the texture manager tries to maintain a buffer of about 220 megs free. Here's the more demanding area. There is 1.3 gigs of textures loaded. However, only 1.13 gigs are required. During development, the art team would play the game watching these statistics to verify that the budgets were satisfied. By the end, the only questionable area in the game was the final cutscene where the secret garden is exposed. And of course, the minute you tell anyone that the texture budget is exceeded, you need to be able to show them why and what is loaded. It is only then that you can properly debate how big must the eye texture be. Y'all get that? Okay, good.

So, once again, I would like to thank you for your participation. I'm flattered that I've had this opportunity to ramble on. And yes, Bluepoint Games is hiring. Basically, if you are a badass and you know how to how to get done, we want to work with you. Uh use the email address on the slide and mention my name. I'll let everyone know that we are best friends so I can get the recruiting bonus. And then perhaps it'll be time for me to buy you a beer. Uh also remember to fill out the speaker evaluation forms so I know whether or not I should ever do this again. Um with that, I believe I'm done. I believe we have 8 minutes for questions. If anybody has any questions?

I love it. The perfect talk. There's no questions. That's perfect. Let's wrap it up. Get the hell out of here. I had a question. Yeah. Uh you talked about being lock-free using atomics in your allocators. Uh-huh. So, we still live in a world where there's no standard lock-free container library that you can like Do you guys Did you use a third-party one? Did you roll stuff in-house? Uh it's it's Yeah, so for the atomic locks, we basically just sort of rolled our own um just using uh you know, the uh I'm drawing a blank right this second. Uh but basically just using the atomics that are available on the different platforms. Um it really comes down to like four or five lines of code. If you were to send me an email, I could send it to you. It's real simple. Sure. No, I mean did you you didn't do unbounded spinning though, did you? For some of your locks? Uh they will spin. Yes. Okay. Did you ever see stuff where like I feel like spin lock is it's all fun and games

until you know, your spinning core is scheduled in the work you need to be done that it's waiting on to get the lock back is is not, you know. Have you Did you see some of that? Not really. And And part of it is well as anytime that we use atomics, so we'll use the atomics anytime we want we got a really small portion that we want to get in and out of really fast. And we also try to lock it or try to make it so that we're really not calling other functions inside of it so we don't run into this case of I got a lock here, I call this function and then this guy wants to lock. And so they're not re-entrant. And so we keep those sections very, very small. And so we really haven't had any problems with them or having problems of something waiting for any duration of time before it could get back in. We also do a lot of like performance profiling looking to make sure that in order to maximize CPU saturation across all of our cores, making sure that they're just for that fact that there is nobody that's stalling or waiting for too long. Yeah, I guess I've seen when

spinning goes bad is usually when you're over subscribed. Mhm. I don't know if so if you Did you work hard to avoid over subscription on the platform and stuff like that? Yes. Okay. I would say we work hard to avoid it. Yeah. Okay. Cool. Uh you said your your next game is going to be more of a re-envisioning. I was wondering if we get some color on like what that kind of means. What is What is a re-envisioning versus a a remake? Sure. So, if you were to take a look at like uh Shadow of the Colossus, like it's a PS2 game, right? And so when you when that was built, the side of the hills were very low triangle density. Like what was the environment What was the atmospherics that they're actually trying to um achieve? And when you look at it, a lot of it you're I feel like your mind fills in a lot of those blanks. You look at a green card that's filled on the back and you're like, yeah, I assume that's a grass plane, right? Um and so for Shadow, there's a lot of us looking at that and then

building it out sort of trying to remake it but sort of figure out what the original was trying to do. As we move forward in some of our other stuff, we're taking stuff further um with not just, you know, trying to get the two engines to run simultaneously but more of bringing more and more stuff into our engine and repurposing our technology so that we have greater control of, you know, redoing the complete animation systems um upgrading, you know, besides graphics and other things but even starting to upgrade more of the gameplay paths and such, which are all for us are always very, very tricky. We've been very, very fortunate to work on games that we've absolutely loved. And for Shadow, like you wouldn't take that game and say I want to make it like Assassin's Creed running around doing stuff. Like you wouldn't do that. And so you want to maintain core pieces. Um so it's a very difficult question but it's just take Shadow and I feel like we're about doubling the amount of involvement that we're putting into it. Thank you.

Uh I um I would like to ask you uh uh how do you deal with frame rate issues especially with doing the sort of hybrid approach you took with like two game engines running at the same time? So, how do we deal with frame rate issues? We As I mentioned, anybody at Bluepoint has heard me rant about Digital Foundry at least a bazillion times because I don't know if you've seen their evaluations where they do the frame rate graph and they show it going straight across the screen. And I tell the team if we have a single dip, they're going to find it and I'll lose my mind. And so we do a lot of profiling. We our QA department is great. They take our our tools and and our third-party like PS4 tools and they'll have graphs that are running on their machine. So, they're playing the game with the frame rate graph. And the minute they see any sort of spike or anything like that, they go and they capture it, create a bug, and off it goes to engineering. Um on Shadow, we have a 60 FPS mode on the um

pro kits in performance mode. That was not easy hitting that. It's a lot of testing, a lot of looking at CPU captures, and exactly what jobs are on what cores when and why and, you know, trying to load balance that out. And for our next game, the big thing that we're really looking at is how do we maximize our CPU saturation. You know, I feel like we've got a good job system to break stuff up. But you find at the same time like you during your frame, you've got areas that are Here's the hotspot. Everybody wants this last third quarter of the frame and nobody's doing anything over here. So, a lot of analyzing and rearranging code in order to get it to fill in all those spots. Yeah, I I guess the second question is how when you convert from like 30 FPS game to 60, how do you deal with it like game-breaking issues? Oh, uh yeah. So, the original game ticked the update at a

hard-coded, you know, 30 FPS. And so on the PS2, I don't know to be honest, I don't know if the game actually ever hit 30. Um and so yeah, the minute we went to 60 or the minute we even locked it at 30, we found that you couldn't climb the Colossi as well. Your grip would fall before getting around some of the edges was twice as hard. And so I really got to, you know, give credit to that to our QA department. They test the game like no other and are are looking for parity issues. They identify those sort of issues and then we go back and we fix them. And a lot of times programmers love when it when they look at a bug and they're like, I don't know why it's behaving this way. And it's like, well, go play play it on the PS2 on a PS2 dev kit. And it's like, I don't even know how to make that thing run anymore. Right? But it has to be done. Okay, thank you. Um hey, great talk. There's a lot of issues that speak close to my heart. So,

I was wondering if you have any solution, any utility libraries, or whatever to deal with the 32-bit to 64-bit address space conversion? Any libraries that deal with 32 to 64-bit conversion? of like injection or, you know, remapping of the 32-bit address space into 64-bit or something like that to ease the process of porting and moving to a 64-bit platform. Yeah. Um We don't really use a third-party library to do that. It's It's a lot At the end it comes down to us doing a lot of hard work and updating code. The one thing that I would throw out that I didn't really talk about, but is mentioned in the slides, is what we use is we call it a T offset pointer, which basically allows us to represent any address as a 32-bit number. And then when you actually do the lookup, when you dereference it to a pointer, we actually add an offset to it. Um we use that a lot in like memory-mapped files that have to do that so that we don't

break everything right out of the gate. Um but even then by the end, they get annoying because now every single pointer dereference is basically a dereference and it's adding and I don't know. It's It's stupid like that that just like, why are we wasting processor power? Drives me crazy. Um and we end up going through and fixing most of them. All right, thanks. Yeah, I'm curious about you have the Bluepoint engine and then you've got the game engine. And I presume that you take the Bluepoint engine from project project. That is correct. And you have the the game engine, does it call directly down into the Bluepoint engine? And how do you keep the Bluepoint engine side of things from not having references up into the game that have to be cleaned up from project to project? Right. So, you could say I've been writing code for a really long time and you could say there's certain things like coding standards, you know, it's like a religion. What what side of the fence do you fall on? There's certain things with coding standards

that I strongly believe in and drive me crazy. One of those is dealing with library dependencies. If I've got a base library and I've got my game library, there's no way in the code base that I allow the game or the core library to start calling up. And so we make very explicit rules about what library is built on top of what so we have a very clear hierarchy. Then it's going through and we'll add layers in the middle that's sort of like, here's a wrapper piece that is basically the bridge between shadow code and the BPE engine working with our components and game objects. And it can basically go and do like reach into shadow code or shadow code reaches into it a lot. And then I'm I'm okay with that because at the end I know that whole piece gets thrown away. And so I don't incur that debt. So, it's essentially just a wrapper for it so that the game can preserve its original engine calls that wrap to calling into the actual Bluepoint engine. And if you think about

it, like any game engine, it all boils down to you've got a main loop somewhere. You've got a forever loop that's looping doing whatever. And then the Bluepoint engine similar is got a forever loop that they're both basically looping. Well, now rather than looping, I'm just going to make this one when it loops, it calls in over here, does its thing, comes back out over here, goes back at the top and calls. And so it just sort of ping-pongs. And then the next thing you got to do is come up with areas of responsibility. And so then you start going to the original game engine. You're like, no, no, no. You don't do anything with audio. You don't do anything with rendering. You don't do anything, you know, dealing with I don't know, scene management. Um that's all going to be over here. Um that's that's the the hardest part is defining all those rules and levels of responsibility. Thank you.