Making "Big Data" Work for Halo: A Case Study

Awesome. Well, thank you everyone for making it out here. It's good to see people. Doors are closing. Uh house cleaning tips, please silence your cell phones. Um and then because I may forget later once we get to Q&A, make sure you step up to the microphones. So, quick show of hands. If I could get some audience participation, how many people here uh just plain up gather telemetry? And I'm hoping that all the hands go up. I'm seeing about 70%. And then lower your hands. Hold on. Uh how many have access to reports by end of week? It So, that How many people are more than a week before you can see the results of your telemetry? Okay. So, it went from like 70% to like 30% and

then 0%. So, I'm not sure what that 40% delta there is. Okay. Um how many people get access to their information sub minute from the telemetry that they're gathering? That's awesome. We should definitely talk afterwards. I want to hear what you're doing. That's cool. It's a hard problem. So, uh my name is Tom Matthews, as you can see. I started in the the mid-90s working on line of business applications uh for Best Western, Starwood Lodging, Universal Studios. After about 5 years there, I moved on to Microsoft uh working on SQL Server Analysis Services, which is a multi-dimensional database architecture. 5 years into that, I moved on to the Advanced Technology Group helping game developers improve their audio implementations. The 5 years into that, I moved on to 343 Studios to help them

maintain the existing Halo architecture for gathering telemetry and architect the what you guys are here to see, the subsequent iteration of telemetry gathering for Halo 5. So, if if you've noticed a pattern, every 5 years I seem to do something different and that was about 4 years ago. So, I'm no longer a software senior software engineer engineer. Uh I am now going to be a senior data and applied scientist. So, uh make sure you fill out the survey. So, when I come back next year to talk about all the cool investigations we've been doing. But, enough about me. We'll start out with a bit of uh interesting stuff. Uh some of the the numbers of our launch and then we'll go into how we actually achieved So, events per second seems like not a lot of people talk about it. Uh it was

hard for me to find publication numbers. I found one large casual app that was saying that they hit 2 billion events uh on their their launch day when they launched their big popular app. And when we started working on what kind of architecture we would use and land on, what kind of technologies we would use, that was about 4 years ago, 3 and 1/2 years ago. And I was putting this talk together, I was like, "Oh, did we really hit big data scale?" Uh I mean, it seems like we did from 3 years ago, but times change rather rapidly. So, uh for our launch day, what we were able to do is uh 700,000 events per second, which is 2.5 billion events per hour. So, that's a pretty significant amount of of data uh churning through the systems. And our peak peak volumes were 831,000 events per second.

The funny story there is that half of that volume was actually one particular event a developer had left turned on for release, which was a composer event that was firing for every object in every frame and resulted in in 400,000 events per second for the first few hours until we we finally got around to to turning that off. So, first first finding is make sure you're aware of the data that that your game is is transmitting in the release configuration. And the the next highest event is is down at 90,000. So, it's a pretty exponential tail that we have going. So, overview of the general slide deck presentation that we are going to be presenting today is where we started, like what the the Halo 4 and earlier code base looked like. What goals we wanted as we looked into

overhauling that and creating something new. The implementation, naturally, and the results. There will be some some findings at the end. So, the existing uh at 343, we inherited the Halo code base from Bungie. So, this was the existing implementation of logging. They had two different logging systems. One was this log event macro. And this macro was very easy for developers to use. It was like a printf statement. You would key in the string that you wanted printed out over Telnet or to the screen or to log files and the parameters that would go into um that that system. And then it would feed into a database

architecture called called DataMine. The The good thing is that it was easy to use, right? So, developers could just write using the smack of printf statement. Uh the bad thing is that there was not really strong typing. They could pass anything in there and printf would do the best uh that it could to create a string that was fairly legible. And uh there wasn't enforced parameter naming. So, there were several times where developers would swap the events uh seek the order of the parameters in the event. Uh and so the meanings of the reports that were coming out would would change, right? And there was no warnings that something was was going on different. Uh other times they would change the um data type itself to be more or less precise. So, they would go from like a uh map coordinate system that's in world units to a normalized map coordinate X

and Y that was minus one to positive one. And naturally that would mess up reports that were were based on that. And the the worst part um about it is that what it did was it actually zipped up the results of all the telemetry generated during that particular match and upload that zip file at the end of the match. So, it was incredibly slow. During internal Microsoft wide betas, uh you're looking at up to several minutes to upload the telemetry that was generated and players are just sitting there waiting for the next match to start. So, naturally this was compiled out of release. So, there were on the order of 9,000 events that were using this log underscore message or log underscore critical format. Uh but you didn't get any of those event information from the release build.

What they did have in the release build and and in the development builds was the binary logging. And so, this is uh the binary log format BLF files. And what they they pretty much were was um they were good from the perspective that they were they were more compressed. The devs found it very easy to use. What it was was basically just a mem copy of the struct. So, they would make sure that it wouldn't be doing any reordering or padding, and then they would mem copy that struct off and uh send it on the wire. The The downside is that this is really really frustrating to use from a services perspective because as soon as a developer adds one field in the middle of their struct, now the C# code that's trying to like read these two bytes and interpret it this way and read the next three bytes and interpret it this way, uh that starts giving you garbage results. And hopefully, you have the testing that identifies that bogus values are coming through. And hopefully, the change that

the developer did in the client uh results in bogus values that would trigger that alerting. So, the big pain from the services side, besides the fact that uh stuff could change on the fly, is that it requires the source code to understand. And uh depending on the level of commenting going on in your source code, that could be easy or very difficult uh for a services engineer to understand the intent of a particular field. And similar to the previous logging system, uh this is transmitting at the end of the match. So, when the match is over, the game takes all of the statistics that it's gathered and uh aggregated and uploads it at the end of the match.

So, when looking at where we wanted to to move to, uh one of the big key things was thinking about those easy-to-use printf-style statements in the first logging system. And trying to come up with a way that we could use that in release if those events were turned on. So, having a configurable system that would generate telemetry in a performant way. But, the real goal of this is to um what I was calling in in the the slide summary, uh gaming intelligence. So, we've all heard of business intelligence, which is the information that the business guys want, right? You know, return on investment and ARPU and um churn numbers and that kind of thing. So, they're looking at, you know, new unique users coming in, that kind of

thing. But, what we really wanted to do is make sure that developers had available to them the data that was most valuable for ensuring that the systems that they were writing continued to operate as they expected. So, towards that end, we were going to focus on reliably transmitting telemetry uh in real time, uh enabling sub-second service response. So, one of the the earlier requirements for this was uh having the player do something and within 1 second having from the the user's perspective, from the instant that they started an action, having services perform a calculation and then having that reflected in the uh title and the user's perception. So, you can think of it as um you know, metals essentially. You you you get three kills in a row within a certain time span, we want the

services to be able to be tracking that and then show the metal within 1 second in a timely fashion. So, what we have is uh the the rough budgets that we we sketched out on the back of a napkin essentially when we first started out was that uh for the highest priority events, we would want uh message pump that was firing uh every frame. Uh primarily because we're supporting up to 400 milliseconds latency. So, if our goal is to be responsive in a 1-second time frame, if that's our SLA, 400 milliseconds up and 400 milliseconds down, that's gone right away. So, that leaves us 200 milliseconds in client side and in services side to do our operations. So, we want to transmit as fast as possible. If we're waiting even two frames or three frames, that's eating up a good 20 30% of the budget that we have.

Once it's in the cloud, there's a queuing and dequeuing stage that has to happen in order to uh support uh horizontal scalability. So, that we're budgeting at 50 milliseconds. Fortunately, uh the the queuing system we're using uh comes in much under that. And then service code it basically leaves at 70 milliseconds with a tiny bit of head room and then you've got 50 milliseconds to throw back on the queue and 400 milliseconds transmission latency again. So, for the purposes of this talk, uh the the component that is that I'll be referring to as cell is um the client-side component and the component that I'll be referring to as maelstrom

is that those those bottom three boxes. When we were going through this, there was um uh storm and spark. Um you know, they were like research projects cuz this was 3 and 1/2 years ago and you're not sure which one is actually going to become adopted by the community and which ones are going to stay uh university projects. Uh but they all were having um meteorological-themed names. So, that's that's one reason why we settled on these. So, in the logging architecture that we're creating, there's there's three primary requirements that we have. The first one is sequentiality. The reason why this is important is because in a lot of the um real-time modeling um processing systems that are out there, you'll actually find that they require

you to have a window of operations. So, a lot of these systems, what they do is you'll collect for uh 5 minutes and then what you'll do is say, "Okay, as of as of this point in time, I'm going to sort all of the events that I've received. I'll take the oldest 2 and 1/2 minutes and say that anything that comes in more than 2 and a half minutes ago uh is now going to be thrown out because I've waited 2 and a half minutes. That's the latencies that I'm willing to to work with. And uh I know that anything older than 2 and a half minutes is is bogus. So, I'll work on the world view that I have with the events 2 and a half minutes out to 5 minutes in the past. And then it'll do its aggregation and calculations and then 2 and a half minutes from now it'll do the same thing. It'll sort the last 5 minutes, take the oldest 2 and a half minutes, and process that again. What Cell and uh Maelstrom

uh implemented, what we implemented with Cell and Maelstrom is the concept of uh for high-priority events we guarantee at least once sequentiality. So, uh it's kind of like the TCP style. We will have acknowledgements. If the host is not if the uh the cloud service is not responding with an ack in a timely fashion we will disconnect from that um service and then connect try connecting again, which hopefully with the load balancer will give us a host that's um or I I should say a a cloud service entry point that is in a healthier state. And retransmit the events that we had been expecting to get ack. So, you'll get repeats but from a consumer perspective that's reading this event stream uh if they see an event they haven't seen before they can act on it immediately. There's

no ordering that they have to do uh and that really improves on the latencies that we have in the service itself. We also have a low-priority stream and these two concepts are very decoupled um high-priority streams and low-priority streams because we don't want something like a, um, super spammy event that should be off to affect the high-priority stream. So, anything that's user-facing is going to go on that that high-priority stream, deaths, kills, um, which are the same thing, uh, but spawn locations, that kind of thing, those are high-priority. Oh, what one one thing to note is that the low-priority stream, that's more of a UDP style. So, it's an at most once transmission because there's no acts on the low-priority stream, um, because having to maintain that much state on such a high volume would be, um, a bit catastrophic.

Contextual events is also important. So, the events that we transmit are very small. And because we're working with the concept of a streaming architecture where we know the events are sequential and we are subscribing to the events from the beginning of the stream, um, the consumers and they're guaranteed, the consumers can primarily be paying attention to the client session. So, when the client spins up and connects to the cloud service for the very first time, it will be, um, creating its own unique GUID that's associating itself for the lifetime of that executable. With that, um, one piece of contextualization,

that's associated with the stream. So, each individual event doesn't get that GUID attached to it. The, um, a and I've got numbers later in, the the reason why that's important is because adding even one good to an event would increase our event sizes by something like 20 to 30%. So, our events are so small and compact that adding a lot of contextualization information would be detrimental to the point of having this very fast system. So, the consumers uh it does shift some of the um processing work onto the consumers of the streams because they have to monitor for a match start event and say, "Okay, from now until I get a match end event, the events that I'm getting are actually associated with this match. Um and if they wanted to do contextualization on um the duration in which a person was

alive, they would also have to track spawn to death events, that kind of thing. But, if you work with contextualization from the beginning, that means that uh you're thinking in terms of contextualization. So, it's it is a tax on the services side, uh but once they start adopting that um mindset, it becomes easier uh to be creating these kind of uh contextualization checkpoints. Uh and some window processing windowed processing architectures like Trill um make it easier uh to identify new boundaries by creating new events that are injected into the stream. And finally, there's uh compatibility. So, I was I

was I alluded to this earlier with the other two systems where developers making changes to the events uh can have very deleterious effects to the services downstream. So, the architecture we wanted to implement from the the beginning, the concept of having um events that were both forwards and backwards compatible. Meaning that uh as you create your events, the events will be uh you can continue to add new fields to them, but um you can't change the types of the fields that you've already transmitted. And this allows us to make our reporting systems such that the reports that we generate will continue to work. I mean, there may be nulls there because people stopped transmitting old fields kind of thing. You still have to handle that, but uh you don't have bogus data creeping into your your code paths.

So, CELL stands for Common Event Logging Library. Within the CELL architecture on the client, uh we have a component called the Telemetry Manager. And you can have a number of these Telemetry Managers. In In Halo 5, we have a high-priority Telemetry Manager and a low-priority Telemetry Manager. But, you could create one for for different um services. For example, um some studios that are adopting the technology, um they have one manager that's for their own personal services, and they have another manager that's for the um Hm. I forget what the the latest name is, but the uh event tracing for Xbox or ETW style eventing um that that's on the the Microsoft platform. So, there's a Telemetry Manager. Uh Uh

developers interact with this using a macro uh that's been optimized down so that the client-side impact is 15 to 30 microseconds. And I would get um calls uh if that ever crept up to 50 microseconds. So, we we spent a lot of work uh optimizing that that macro. And what it's doing is it's um pretty much just it it it has very two very small branches to determine if the overall category of event has been turned on and the overall priority threshold of the event is turned on. So, uh in release, we can set a priority threshold and we can um turn on or off um the categories uh that will be emitted. And that's the kind of like nuclear option. That's the the most effective and broadest scoped configuration option in our on the client side.

And it's also the fastest. There are other filters that you can apply, but those have to be processed on every event as it comes through. You have to crack open the event on the client, and that that consumes more time. So, that macro is uh essentially mem copying onto a uh circular lockless buffer. And I put quotes there because it was originally implemented in a lockless fashion, and uh turns out that that lockless programming is is hard as all of the books say it is. Uh you'd think interlocked increment, you know, what what can go wrong with that? But uh you have to do a lot of um fencing and thread marshaling. And uh yeah. So, we addressed all those issues. We're pretty sure. Uh but for the preview, they didn't want to take the risk, so we threw some locks around it. And that 15 to 30 microsecond uh number up there is actually using those

locks. So, we didn't take the locks out because we didn't have uh performance requirement to remove them, and it was keeping it safe. Uh so, we actually left that in. I I I think we could get a bit faster performance. Um but at at these speeds, you're pretty much waiting for uh cross-core synchronization issues, right? Which you have whether you're dealing with interlocked increments or if you're you're working with um fast lock semantics. So, might save a a few microseconds off. The um within the telemetry manager, there's a concept of an endpoint, and there's a collection of endpoints that you can create. Uh we've got network endpoints and Telnet endpoints and uh screen endpoints that display on the screen if it's critical or higher. And that's where I was saying each endpoint can, as it pulls an event from the circular buffer, uh and it's working directly with the copy that's on the circular buffer. So,

it's not getting a copy and incurring that overhead. So, theoretically, if you have a slow endpoint, that could become a bottleneck and and and halt the the telemetry gathering system. Um but we have an awful lot of testing um in the studio to make sure that that's not going to happen. And in development time, we run with uh probably I want to say 10 times the volume uh 10 to 100 times the volume uh as the re- retail uh configuration. So, we're we're putting that a lot of stress on that circular buffer already. So, the one thing to note, if I go back one slide, uh the uh message pump on the network endpoint.

So, we don't have a Telnet endpoint turned on in release. Uh the the main endpoint that we have is the the network endpoint and uh an endpoint that that is integrating with the Microsoft uh achievement system and presence system. And our budget uh that we have is um 1 ms uh per frame on a single core. So, we've decoupled the system in that the C++ macro is very very fast in all of the different client subsystems that we're interacting with, and then we have a worker thread on a a low-priority core that's waking up um every frame to pull off those events that have accumulated in the the circular buffer. And with that, our targets were uh 2 to 4,000 events per second uh before the circular buffer would would fill up uh and keeping to the 1 ms per frame CPU guideline,

and we were able to hit uh 3,000 events per second. So, it worked out pretty well. So, to enable this, we have a um build time preprocessor. So, the macro covers that ease of use bullet point that that was important to us at the beginning. Um and it and it looks a lot like the log underscore event macro. The preprocessor, what it does is create the uh a global schema store. So, at compile time on a developer's box, the preprocessor will run as part of the build step. It's a pre-build step. It'll parse through writable code as an optimization um uh Um, that those are the the checked out files that they may have edited. And identify if there have been new event parameters created or if there have been new events that haven't been registered in the global database. Um, this is configurable. Right now

we're using TFS because it allows us to do um, ADF security credential stuff and we can share with with studios outside of the Microsoft internal corp net and they can still access the external public TFS address and all that. So, the important thing here is that events are the schema store stores events at the category level. So, if you have two networking engineers that are adding events, you need to create a unique identifier for that event type that they're about to check in. Uh, and you want to make sure that they're not stopping on each other and both saying plus one to whatever view that they have. So, the the global schema store supports that. It also supports the enforcing the ever-growing schema requirement. So, uh, which which makes sure that the parameter types

aren't changing in an in an enforceable way. You know, they're not going from a UN64 down to a a byte and then all of our reporting infrastructure, um, I guess that would be supported going from a So, we just don't let them change the types, right? Because figuring out that logic and trying to message that, uh, is a bit challenging. But, uh, that allows us for it to have a reporting stay consistent. Now, this was a big pain point, uh, unfortunately, because developers, sorry, my my slide timer is like 20 minutes ahead of the real world time, so I had a little freak out there. Uh, developers were uh, iterating on the events that they wanted to generate as we all want to do. And uh every time they were compiling, they were compiling with each iteration, which was registering their event in the global schema store, which means that now they can't change the event types cuz they realized they needed a UN 64

instead of a a UN 32. So, it's on the books. We haven't implemented it yet, but one of the things we need to do is towards an offline um do not transmit this data to the cloud because we're iterating on what kinds of events we will want to transmit. The reason why this defaults to the behavior that we implemented is because we would have engineers that would create uh local one-off builds with a bunch of instrumentation, transmit that run the game with that instrumentation cuz they're trying to track down some issue that's that's hard to to debug. And then want to report from us saying, "Hey, in the cloud, you know, I'm I'm pulling up the session that I just ran through and I don't see my data, right?" So, there's the power of this is that developers have access to their data uh in less than a minute. They have access to the full log of events that's being

generated in real time. And our developers are most {slash} some uh developers are making use of that particular feature. Um so, and it's a really powerful feature. The ones that use it are are strong advocates for it. And finally, the preprocessor uh handles the bond serialization. And we're using bond, which is a Microsoft protocol that's open source, and it's similar to Google's protocol buffer. And so, it does bit packing. Like in the the first four fields, the first four fields on an event, their ordinals, their position in that that event one one two three four is actually stored in like the first couple bits of a particular entry within the bond serialization format. And then the next few bits store the the type of the entity. And this lets us get away with an average event size of only 120 bytes.

You can't do bit packing on goods, which is why if people are adding in match ID and and vehicle ID and life instance ID and all these other things your your event sizes start start getting rather large. So, that was cell. That's the client-side architecture. On the server side, we've got mailstrom. And this is is the mailstrom component is is fairly small because it is the it's small in concept, I suppose, but not not necessarily in implementation. Because its focus is to be the event pipeline that all of our other services can tie into and work with.

So, we have the ingestion service and this is the primary service that the clients are connecting to. The clients, what we're using is web web socket protocol to connect up to the cloud to the ingestion service. We're using a single web socket connection for both the high and the low priority channels. Which which adds a little bit of complexity. And in order to multiplex over that that path, we're using AMQP, uh which is a um queuing protocol. Um So, clients are connecting to the ingestion service, and then the ingestion service, as it receives the events from the client. Now, remember I said we weren't transmitting the uh session information or the uh authentication claims uh with every event payload every frame,

what we're instead doing is transmitting those during the connection negotiation for the web socket, and then the ingestion service keeps track of that as part of that that web socket information. It will then stick that onto the payload that it receives from the client and injects that into the um event queue that we're using in the cloud. And that way, whoever's reading off the data, then they can have access to that session good and say, "Oh, this is something I've subscribed to. Uh I want I want this event and this event and this event um off that queue." And then we're we're transmitting uh our our primary uh queuing uh system that we're using is the the Azure Event Hub, which has performed uh quite well uh for our needs. Uh it's it's stood up to some pretty high-volume stuff. We also

in the ingestion service uh support non-AMQP paths. So, we're getting telemetry from other components that might only transmit uh JSON JSON payloads, for example. And so, the ingestion service will take those JSON payloads and convert them into bond. Those are much lower-volume streams, and so it's something we can do in the ingestion service. But for the most part, the ingestion service never cracks open any already encoded in bond payload. It's sole purpose is to take the claims and the session ID associated with whatever it just received from the client. It lets someone else deal with um authorization. So, ingestion service will do authentication to ensure that you're allowed to talk to it. Uh but then the authorization is offloaded to whoever's going to actually use that information.

So, I mean, that's Maelstrom. It's the event hub in the ingestion service. Uh mostly. So, there's also the common API we have for consuming those event streams. So, the API layer that allows you to connect to that event stream on event hub and then allows you to read the stream of events coupled with the global schema store I was talking about, right? So, we have a global registry of what every event um should be interpreted as. Field number two, it's an int. Okay, that's part of the bond protocol. But understanding that that is uh weapon ID, for example, that's metadata that's stored in the schema store. So, we have a common set of APIs that allow for uh understanding of the event streams as they're coming off. Then we have several services that feed

off of the Maelstrom pipeline. The uh stats service is the primary one that is in charge of the leaderboards and the end-of-game stats that you see, hopefully, at the end of the game. Um and the storage service is the one that uh takes off event Hub um because that that retains for so many days into the past. Uh you can recover your services anytime within like a 7-day window. And as long as you've persisted where you were in the Event Hub queue, you can spin everything up and and read everything out again. Um make sure that you're using the right offset. Uh funny anecdote time. There was a partner team we were working with that uh always spun up their service using offset zero. So, they were reading from the beginning of time. And um at that time there weren't uh very many

safeguards to to throttle the reading. So, it kind of took took things down during the um the Halo preview. Uh so, lesson learned. Uh make sure that everybody's using and persisting uh their offsets as part of their checkpointing process. Uh and also make sure that you've got throttling uh naturally uh in there. So, storage sends stuff off to blobs. And then later on, as I'll be getting into, we can crack open those blobs uh for doing some some larger scale reporting. The stats service is essentially a state machine. So, since we don't have to do any of the windowing processing, um I think at one point they did have uh Trill implemented so they could do more complex um window processing aside queries as part of the event stream. Um but at its most basic level, it's not necessarily necessary. They just read the events and um

increment their internal counters and persist their state um using I I didn't go into it. Uh we have several talks. If you go into I think the GDC um um vault as well as the I think we've given talks at Build that cover how 343 uses the Orleans technology, which is another open source technology bed that allows for um these things called virtual actors, which are essentially your your uh they call them stateless um but stateless state machines because they're they're state machines, but then they can migrate as the servers go up and down and everything's handled for you and it's really cool tech that we leverage a lot in 343. So, I recommend looking into that. Finally, we have the uh librarian. And

what the librarian does is it's cracking open, finally, we're cracking open these events uh from a um a BI perspective, right? So, the the stats components are cracking open the events that they want to read and they're subscribing to the events that they want to subscribe to in order to drive the end of game uh leaderboards in a in a real-time fashion. But, they're only really subscribing to the high priority feed, which allows them to shave off like 80% of uh the network volumes and processing requirements that they would otherwise have. So, the librarian what it does is it's actually parsing um the events and we've we've stored just enough information in the headers of the events. I mean, Bond is really great in that you can deserialize in the event stream just as much as you need and then skip the rest of the payload

and not incur any of the overhead in deserializing the rest of that event. So, our events are architected such that um the information you need, the event type ID, of title, ID, uh the session information timestamp. Those are all in the very first few dozen bytes. And so, the librarian is parsing out that header and making a note of what timestamp different events happened. And so, now we can do reporting on the session observed this many of this kind of event. Uh and then if that's you know, if we're looking for a very rare event, then we can say, "Well, just give us the 20 sessions where this event happened." For example, and then we get the full context. Finally, we've got the telemetry event viewer. And the the telemetry event viewer is a is a work in progress. So, uh right now it actually just outputs a CSV file that developers can open in

Excel. I wish I had a screenshot of it actually. Um because it it gives you a lot of data, uh but it's it's essentially the event stream in human-readable format, but also with every column uh each field has been separated into its own column, which allows you to do filtering and sorting and what have you on the fields within those events. Um and this has been an an amazingly useful tool in debugging what exactly is going on in the game, especially for like the networking guys, because uh networking is inherently a um integrated problem. So, you've got 24 clients and a server and several services that are all interacting, and as soon as you hit a bug, then you've got to figure out what it was, what were the states of all the different actors at the time of that

that negative event happened. So, with the telemetry event viewer, you can actually pull down all of the events from all of the clients that were connected within, you know, like a 5-minute window from when the event happened, and you can see what everybody's doing up to when the event happened. So, that's been a really useful tool, and because we're flushing every 30 to 60 seconds for the larger blobs in storage, we're able to pull up sessions as they're they're live. So, somebody's saying, "Hey, you know, I'm getting some warping going on in this game." We can actually load it up and see what's going on in in less than a minute. The corollary to that, one reason why we we have to support so much volume, is that the networking guys generate the most

events. So, there were something like 9,000 events in the legacy system, the old system, that that was log underscore event. We migrated probably 1,000 events into cell before we we launched. And of those, I would say probably 3/4 of them are networking related just because they have so many things that they need to track. And then for doing our big data processing, we're using Hadoop and Hive, and we have a custom Java bond surday that is reading the raw blobs. So, we have a job every hour that runs that traverses through the raw blobs that are that are stored in bond, and converts the events that we're interested in into the orc format, which is

it's It's columnar format. So, it's very uh if your data is slowly changing, it's a very highly compressed and efficient way to uh to to to store your data. So, if you have data that's like um if you always have a map name in your event, then it only stores the map name once and then has like a vector that says, "Okay, well, this vec this value is like run-length encoding. This value applies for the next um 20,000 entries for this hour." Uh so, it it creates very small files and it's fairly standard in the the big data world. Um and I I just spoke to all of that. And then we we've got ad hoc uh and regular reports that are running in in the Hive query language uh and we're also using um Tableau and R for visualization tools um and to to create our our dashboards.

So, some of the gotchas in implementing this. The client implementation took time. Uh it took a lot of time, I have to say. Um I think we we originally anticipated it uh to be, you know, something that was on like the 6-month scale and I think we ended up spending um most of the 3 years continually working on the client-side implementation and ensuring the performance was right. Um and a fair amount of that was working with the the the build integration systems um that um that are part of of game development, right? You know, somebody breaks the build and you've got to integrate and you've got to figure out was it you that broke the build, etc., etc. So, um a lot of people that I've been working with are more on the services side. So, helping those teams understand the

implications of doing client-side development, uh making sure that they understand how absolutely critical it is for a high-performance, low-latency, client-side impact uh was pivotal for the success of of making this happen. We wouldn't have had client team uh buy-off if we didn't uh convey how much we understood it was important to keep this at the microsecond level. Uh there's this thing called a statistics event that I I haven't spoken to until now. Um every minute the client side writes a little histogram event into the stream that says, "Here's all the events that I've seen so that we can do matchups to say, 'Oh, well, we didn't um it was a high-priority stream, we shouldn't have any dropped events, but apparently we did drop these three events. We need to go investigate that.'" Um we actually had to turn that off for uh release because um actually, no, it is in release, but it's got an issue in

that um every event is sequentially incrementing its sequence ID, right? Everything is sequential. Uh the um statistics event is attached as part of the header of the event, but its sequence ID is allocated at the end of the event because the batch it's about to send it wants to report on it. So, uh we had some issues uh with people that were doing the right thing and making sure they only interpreted sequential events, but they were having all sorts of dropouts because once a minute this particular event would show up out of sequence. So, whenever you're doing these kinds of custom um scenarios, uh you really have to to keep track of all the different systems that that could be impacted by it. The decision was made uh from the beginning that a client session would be the instance of the executable from when the

executable started to when the executable was terminated and released from memory. The thing we we didn't really account for is that on the Xbox One titles go to sleep. And it could be a week before the person comes back and fires up their console again or they could go off and play Netflix for several hours or days and then come back and start the the title again. At which point the title instantly comes on. You don't have to load the the waiting screen or anything, but you're using the same session ID. And that's caused a fair amount of headaches for our reporting systems because now instead of carrying forward the information from the previous hour and aggregating that information now like if they're playing a campaign game and they're they're mid-game and then they go to sleep now you wake up a week later you need to figure out what map

they were on. But because of the contextualization decisions we made that information is stored a week ago, right? So um that that was a headache and it's still not solved. We we just have to we basically carry forward summary information for players as a stopgap. Um And then things came in a bit hot and we needed to get our actual business intelligence reports going. So the stream querying that we really wanted to make a lot of use with. So we're using the non-windowed streaming analytics in our stats grains. But we haven't had an opportunity yet to implement trail and some of these other technologies like Esper is one that we've been looking at to have kind of business intelligence driven streaming queries off off the stuff.

And as I said earlier the schema store caused some problems during developer iteration. So running low on time but I do want to show you some some cool graphs, right? The um map balance. So I don't know how many people here saw the the community um news article about red team versus blue team. Awesome. Great. This is going to be new to all of you. So there was there were concerns in our community that our our maps were not balanced red team versus blue team. And indeed if you if you look at the the raw numbers you can see that blue team seems to be doing they These are wins versus red team wins and blue team seems to be winning more frequently except for the line at the very top. And this is all on one map. These are

just different game modes on that map. And then the previous one was all up across for that entire day. This one is broken out by um Let's see. I think the previous one was was all up for the entire week. The next one is broken out by day but significantly it's looking at even team versus even team. So it's taking out the noise that can happen if a really strong team uh is playing one color consistently for some reason which which did happen. And so these are these are even teams that we're comparing and we see a lot of blue there. And then you can also do something cool like um looking at what is the uh matchmaking rating um delta at at which you now get an even win-loss rate.

Uh so with that, you can see that in some game modes um red team has to be several It's like 0.3. Uh let's see. I don't I don't really have a pointer, but some of those lines indicate that you have to be 0.3 mu uh better on the red team versus the blue team before you start having an even match. Uh and so we wanted to look into this in our uh hypothesis was that some of the maps um allowed for blue to blend in easier. In fact, we think that blue is just generally a um harder color to get target acquisition on because red is a much more high-contrast color versus blue. So, we were looking at um heat maps, and I'll I'll speed up through this. Um So, deaths. Red versus blue. Um there's more red deaths there. Uh but it's kind of hard to make out. Um

a pattern doesn't really show up well. Uh so what you can do is you can look at traversals, and you can see that red base is at the bottom, and blue base is at the top, right? So, on the previous slide, there's more reds deaths on the bottom. But that's red's base, right? So, I mean it it makes sense that red would die more if they're spawning down there. And so you can get this uh lethality um matrix. Although looking at it at at the individual dots is uh it's hard to make out, but you can see that there's some patterns starting to emerge. So, uh what I did was we've got these things called callouts in the in the maps that uh tell you where you are in the map. So, you can say, "Oh, there's somebody in upper catwalk or lower catwalk or or what have you." So, I actually walked the borders uh of these callouts just as a prototype and uh outputted my my XYZ coordinates at a

regular interval, like every second, uh threw them into Excel so that I can make sure that I was collecting the right kind of information. Uh imported that into SQL Server using a um uh two-dimensional um They've got a 2D query uh extension that's part of SQL Server. So, you can do uh bounded queries. And then um correlated the traversal information with the death information to say, "Okay, inside this polygon, how many deaths happen?" That kind of thing. That resulted in uh this heat map, which is much more useful, in my opinion. Um and because I have the Z-order, I can layer the polygons one on top of each other. So, you can see the catwalks are above the courtyards. And now you can see that red is not having the greatest time. Um red seems to be dying uh regardless uh now that we take into account uh traversals, especially when red is

entering blue base and blue is finally dying when blue is entering red base. And the walls have a slight hue um that uh make those opposing colors much more visible. Except for this little blue circle, which is really, really strong in the bottom right. Uh there you go. I circled it. And so, I loaded up the map and I'm like, "What's going on with this little circle? That's that's curious." And uh there's this this big red wall right behind this little platform that people can stand on and shoot. So, um it was kind of a nice confirmation that the hypothesis of of hue versus background color and what have you was was being uh born out. So, we took that to the map designers and uh and they what they did was they said, "Well, so this particular map uh we're at like 54% blue win versus 46% red."

And they said, "Well, okay, there's there's a bug where uh the best player always uh in particular scenarios the best player is always put on the the blue team. So, uh maybe that's what we're seeing, right? So, it's like fine. We we waited for that fix to go out and um Did my Yeah, there we go. And so, we went down a percent point. So, the effect was there and we're able to measure it and show that. Uh and then they they tweaked the hue of the color to be a brighter hue. And uh the day that that or the week that that patch went out, there was an immediate um advantage seen. Now, this is across all maps. So, the thing to remember when you're looking at the data is uh there's always another story, right? So, you go digging into it and you're like, "Well, you know, let me look at a a map that's that's kind of uh reddish." So, in this case, the um advantage that red has is like a few

percentage points uh versus the column on the left, which is the overall advantage. And the fix went out and sure enough that dropped by a percentage point just like it did for the average across everything. And the the blue color went out. And uh now red team is actually getting an advantage that they didn't have before. So, blue is now at a consistent two or three percentage point disadvantage, uh but that is now being bad on the the redder maps, right? And there's one other map that's that's even worse than this and then and they they went really not not good. So, um we're going to have to do something like a per map color variation or something, but Um I'm I'm out of time, so I'm going to I'm going to play these two things and then Um there you go. This is a war zone map, uh blue team crushing red team.

Um Play it one more time. And then uh conversely, this is uh blue team versus red team are evenly matched and so they uh just sit there duking it out. Oh, except the wrong animation got in, so it was the same thing. That's a bummer. Uh Get my card afterwards. We'll go outside and we'll have lots of questions, I'm sure, and and I'll email you the uh the real animation. So, uh thank you all for your time. I appreciate it.