A Recipe for Mixed Reality in 'Mario Kart Live'

Hello. I'm Ed Tombush, and today I present a recipe for mixed reality in Mario Kart Live. Now, I've been making video games for around 15 years now, and I currently work for Velan Studios, a newer startup located in upstate New York. Mario Kart Live is the first game released by Velan. And this is a game that I worked on along with a small team for 3 odd years. So, what is Mario Kart Live: Home Circuit? Well, first off, it's a game for the Nintendo Switch. And when you purchase the game, it comes with a remote controlled kart, and you get to choose between Mario and Luigi. When you power up the kart, it connects to the game software running on the Switch. And the game allows the player to control the kart using the Switch controllers. The karts have an onboard camera, where a live video feed is sent back to the Switch and shown on screen, giving the player a first-person view of driving.

The game also comes with four gates. And the gates are important as they help to define the play space. After a player has connected the kart for the first time, the game guides them to place the gates around the room to form a circuit. And this lets the player create any sort of track that they would like inside their house, using the space that they have available. And even at its core, the driving experience is super fun. In fact, this is how we started experimenting in this space. In the beginning, we used drone parts like cameras and video transmitters and cobbled them together on some RC cars. And we found early on that it's really fun to drive around your house from the point of view of a tiny kart. And the sense of speed that you get from a toy kart zipping around your living room is pretty amazing.

But of course, we're making a Mario Kart game, and we need more than just that first-person driving. And this brings us to the mixed reality experience that we created, where we combine the digital content and interactions of the video game with the physical play space of the player's home, and hopefully seamlessly integrate the two. It's really the digital content that we place in the player's home that turns the experience into Mario Kart Live. Content like item boxes that you collect at the gates, the bananas that you drop on the track for your enemies, coins that you collect, the red shells that chase after other racers, the AI competitors. And it's extremely important that all of this content feels like it belongs. It needs to feel as if it's in the same room you're racing in, like it's part of the video feed that you see on screen. And here's the interesting bit. When you get it right, nobody notices. It feels like it belongs. But it's when you get it wrong that it stands out.

And that's the point of this talk, to explain the tools and techniques we used to get everything just right, to make the user feel like they're playing in the world of Mario Kart inside their home. And unlike a lot of augmented or mixed reality experiences that you see, this is not a tech demo. We're making a game that needs to have replayability, and it needs to be robust to normal gameplay patterns. It needs to just work all of the time. As we started this project, we did look for existing tech that could help us, but nothing quite matched our use case. So, we ended up rolling our own solutions to the problems that we needed to solve. And that had some advantages. We have control over the entire pipeline. There are no black boxes, and we understand all the bits. And it allowed us to build technology tailored to gameplay, and conversely, gameplay tailored to what our technology is good at.

Now, I'm going to talk about the ingredients, all the tools and techniques that let us execute on the content that we just saw. And we start with a model of how we represent the physical world in our game. Now, everything is measured exactly. We have a rig in game that matches the exact dimensions of the kart. We assume its pivot is dead center on the floor. It has a camera with an offset and a rotation from that pivot. Everything is modeled in the game in understandable real units. Distances are measured in centimeters, velocities centimeters per second, acceleration in centimeters per second squared. And this is going to be a reoccurring theme. Everything in game is modeled to match the real world as precisely as we can. And the same is true for the play space. I'm going to show you how we generate a map of the user's course. And everything in this map is modeled in

real world units. The gates' positions are estimated in centimeters, the same as the kart's position. And all the content that we put in the world is modeled at that same centimeter scale. So, when all this comes together, in the physical world, our digital representation of the world, and the content that we put in it, when it all agrees on the scale and units, well, that's when things start to feel as if they fit together. Just as we strive to model the physical world in game, we also need to model our in-game camera to match the physical camera on the kart. Now, we have a video feed coming to the game, and we want to draw digital content over that video feed. We need that content to feel as if it's in the same space. Those item boxes need to feel like they belong in front of the gate. And in order to do this, our cameras need to match. We need to make the digital camera in

game match the physical camera on the kart. There's two main parts to matching the cameras. A physical lens will often add barrel distortion. And this can be observed as straight lines that appear to be curved, especially when you move towards the edges of the frame. Of course, this distortion isn't present in the game camera, so we need to remove this from the video image. We call this undistortion. This will make the image rectilinear, where straight lines remain straight, and we can very closely align it with our game camera. But we need one other thing. We also need to calculate the field of view of the physical camera, so that we can apply the same field of view to the game camera. Now, these screenshots show an image from the physical camera. The first image on the left is the raw view with the barrel distortion still present. The image on the right shows an undistorted view.

Now, after we undistort, we're actually left with some extra pixel data that we end up cropping from the view, and that's kind of highlighted by that blue rectangle in the right image. We only end up drawing what's inside that blue box. But this example, it helps to show what the undistortion is actually doing. In order to remove distortion and calculate the field of view, we need to calibrate the camera. So, we capture images, a lot of images, of a chessboard. And you need at least 20 images, probably more, with the chessboard near, far, different parts of the frame, different angles, before you start to get a uh a good some good results. And we use OpenCV for the calibration. First, we use OpenCV to find the chessboard corners in each of the images. With a set of images and corresponding chessboard corners, we can use OpenCV to calculate the intrinsic parameters of the camera.

Now, these are the parameters that numerically describe the camera, such as the distortion coefficients, projection matrix, and in turn, the field of view. Now, with the distortion coefficients, we can now undistort the video image from the camera. And in addition, we can set the field of view of the in-game camera to match that of the video image. And so, here's an example of a scene switching between a raw undistorted a raw distorted image here to an undistorted final image here. So again, this is the raw image. This is the undistorted image. There are a lot of resources online that describe camera calibration, and I've given some useful search terms to help find them. And it's worth noting that we often use Python tools that are Python for tools

that are external to the game, and this is one great example. So, we have a simple tool for extracting good chessboard images from a video, another tool for finding chessboard corners and running the calibration. And we have scripts for undistorting still images that allow us to preview and evaluate the results. And it's very easy to get started in experimenting with these concepts in Python. The next thing to understand is what kind of data we exchange with the kart. And it's pretty straightforward overall. From the game to the kart, we send steering and throttle information. We sample and condition the user input from the controllers and send that as commands to the kart for driving. Now, from the kart to the game, we get a stream of video data. And in addition, we get a stream of data from the IMU. What is the IMU? Well, the IMU stands

for inertial measurement unit. And it's a sensor that can measure the movement of the cart in 3D space. IMU is what called is is what is called a six axis sensor, which is the combination of a three axis accelerometer and a three axis gyro. The accelerometer measures the linear acceleration on the three axes X, Y, and Z. And the gyro measures the angular velocity or rotation of the cart on the same axes. So, it's important to note that our IMU samples are time stamped. And that the IMU data is sampled at 160 hertz. And of course, we receive a stream of video data as well, which as you might expect consists of a series of images. And maybe the most interesting thing about the video frames is that they're time stamped just like the IMU. And importantly, they're time stamped from

the same clock as the IMU samples. Unlike the IMU data, which is sampled at 160 hertz, the video data comes in at 30 hertz or 30 FPS. Now, because of this difference in frequency, we end up receiving about five to six IMU samples per video frame. But with a time stamped video and IMU stream, we can correlate the IMU samples that belong to a video frame. Now, for each frame that we process, we want to know what IMU samples occurred in that same time slice that the video frame captured. And this is basically our unit of work in our mixed reality pipeline. Now, it's worth pointing out that sometimes this data stream can be imperfect and we can lose data. After all, we're using Wi-Fi for communicating between the cart and the switch, and we can often drop data, especially

as the cart moves far from the switch or even if the user is in a noisy network environment. Most commonly, we lose a video frames. In that case, we still want to process the IMU samples that correlated to the missing frames. Now, the IMU samples are very important. We don't want to miss any of those. Now, can we miss them? Yes, we can. But because of their smaller size relative to the video data, it's less likely. And we also have mechanisms in place for reconstructing missing IMU data should we lose a small amount. And we have larger fail safes for when we miss significant portions of data. So, what do we get from this data? Well, first we'll talk about banner detection, our computer vision pass. Anytime we have a gate banner in the video frame, we need to try to find it. Why do we want to find it? Well, there's two good reasons.

The first is so we can place content around the gate. Now, the gate is a great place to put content. We had can have high confidence in their location and they're fairly stable once we detect them. They also provide great information about the world around the cart. And actually help us to build our digital model of the play space. Now, I'm going to give a high level overview of how we find and identify the banners. And our goal is to find the eight points that define the inner and outer rectangles of the banner. And to find those eight points, we're going to try to identify the black border around the marker. And we're going to start by trying to find the sides of the banner. So, let's keep that in mind as we go through this process. The first few steps of the algorithm are going to be searching for the vertical-ish sides of the banner. So, how do we find those?

Well, we progressively scan the image from left to right looking pixel by pixel. We're looking for high contrast transitions, and those transitions are represented by circles in this image here. And we don't scan every row. We scan at a fixed interval, so we don't have to look at every pixel. And in this result, the green circles represent a transition from light to dark. And the red circles represent a transition from dark to light. And once we have found these transitions, we walk the lines that define them. So, when walking these transitions, we want the results to be mostly straight and mostly vertical. Remember, we're searching for the sides of the banner. Now, we have a large set of straight lines. And we know that they define a transition in contrast from light to dark or dark to light.

We search these lines and look for pairs, line pairs that could possibly form either the left or the right side of the banner. As we can see, that removes quite a few potential transitions. We're left with possible edges, pairs that could form the left or the right of a banner. And we know that a banner is going to have a left and a right edge. So, we look for pairs of edges that could potentially form a banner. And these edge pairs form quads. We can see that there are only two possible quads remaining in this image. And we can refine these results a bit more. We could walk what should be the top and the bottom borders. We can confirm that they are straight, horizontal-ish transitions where we expect the top and the bottom of the

banner to be. Now, you can see this process by the blue and red hash marks on the top and bottom of the quads. The red hash marks in that top quad indicate a miss. That top quad is probably not a banner. It couldn't find its top and bottom edges. And that leaves us with one result in this image. So, we think we have found a banner. Now, we have to try and identify it. We want to know what gate it is we're looking at and whether or not it's the front or the back of the gate. Well, we start by unwrapping the found banner. That's where we take it from a skewed perspective shape to a flat rectangular shape. And then we threshold the image. We take all the pixels that are closer to white, bring them to full white, and all of the pixels closer to black and bring them to full black.

And we could do a fuzzy comparison against a set of reference images, and these are shown on the right. This lets us determine what gate it is we are looking at. Now, we know what banner we're looking at. We need to estimate what we call the pose of the banner or the camera relative translation and rotation. We know the eight points in screen space that represent the banner. We also know, because we can measure it with a ruler, the physical dimensions of the banner and how the points in screen space relate to the physical cardboard gate. Now, this association of 2D screen space points to 3D object points from the physical banner, we can use OpenCV's solve PNP or perspective in point to calculate the 3D translation and

rotation of the banner with respect to the camera. So, that's what we do for every frame that comes through the pipeline. We search the image for banners, and for each banner that's found, we identify it and calculate the camera relative translation and rotation. And we call this CV results, and every frame gets its own set. Now, while our system works well most of the time, it's worth showing off one of our extreme failure cases, the dreaded dog crate. Now, the dog crate has a lot of contrasty transitions, so our algorithm finds a ton of edges, and in turn, some quads. Here's one, and a few more, and more.

Okay, that's a lot of quads. And that's over 600 potential quads. And iterating over this many quads can have a noticeable impact on the processing time. And it was situations like this that forced us to come up with some scoring and early out techniques to avoid the processing overhead that these kind of situations can present. And while we use a custom implementation for finding the banners, there are open source solutions available. In fact, our first prototype, we used what are called ArUco markers. These are large pixelated markers in this image here. OpenCV has an implementation for detecting the ArUco markers in an image and returning their corners. And the pixels in the ArUco markers allow OpenCV to uniquely identify them. Once we know the corners of those markers, it's the same call to solve PNP to estimate their poses.

Now we know about the gates that are present in any frame. We know their ID and we know their pose relative to the camera. And those banners, they provide really good information. But we want to interact with content at the gates even when we can't see it, like when we're driving past them. And we want to interact with other things in the world even when we don't see any gates. So in addition to gates, we really want an estimation of the world around the cart and the cart's position within that world. We need slam. So what's slam? Slam stands for simultaneous localization and mapping. And it's a very common field of robotics. So I was looking for a nice and concise definition for slam and Wikipedia had a pretty good one. They say that slam is the computational problem of constructing or updating a

map of an unknown environment while simultaneously keeping track of an agent's location within it. So basically, slam is building a map of the world and figuring out where you are in that world at the same time. So this is what we're going to do. We're going to build a map of a user's play space. And there are many approaches to the slam problem. This chart shows many slam techniques as they've evolved over the years. And if I had to place our method on this timeline, it would fall close to the 2007 or even prehistory time frame. Now there are a few reasons to use older methods. Often newer methods are struggling to run real time, let alone alongside a game that has its own processing needs. And while creating our own solution was a big lift, it allowed for a custom implementation

that exactly fit our use case. And I couldn't talk about our slam solution without mentioning Klaus Breiners' amazing YouTube tutorial on slam. In the beginning, his tutorial gave us the baseline understanding to even go down this path. We actually started by following in the tutorial, modifying it to match some of our differing assumptions. And while our solution has greatly evolved as development went on, there's still plenty of inspiration in there from Klaus's tutorial. It's a great introduction to the slam problem. He uses Python and provides code samples for the entire implementation. So this is if this is something you're interested in learning about, this is a great place to start. So we ended up with our own slam solution heavily inspired by Klaus. We don't really have a name for it. Maybe Valen slam, B slam. Now if we wanted to get cute like with the acronym like many other

implementations do, we might call it SLIEKF slam. Well, what would that stand for? It stand for sparse landmark and natural extended common filter simultaneous localization and mapping. So let's break that down. Sparse landmark. We use the four gates and only the four gates as our landmarks. And that's very sparse as far as slam implementations go. Most methods track thousands of world features while we only consider the four gates. Inertial. We use the IMU to help make predictions about our motion. In fact, this is what allows us to use such sparse landmarks. Our motion estimation alone is very good. And it allows us to keep a consistent map even with only the four gates. Extended common filter. Now at the very highest level, the extended common filter gives us a probabilistic estimation of the state of

the world and the cart's position within it. And this is a huge topic. Could span its own talk, so I'm not going to go very deep into this, but I will give a high level overview and provide some resources. And finally slam, simultaneous localization and mapping. We talked about that. It's building a map of the world and figuring out where we are in it. Now the one thing I will talk about with the common filter is its update loop. A common filter contains a set of state variables that it's continuously estimating. In our case, things like the position, orientation, and velocities of the cart. And anytime you're working with a common filter, you would expect to see it in two see it update in two phases. Predict and measure. Now during the prediction phase, the filter is updated given the current state and optionally some input. After the prediction is made, the filter

has an updated probabilistic view of the state variables that it's estimating. The second phase is measurement. Any observations that could be made about the state can be applied or measured. Now these measurements to the state help the state variables become more accurate and hopefully converge towards reality. When we update our filter, we predict the new state using the IMU data stream. We make measurements against the state using some assumptions about the vehicle motion, the way that we expect the cart to move. And additionally, we make measurements against the state given our gate observations, the CV results that we talked about. So I'm going to show some example output from our slam system that will demonstrate how it works. And I'm going to be using real data from the game. As the game is running, we could save

off the stream of IMU data along with the CV results for the banners that we found. Outside of the game, we have a set of tools that let us run our slam simulation in a Python environment. And we're going to be looking at the output from those tools. Now I've captured data from a real session while driving a five-lap circuit around my living room. This video shows the path that I followed while collecting the data and you could see the mini map shows the trajectory that we'll hopefully see in the simulated output. So let's start by thinking about how we predict the motion of the cart given the IMU data. Naively, we can observe our stream of IMU samples and integrate our accelerometer data and hope for a good estimation of our position over time. The accelerometer has an inherent bias. And a bias is a consistent offset in the output from the true value.

If we knew the bias, we could subtract it from our IMU readings to calculate the actual acceleration along an axis. With standard motion equations, we could estimate the velocity and in turn the position of the cart over time. There's some pretty significant problems that begin to show up. Now in general, data from an IMU is very noisy to begin with. And then we have that IMU strapped to a moving vibrating vehicle. And as I mentioned, the data is biased. And I talked about how we might remove it, but in order to remove it, you need a good estimation of the bias, which may even change over time. And because we're integrating from acceleration to a velocity and then from a velocity to a position, any error that is present grows exponentially. Integration alone will quickly drift away from the actual trajectory.

We can observe this drift in our Python simulation. As I mentioned, this output represents a sim a simulation driven by data from a five-lap circuit in my living room. And in this example, we only show the integration of the IMU data from our data set. Now I must admit, I cheated a bit here. I actually had to fudge things for this example to get a trajectory that looked even close to something that was recognizable. Anything less and I had an estimated position that quickly drifted from the origin without any recognizable motion. And that's kind of the point. With IMU integration alone, the estimated position quickly diverges from the origin and the results aren't very useful. So integration alone just isn't enough. But we have some tricks up our sleeve. Now while the IMU can tell us when the cart is moving, it can also tell us when it's not moving. We can look at a sliding window of IMU data and determine when the car is at

rest. Now our slam simulation is continuously estimating the velocities on each axis. If we know we aren't moving, we can make measurements on the slam filter that both the linear velocities and angular velocities should be zero. We're basically asserting that the cart shouldn't be moving or rotating when we know it's at rest. We also know that the vehicle is a cart and that a cart drives in a certain way. We can assume that the velocity on the up axis is going to be close to zero most of the time. We can assume we can also assume that the lateral left-right axis is going to be zero most of the time. Definitely when moving in a straight line. And even when the cart is turning if it isn't sliding. Now we can use these assumptions to measure that the velocity is zero on the

up and lateral axis even when the car is moving. Now these constraints are a bit more fragile since they are always guaranteed to be true. We have to be careful about how confidently we make these measurements. These measurements certainly won't be true if the cart has been picked up if it's not driving on the floor. Now in order to combat this we have a state machine that is evaluating if we think the cart is driving or if it's been picked up. Now this is where the magic of common filters comes in. When we make these measurements that put some constraints on the velocities the filter is actually converging on better estimates about its entire state. Like what? Well like the estimated IMU biases. Like the estimated position within the world. Let's see what that looks like. This example shows the same IMU integration on the same five lap data set that we saw before.

The only addition here are the previously mentioned measurements, the motion constraints that we just talked about. And this is a pretty significant improvement. Once again there's no additional information provided here, just IMU data with measurements that constrain the velocities. But we have more data available. Now if you recall our computer vision pass provides us with information about the gates. We know about gates that we currently see by ID and we know the camera relative translation and rotation of the gate. We can combine this data with our estimated cart position to calculate where a gate is positioned in world space. And we can make measurements on our filter about a gate's position. So let's see what that looks like. And once again this is the same data as before, five laps around my living room,

but this time we make measurements about the gates when we see them. Imagine we see a gate on a particular frame. If we haven't seen the gate before we add its estimated position to the state. If we have seen the gate before we use its prior estimate and current observation to refine both its position the cart's position and to further refine all the other variables we're estimating. And now we're doing slam. Simultaneous localization and mapping. So what do we get out of all this? Well we get an estimation of the cart's position and orientation within the world. We get an estimation of the cart's velocities along all axes, X, Y, and Z. We get an estimation of the angular velocities or turn rates along all three axes. We also get an estimation of the accelerometer biases on each of those axes.

Additionally we get an estimate of the physical camera's rotation relative to the cart. Now this is really interesting because while the camera is fixed to the physical cart there can be some variation to the camera angle both because of manufacturing tolerances and the fact that it's in a mount that's not completely rigid. So this estimation of the camera's angle helps to improve the overall accuracy of the slam model. And we even use this estimate to adjust our in-game camera rig to better align the digital world to the physical world as we see it through the camera. And finally we get an estimate estimated positions of all of the gates in the world. So put a different way we get an estimated view of a user generated play space and the location and orientation of the cart within that world. Now slam is a large field with multiple solutions using different techniques and

tons of resources online. So if you want to dive into some of this stuff there's plenty of places to look. And as I mentioned Klaus Brenner's slam lecture playlist on YouTube is amazing and it heavily inspired our approach. Additionally Udacity has a course titled Artificial Intelligence for Robots for robotics which has some great resources, specifically a great introduction to the common filter. So now we're able to understand the world around us, we have a pretty good estimation of where the cart is in that world. But there's one more piece to this puzzle and that is information about the shape of the track that the user intends to race on. After all we intend to host a race on a course that the player has constructed. So we need some information about the shape of the course and the path that it takes through the user's play space. We call that track space. This video shows the user facing track building experience.

The in-game fantasy is that Lakitu pours paint on your tires and instructs you to drive a pace lap leaving a paint a trail a trail of paint as you go. So basically we're looking for one good continuous run around the whole track that will help define the shape of the user's course. Now as the user drives the pace lap we sample their position over time. The green dots here represent what we call breadcrumbs. We add the breadcrumbs based on how you're driving. Long straightaways only need a few breadcrumbs to define them. When turning we add more breadcrumbs to help further define any curves. The breadcrumbs are defined as a sequence of 2D points.

And we can traverse this sequence of breadcrumbs to get line segments. And from the line segments we can measure their distances. And by summing all of the distances we can determine the length of the entire track. So now we can think about a user generated course in terms of a distance around the track. So let's imagine that this course is 800 cm long. As you traverse the course you start at a distance of zero when you pass through the start gate. And your distance along the course will increase as you eventually reach a distance of 800 cm as you cross the start gate again to finish a lap. Now all of the gates can be represented in terms of a track space distance.

In fact any arbitrary point along the course can be described in terms of a track distance. And we can easily convert any distance along the track to a world position. We have an API convert distance to slam position to do just that. And this track distance is actually two-dimensional. We can provide an offset parameter to get a position to the left or right of the center line of the track. And we can do the opposite. If we know the world position of an entity we can calculate the track distance and offset of that entity. We'll often do this in the context of a race. We can compare the relative race positions of different competitors by comparing their track distances to determine who's in first, second, third,

and so on. And getting this in was a big win for us. We went for quite some time with only considering the gate crosses for racer position. As in the person in the lead was the first person to pass a particular gate. But with this conversion to track space we can continuously estimate estimate the race position of all racers even between the gates. We can think about the track in other ways as well. We can think about it in terms of a normalized distance and we call this track progress. Where zero is the beginning of the track and one is the end. And we can easily convert between track progress and our track distance. Additionally we can think about the track in terms of gate progress. Now gate gate progress lets us imagine

that the whole numbers 0 1 2 3 4 represent gate positions with normalized distances as the decimal between them. Now gate one, the start gate is represented as 0.0. Gate two as 1.0, gate three as 2.0 and so on until we return to the start gate again for a gate progress of 4.0. And we primarily use gate progress to think about the space in between the gates. For example we can get a point on the course directly between gates two and three with a gate progress of 1.5. And like track progress, we can easily convert between gate progress and track distance. Now, up until now, we've only been

dealing with our estimation of the world and how we rationalize a user-defined course. Well, now we're going to look at how we use these results for gameplay. Before we dive in, let's recap all the data that we have access to. We have video and IMU data streamed from the cart associated with timestamps. We have CV results that contain gates with ID and camera relative positional information for all the banners that we that are on screen at any frame. And we have slam results that contain an estimation about the cart's motion along with its world position, estimation of the gates, and additionally, the track space information. So, how do we use all this? Well, first, we move the cart. So, in the game, as you might imagine, there's an entity that represents the player and their digital version of the cart. And the important thing to know is that

we're actually moving that entity through the game world. We use the estimated velocities and turn rate from slam to update its position and orientation in game. And you'll note, we're not directly consuming slam's estimated world position or orientation here. And that's because the slam simulation and game simulation don't really run in lockstep. We might drop video frames, have late or delayed frames, or have other hitches in the video feed. Our game simulation, on the other hand, is continuously updated. And we want our in-game representation of the cart's motion to update smoothly and consistently. In addition, slam is continuously updating its estimate of the cart's position based on gate observations. And there can be cases cases where a measurement from a gate causes a large correction to that position. And we want to hide that from the game simulation. So, we do consider slam's estimate of

the cart's position and orientation. Whenever the game simulation receives updated slam information, we interpolate towards that estimated world position and rotation. So, this two-step process allows for smooth updates to the game entity while still maintaining consistency between the game and slam's view of the world. As we'll see, this concept that the cart is moving through the game world is very important and it's basically the backbone of our entire mixed reality system. So, let's talk about the content that we put at the gates. We want our gate content to look as if it's part of the world. It needs to be locked in a line with the video stream each frame. We don't want it to jitter, drift, or pop in and out of existence. And it needs to be robust to imperfect information. There are cases where we don't find the marker. Maybe it was occluded by a chair leg or a cat.

And by design, the user is encouraged to drive through the gate and underneath the banner. While you're driving through the gate, we can no longer see the banner, but we still want that content that is being drawn inside the gate opening to remain stable and convincing in these situations. We can see an example in this video interacting with the item boxes as we pass through the gates, even when there's no banner in the frame. So, let's take a look at how this works. Remember that for every gate that's detected in a video frame, we have a CV result that represents gate, its ID, and its camera relative translation and rotation. We also know the world position of the cart. And from our cart rig, we know the relative offset and rotation of the camera on the cart. We can concatenate these transforms to arrive at a world position and orientation for the gate that perfectly matches the banner that we detect in the video frame. And we can place digital content at this

location in our game simulation. It has a world space and position just like our in-game cart does. Now, the next frame, we move the cart again. We update the position and rotation of the cart based on our slam estimation, just as we did before. And if we happen to see gate three again, great. We can use the CV results to update its position to match our new estimate. And in fact, we add a bit of interpolation to this update to keep things looking smooth. Sometimes the CV results can be a bit noisy, and the interpolation helps to hide some of that. But what if we didn't see gate three again this frame? Well, that's okay. The cart has moved in the world. The digital gate hasn't moved relative to the cart. And because we're estimating the motion of the physical cart, this is hopefully very similar to how the gate is moved relative to the gate in the real world.

And once again, here's the result. So, next we're going to talk about world items. Items that exist around the track in the space between the gates. And there's plenty of content that appears simply through the course of playing the game. There are bananas thrown on the course, the bombs dropped on the track, coins that you drop when you get hit. But let's focus on a thrown banana peel. How do we make it look as if it's sitting on the floor of your room? How do we make it appear as if it's stuck in one spot on the ground as you drive past it? How do we make it look as if it's in the same spot when you see it again the next lap? Well, really took all of the tools that we've talked about to make this possible.

So, let's look at how it all comes together. Well, we know the cart's position in the world. When we throw banana peel, we can calculate its world position as an offset from the cart. And of course, in game, the cart is moving through the world by updating its world position. And this world position is constantly updated to match slam's estimated world position in your house. And because our game camera matches our physical camera, the banana peel looks as if it's anchored in the world as we drive past it. And it appears to be in the same spot where we left it when we see it again in the future. So, next we'll look at track space items or content that we want spawned relative to the track.

And typically, these are items that we want to think about at design time. We need to be able to put things like hazards and coins around the course in a meaningful way and depending on what kind of race the users engaged with. But here's the catch. We don't know about the course. We didn't define it. The user did when they placed their gates. So, we need a method to think about the course and how we're going to place objects and hazards along it at design time, absent any knowledge of what it might look like. This video shows a great example with track coins. For certain races, we want to place a cluster of coins on the course between each of the gates. So, let's focus on that cluster that we want to spawn between gates two and three. Well, we can use our track space API. For a spawn position between gates two and three, we first use our convert gate to distance function with 1.5 as the gate progress parameter. That call to convert gate to distance

will return the actual track distance in centimeters to that point on the track halfway between the gates. And we use that track distance to determine a world position using the convert distance to slam position call. And that's it. That's the bridge we needed to cross. Now, our newly spawned coin cluster has a world position. And everything from the banana example applies here as well. We can drive the cart through the world, and the coins appear to be anchored within that world relative to the cart. Using track space, we can think about the track in meaningful ways at design time. And we can place content anywhere we like on a user-generated course. Now, Dan D'Amato, the game director on Mario Kart Live, did a talk on this as well titled Mixed Reality Racing Fuses Deeper AR Experiences with Physical Gameplay. And he talks a lot about this, how we think about user-generated content or user-generated courses at design

time. Finally, we'll talk about dynamic items, items that need to interact with the court course itself. And two good examples of dynamic items are AI and red shells. Now, these things need to move and behave intelligently on a course that the user defines. We'll look at a red shell for this example. As a red shell traverses the course, it tends to operate primarily in track space. It holds a track space distance that represents a point along the course. And to move the red shell, we can increase that distance by some amount given the shell's speed. Our track space API allows us to convert that distance to a slam position, which gives us the new world position of the shell. We can even have the shell or an AI racer move left or right on the course

with our offset parameter. And that's really it. Using our track space API, we can make dynamic items like AI racers and red shells move intelligently on a user-defined course, course that we had no prior knowledge of. The final thing we need to do is to get everything on screen. We have a lot of data that we have accumulated. The data from the video frame and IMU data, our CV results, the slam results with tracking information, and of course the output from our game simulation. And now we need to render it. Functionally, we're rendering the undistorted video image, and we're drawing all of the game content on top of that video image using a game camera that matches the physical camera.

And one thing I haven't really talked about is latency, and that could be a talk of its own. We measure our latency in terms of the amount of time it takes to get a video frame that's captured on the cart onto the screen of the switch. And as you can imagine, the processing that we do takes time. From the computer vision algorithm to the slam update to the update of the game simulation. Often, we even get a new video frame before we're able to get the last frame on screen. You might be tempted, as we were, to try and render that frame as quickly as possible, maybe with the game simulation results from the prior image. And we've tried this in many different ways, and we failed many times. The simple fact is that any discrepancy between the video frame and the results from the game simulation will be seen. Things will be out of sync, and it will break the illusion. This is the really important bit. Everything needs to be rendered together

as one unit. And yes, that means that there's latency built into this approach. And for that reason, we spent a lot of time on optimization and techniques for further reducing the latency in other ways. So, that's it. An overview of our pipeline and a lot of the tools and techniques that went into making Mario Kart Live. I'll leave you with a closing thought, a thing I'd said hundreds of times during development. And that's we need to lean into our strengths and avoid our weaknesses. Making a mixed reality game is hard. And there are a lot of things that we tried and content that just doesn't work well in mixed reality. And that was a big part of development, finding the things that work well and leaning into those. While avoiding the things that don't work well, content that emphasizes the flaws in the system. Be sure to check out Velan Studios online. You can learn more about what we are up to and explore our job listings.

And that's it for now. Thanks for your time.