Can You See Me Now? Building Robust AI Sensory Systems

[Music] hello everyone like dave said my name is eric martel i'm a technical lead ai programmer at the ubisoft quebec city studio so we're slightly different than montreal but it's a quite nice studio so come see me if you want to learn more about it um so i programmed a little something as soon as the slides are done updating there should be a little camera in the right side corner so that you guys don't take a picture of a half animated slide it's a gimmick but i've seen so many people take pictures of slides that had only half the content that maybe will help you guys so first uh thank you to be to being here uh it's really appreciated to have people coming and seeing uh the the kind of work we do and also huge thanks to ubisoft quebec because they're really supportive and i wouldn't be here

if it wasn't for them so senses most of you know that most human beings have five senses vision hearing smell touch and taste in video games sometimes we have what we call a sixth sense and that's not about seeing dead people it's uh usually feeling stuff that that is not in direct perception of the npc so for example standing next to an npc in this little bubble it could trigger the sixth sense for the npc to maybe do a little look at over the shoulder and see if the player is hiding there i'll be making the the differentiation between perception and detection to me perception is really mechanical it's on or off you're either perceiving or not it's the ability to see hear or become aware of something through the senses whereas detection is more about

identifying what i'm perceiving a lot of gameplay code works in the detection space but in this talk we'll mostly talk about perception since it's a quite short talk i'll be mostly talking about vision i'll be explaining how to build geometric sensors how to do point-to-point perception and how to provide those sensors behavior basically those are the building blocks to create a simple two somewhat complex sensory system for the npcs for the people who are interested in seeing other senses in case of hearing most systems will simply use a spherical perception so just a radius check if the sound is emitted within a certain radius higher it if you guys are dealing with interiors so basically complex buildings you might need to implement some propagation

that's something we did on assassin's creed syndicate by doing 3d path finding using volumes and portals which were already used by the audio designers to manage the sound that was being heard by the player so reuse assets if you can but 3d pathfinding is the way to go for for that for a sense of touch uh it's really rare that the npcs will need to receive feedback on what they're touching on their own like the the decision they made to go interact with something they don't really need feedback to it uh and whenever something is touching them in most of the cases you can simply forward the event that you receive directly from the physics engine it can be quite simple just a bump event or a hit and from that the the npc can do some deliberative decision making for smell very little games use the sense of smell

but we played a bit with it in a very old far cry title where the player was emitting some sort of breadcrumbs representing his smell so it started with a really intense little breadcrumb and over time it would expand and lose intensity so from there you could follow the breadcrumbs and find the player for the sense of taste until unless cooking mama requires an ai to test your recipe i don't see a real use for the taste but if you guys have any ideas just go for it i will also not be talking about serious stuff right now the sensors are quite trendy if you're talking about real world sensors like smart homes or autonomous vehicles they tend to use computer vision deep learning all that kind of stuff i wish we could use that in video games

but in most of the cases we just don't have the cpu to do it and the computer vision would actually solve quite a few issues that we'll be discussing later on in the talk so hopefully by the end of the talk you'll have a better understanding on how to create sensors how to give them proper behavior how to be designer friendly how to be animator friendly tips to make it faster and more flexible in my experience sensor systems is really a team effort it's not just the programmers working on their own you need to have a good communication with the game designers and with the animators and this way you can achieve a better game before we dive in everything that i'll be showing you has been tried tested and shipped in some game so it's stuff that's that works and hopefully you can base your own implementations off of

this i'll be showing you a quick demo it's built with play canvas so it's an html5 webgl rendering framework with assets from mixamo and cgtrader so you'll see it's quite simple my goal is to show that within a few hours you can build something that's simple that's easy to understand that shows off what the sensor system can be so that's basically the target we're going for we'll be using agents they could be simple or animated in our case they'll be animated because most games use humanoids characters we'll be using point of interest so it's basically what we're trying to see in the world in this case i'm not re-original i use the target and finally for the environment it's whatever's blocking the vision of the npcs so in this case i use the crate because most games have crates um

as gizabi was saying yesterday um vision is quite simple there's a few steps to follow and most of the times will be good enough for your game so basically filter which entities you want to try to see test to see if they're within the the sensor range and or shape and then use a raycast that's the expensive part that's the part you want to do the least you will have multiple target types uh depending on the complexity of the game your npcs might be trying to see other characters or only the player it really depends on your gameplay particle particular events so as was described earlier i think yesterday sometimes you want to try to see something that happens in within one frame and sometimes it's something that's not even there so you just leave some piece of information in the level and if the npc sees it

you can understand that something happened and react on it some useful data whenever you're dealing with sensors is to convert everything in local space it makes the computation really simple and compute the relative angles sometimes you'll want to do it on a top view and sometimes a 3d angle so we'll dive in the anatomy of a vision cone but to me in most games i worked on it's more like a pie so it's a triangular shape with some sort of height it can be quite simple if you're working on a top view game maybe you can stick to simply have an angle opening of how wide your sensor is seeing and then a far distance for your sensor describing how far the npc can see then if you want something more complex you could add a near distance to control

which region of the the the environment in front of the npc is being seen by the sensor if you're going 3d you will obviously need to control the height by describing a top and a bottom cutoff for the sensor so again it's something quite simple and if you can make it dynamic for your designers to play with and see in real time how the sensor is being affected i think everybody wins obviously something simple as putting an offset on the referential for the sensor and then attaching the sensor to a specific bone you can either attach it by position or position and orientation in this case i'm showing off the same sensor being attached to the root of the npc then to its hips and then to its head you can also control the shape in more complex ways by

for example pitching the near-end pinching the near hand of the sensor to be able to drag it up or down so that you can have sensors that go upwards or downwards you can also smooth the edges of the sensors using splines so either to smooth out the sides of the sensor or to have the the the close part of the sensor detect less high than the far end and if you want to be really fancy you could use some height maps to draw bitmaps describing what's the height of the sensor at the given x and y coordinate coordinate relative to the npc you should try to build like a big library of shapes that your designers can play with so a sphere is quite simple and you might need one for your auditory system anyway so it's a perception in all directions cylinders that's quite useful for six senses you can have just a small cylinder around the npc and whenever something's

entering that bubble the npc gets notified and if perception is not if you're not building a stealth game or a game where the the complexity of the sim that's of the sensors uh have almost no impact you can simply use bounding boxes so whenever something is within a box either object aligned or access aligned you perceive it build anything you can think of the more tools you're providing your game designers the better they can build and experience for example a few years ago martin walsh presented the splinter cell sensors they had success using coffin shapes sensors just do it it's as long as you can build a primitive that's easy for you to test whether our position is inside or outside just go for it and experiment spend some time with your game designers and where with your animators and try to think of new ways you could provide them with

tools and simply do it um more advanced systems will be using multiple shapes so for example in this case you could have some metadata on the closer sensor saying whenever something enters this sensor it's highly dangerous whenever something is perceived in the yellow sensor it's not as dangerous so maybe you can change your behavior accordingly the attachment of the sensor on an agent uh well usually your agent will use a skeletal component you can either attach it on the entity root the hips or the head like i was discussing before but you can have some weird results whenever you're starting to apply animation of the npcs if you're um adjusting the rotation of the sensors based off of it so right now i'm using assets that were not meant to be used as sensor providers so in this case you see as soon as i'm putting a more dynamic animation

the sensor will be going all over the place this could be the desired behavior but in most cases it just makes it harder for a player to understand when he's getting detected because the sensor keeps moving um what i prefer is to use some sort of hybrid bone in your max scene you can simply parent the the hybrid bone to the root so that whenever the root is turning the bone will be following but then to have some mac script for example allowing the animator to say for particle or scene i want the the bone to be following the orientation of the head so the animator is in full control of where the the sensor is looking at any time and can allow you to create some handcrafted behavior so you're not relying on the visual asset you can create gameplay just through your animation and it's also really good to delay some abrupt changes sometimes players feel cheated when an

npc does a 180 turn because the detection cone can turn really swiftly in this case the animator could simply put a slight delay in the rotation of the detection bone making it so that the player gets the impression that he's about to be seen goes into cover and then the sensor turns making it giving him the impression that he's better than he actually is adaptable setup uh what what i like in some some system is to provide my animators with markups so the animator themselves can control whenever the sensor should be attached to the head to the the hybrid bone just let it so that data can control your sensors also in some cases the sensors won't be able to well the animation selection won't allow the npc to properly

track a target in some cases it can lead to behaviors where you have your sensor toggling between i'm seeing the target i'm not seeing the target because the target is moving next to the npc and is trying to track it in that case what i prefer simply putting a small state machine on the sensor and detaching it from from its referential from that point the sensor is simply following the target and keeping a line of sight to it as soon as there's occlusion the sun the the state machine goes back to uh following the referential making it that the npc is simply following its target and never losing track of it whenever you're targeting something depending on the type of entity sometimes you can simply use the entity position it's as simple as it gets but unfortunately it's usually on the floor making it so that any rock any crate any anything on the floor can occlude the target making it less interesting

for npcs if you're trying to see some i don't know some loot on the floor maybe the entity position is good enough but for most characters it's not then you could use an offset i've seen many games using like a meter 60 offset from the route and looking roughly around the neck or head area and that's what most games use and it works unfortunately it requires that you need to provide with a static offset for every single stance your npcs might have so if they crouch you need to adjust the offset but you also need not to forget ragdolls in many games there's some stealth aspect to it to hide dead bodies so that npcs don't see them if you have enough set sticking up a meter 60 well it's really hard to conceal the the ragdoll by using a bone from the biped again we're tying our logic

to art style and i don't like making it so that the animator needs to think of how the animation will look just so that the gameplay works right um fortunately if you're going to do this at least the changes in stance will work right away but i still prefer a dedicated bone a bit like the the sensor attachment on an npc with the dedicated bone you can easily tweak for any gameplay context so for example if the player goes into cover maybe you want to move the bone towards the wall make it so that he's still detectable from behind but end pieces that are in front will have a harder time because the bone is closer to the wall unfortunately it makes it harder for qa so work on your debug tools make sure that it's easy for them to see where the bone is located the more data you have the more chances you have that an employee will create a

bug it's unfortunate but with that kind of flexibility sometimes you have to take more precautions if your target is the player oftentimes it's better to use multiple bones multi-sampling kind of works right because the point-to-point perception is really easy to disrupt whenever you have even a twig in between the npc's sensor position and the target position the twig could be occluding that position and the npc thinks well i don't see my target when actually he sees like 99 of the target just not that little point he was looking at by using multiple bones uh it's quite easy to just go to another one if the one you're looking at is occluded um as an example i sped it up a bit but you can see an npc looking looking at another one and just the fact that the animation is slightly moving on the npc makes it so that the head bone and the hips bones

sometimes get occluded sometimes they they don't so by having multiple bones then pc walking can always see the left hand and the end of the npc and know that he's seeing the target warning whenever you're using bones you can run into level of detail issues if your skeletons stop being updated you need to be careful not to be using its data anymore so in a lot of engines whenever the npcs get far away only the root and maybe the hips get updated so if you were trying to get the head position it's possible that it's not there anymore so you need the fallback system to potentially go back to a fixed offset on the route using ranged weapons can also lead to some issues and there's quite a difference between a line of sight and line of fire because most npcs will have their gun at the lower level than their eyes having two different

referentials can can give different results so the npc is seeing its target he wants to shoot at it but there's no line of fire therefore he's going back to his decision making and be like well i would like to shoot at it but i don't see it so one of the solutions that i like that i use in the past into unif is to unify the referentials as soon as the weapon comes out so basically the npc starts seeing from its gun or shooting from its eyes most of the times it's better to just see from the gun occlusion what's causing the occlusion well in most games it's static geometry dynamic geometry and see-through geometry i'll explain in detail what's the difference between all of those for static geometry it's basically anything that's building the static part of your level so walls roofs ground anything from rocks to trees it's all static

geometry it's detected using a raycast but the good thing about it is that as soon as the raycast it's one of them you can stop the raycast you know you won't see through it so for as an example investing in some debug display that's some quite simple debug display but it's really useful for your qa to be able to identify why the behavior of an npc is changing based on the the occlusion if you can do something real quick even just lines is good enough especially when you have multiple npcs dynamic geometry kind of depend on the context um you'll be trying to occlude the vision of the npcs using the characters in the environment the animals the moving objects for example in assassin's creed syndicate we add the carriages block the the line of sight but the the the thing is uh sometimes the

rigid body of the uh the dynamic object is a bit too crude and blocks too much so in that case you need to fall back on the ballistic mesh which is much more expensive it's a much more detailed mesh but at least if there's like some sort of slight hole between two parts of the objects the occlusion won't be triggered the relevance also depends on the context by that i mean that if i go back to the example of assassin's creed syndicate then pcs could through could see through vehicles as soon as they got into conflict otherwise it was way too easy for a player to just go run the streets and cut the line of sight and the end pieces would go into a search behavior so in our case whenever the the npcs were in a default behavior they would be occluded by carriages but as soon as they would get excited and going to fight they would simply ignore all of that see through geometry is a quite

interesting whenever you're playing with smokes of clouds tall grass and what makes it so that that the occlusion happens is usually by computing how far you are within that see through geometry and sometimes you want to sum all of the hits that that happen so for example if you have a thick small cloud you might be detected if you're just at the edge but if you're right in the middle the the depth um that the raycast has go through makes it so that you're not detected so by having different thresholds for every type of see-through geometry could be quite useful also quick tip for that if you're using transparent windows just make it so that npcs can actually go through some sort of door and reach the other side of the room because there's nothing worse than having an npc stuck into a room seeing a target on the other

side and not being able to go environment modifiers sometimes you want to reduce the perception perception range in certain conditions heavy storms or thick fog could be modifiers that make it so that the npc doesn't see as far but these can be also applicable for auditory systems for example on a previous game i worked on we had heavy machinery so whenever the npc's was being close to the machinery its auditory system would be reduced making it easier for a player to create noise in the environment and not get perceived if it's possible please update the visual language of your npcs to explain to the player why the npc is not perceiving as he should sometimes something as simple as an additive animation make it make it so that the npc puts a hand over his eyes could be good enough

the modifiers players tend to learn patterns so they play a game they learn the rules of the game and they recognize the patterns if you're breaking the patterns by putting modifiers please just reduce the perception of the npcs if you're making it more punitive by making them see further the player won't understand what's going on and whatever you learned just gets thrown out the window so if you reduce the the sensors the player can still be safe within the ranges that he learned and he can exploit it a bit more because he realizes that the npcs are being modified also make it a stack i unfortunately saw it a few times in game engines that modifiers are global on an npc so a classic example the mission designer makes an npc blind you go next to him you throw a flashbang

the flashbang puts an effect on it blinds the npc as soon as the effect expires the flashbang is like well you're not blind anymore and you just broke your setup so if you're stacking modifiers it's much easier the flashbang will simply push a modifier pop it and whatever the mission designer put in place is still there light and shadows um that's where it gets complicated so if you can avoid it i would suggest you to do so uh determining what is lit in an environment it's usually quite expensive you have to go through all the nearby light sources and do a ton of raycasts and oftentimes it requires multi-sampling as we were discussing before with seeing a target with multiple bones if you have a big thing that you're trying to see if it's lit or not you probably should be sampling multiple points on it though that solution simply won't

fix the issues of having a dark shape in the lit background it's possible that the entity itself is receiving shadows from everywhere but standing right in front of a fireplace for example making it quite obvious for anybody looking at it that there's something there maybe i don't see the detail of it but there's something there and the raycast solution won't fix it unless you're adding even more raycasts at which point your engine programmers will come and hit you in the head and seeing light source moves and seeing projection projection projection projected shadows again it's uh it can all be fixed using raycasts but that's probably not something you want to do reflection is the same problem raycasting could fix your issues but it's really hard for a player to understand what's going on especially if the player is not actually looking at the reflective surface some quick tips for optimization

vision request can be deferred so if you can make it that your npc simply put a request in a system tweet the tweak a bit the priority and then maybe the distance of the npc from the camera its gameplay state or its interactions with the players for example an npc that's actively interacting with the player might require the priority to be put up through the roof at this way you can have uh you can sort your your raycast requests and treat them accordingly and cache as much data as possible if you have multiple npcs testing the same position maybe you can reuse some of that information so just to wrap up go data driven game designers and animators or your friend just give them tools build a library of components that's classic tips that everybody everybody should be using these kind of tips but

specifically for the sensor systems it's useful and um let the the designers and animators take ownership they're not they're not separate from the the the ai programmer they need to work with us into in order to create a great experience and experiment and iterate you won't nail it the first time it really takes a lot of play testing and iteration to find a good setup i'm trying to be fast because dave wants me to stop and fail early and avoid raycasts raycasts are expensive so try to avoid them at all costs test everything um sensors are really binary and they're at the root of everything so if an npc doesn't perceive you it cannot act intelligently towards you so unit test everything and only profile and optimize when needed uh i think that's it i'm not sure if we have any time for

questions awesome real quick the you said it was perhaps a disadvantage to have your eye point wobbling around in the head actually redstorm in fact found out that that was almost mandatory because the animator had a idle loop for a guard there was a player sneaking up on the guard with a knife the guard has an m6 has a ak-47 and the guard's idle loop includes the occasional the occasional glance over their shoulder right at the player who is like five meters away and the and the guard is totally not seeing them because the bone at the chest was the perception bone and not the head exactly that's why i'm trying to have my teams always use some sort of hybrid approach where the animator can control when the bone is turning so the npc never the player never feels cheated that the npc randomly glanced over his shoulder but if he does the player will see the glancing animation

and then the sensor will start turning making it easier for a player to avoid these kind of situations and if you guys have any other questions i guess i'll be at the wrap-up room or you can contact me on my work email address thank you you