Turing Tantrums: AI Devs Rant!! 2019 Edition

[Music] this is a perennial favorite we love doing the the rants it lets us get things off our mind we things get said that cannot be unsaid so lots of fun lots of fun and we're going to uh just get this going right away and just kick this off with brian schwab all right so you guys know kind of how this goes usually you put up some awesome title i talk about a problem you all have i offer some sort of solution i make now loud noises i pause dramatically then i give a surprising blurb and on a final twist involving current events which then leads to a timeless truth rousing call to action nope [ __ ] all that um this is mine you're in my house right now and i just want to yell a little bit and i realized that i'm actually one of the oldest people i know in game a high so i

get to do an old man rant for the first time ever so back in my day i got a computer and a sleeping bag that was the day one and by the way i had to take that computer home nine months after the company folded because they didn't pay me that last month network nope we had no network for the first two companies i worked at if you wanted to actually use version control you you stood up and you said who has the game and somebody gave you three discs with a rubber band on it and if you had the discs at your desk you could check in check in code so don't [ __ ] to me about like not wanting to use version control imagine not having it so by the way no network also means no internet there's no reference materials you know that big box of debt of books behind your desk that give you all kinds of stuff nope none of it you know that few people

that work at the company that have been there for 10 years or maybe some website that has a buttload of information none of it we you guys live in a freaking world soaked with information when i was back then none of it in fact this conference was horrifying back then because you would come here everybody was scared crapless to tell each other anything and so i remember the first two round tables i went to everybody stood there looking at each other they were afraid to not come to the round table in case somebody did give up a deep secret but nobody gave up any deep secrets they all just stood there like horrified poker players with three dollars in their pocket and quality of life i actually thought that that was that man i was in the game industry until somebody described what quality of life was and then i realized no that's not for me that's not why i went into the game industry i'm not sure why there's a square on this screen anyways that's another thing to rant about all right so

finally what i wanted to go over for you youngans is some some vocabulary lessons things that maybe you've never heard of before the first one optimization and i'll say that word again because i don't think a lot of you have ever seen it in print optimization this is when you take your game and you scrunch it down so it runs in a well in a good in a good in a good fashion you make sure that the assets are me are meaningfully sized you make sure that the code runs at a good frame rate all of the things instead of just re waiting for next year's computers to get twice as fast and memory to go down in price restrictions are actually good for you they actually force you to do a better job we barely had any memory and cpu when i was when i was first in games my first game i think the entire game engine fit in in 32k with the data it was ridiculous like i

have icons on my desktop that are bigger than that um i actually built a mechanic into a game once which effectively had the players choosing the the people that went into a mission so that it would be it would budget out the video ram for me it forced me to come up with things that would actually help make the game better that took into account the restrictions of the system we didn't have any polygons we had to make good behaviors because we couldn't do it in art we could not do it in our we barely had anything we barely had any people i that's one of the things that bugs me the most is when people are like oh but my game has three three million man years of content and it's like yeah and you've got 700 people on the team back in the old days i was not only the lead programmer and the designer i was also head of hr and the janitor and let me tell you i still have my broom and i'm proud my first game had less pixels in your average tweet litter count

that is dangerously close to true second word debugging i don't know when this became a lost art but it is a lost art coding is only eight percent of the battle uh um when i was around a debugger was actually just log statements and you had to pray that it wouldn't slow the machine down enough that you could still actually find the bugs and a profiler that was even lamer like like you were printing out a counter during loops or during different parts of of pauses so that you could determine how much time was spent in between chunks of your code nowadays you've got beautiful debuggers with hardware breakpoints and all sorts of crazy goodness and profilers that visually show you code stack traces in the whole nine yards this needs to be taught i can't for the life of me understand why this has never been taught in schools i've never seen a class in debugging i've never once in any curriculum i've ever seen a class in

just debugging and we don't need to just teach programmers this i have because of the profligation of of of design driven uh uh content a lot of designers are doing a lot of scripted logical sort of notions and they don't know how to debug it in either i have literally had this conversation with a designer with a fairly high level designer i'm now a high level designer the ai's broke the ai is broke uh did you just pull down code no did you just change something in your script yes did it work before you did that change yes at which point i i fight down the urge to throw them into outer space um the queen is not amused with your debugging skills uh uh and and the awesome thing is is like once we actually start teaching

debugging skills once we actually start putting more debugging systems into our games we can get more people to debug our games we can have qa people that can actually help debug even your mom can help debug because if we don't start teaching people how to debug i am going to finish the infinity snippet and i am going to get rid of most of you myself and don't make me snap more than once finally the last one innovation and i don't mean innovation in games we're actually in a fairly nice renaissance for for innovation in games there's so many indies pushing so many different types of games right now that innovation in games is not the problem what i'm talking about is actual game dev that that is really horrible and it's because of two things that i say um not invented here syndrome and game education and what do i mean by that right now a lot of people just learned how to code they quickly went to school they learned how to code they don't really know how to program they know how to code in maybe one

language when i went to engineering school i actually took a single course in assembly language programming where they taught us 11 different processors in one semester and what did that teach me it didn't teach me 11 processors it taught me uh data and code flow with different types of architectures all over the place it taught me that syntax is barely barely matters that what matters is what types of inputs what types of outputs how much time do you have blah blah blah all of these things it taught me how to program we don't do that much anymore and it makes me sad the other thing that sucks is that in especially in game ai we we kind of had this quote-unquote revolution where we stopped kind of using state machines and we started using a different structure uh and you know when damien isla as much as i love him as much as i love him when damien

kind of introduced everybody or a large chunk of you to behavior trees i really get the feeling like he sort of traded out cubicles for an open floor plan [Laughter] he kind of took us from one hell to a slightly different hell and the sucky thing about it like because i i agree that for a huge swath of video games behavior trees are actually better than state machines because of their reactivity and because they're easy to add to for a long time but that's when games were like that like nowadays games are barely like that and i'd see talk after talk after talk of people that are like well we started with behavior trees and we spent two and a half years getting around the problems of behavior trees and then the next talk is somebody doing the same thing from a different direction and on and on and on one tool is not the only way to go and you really need to sit down and realize again what is your inputs what is your outputs what are the types of things that you want to do because most games nowadays are not as reactive as

behavior trees want you to be and you spend a lot of time trying to make sequences a lot of time trying to make more authored content all of those things easier to do without blowing up your behavior tree and lastly i really really want like i said to get a full curriculum in education right now i see too many classes being quickly teach them a little bit of cs quickly teach them a tiny tiny bit of game design and then jam them into a game engine and start making a game nobody's teaching anybody how to do the actual things that are right and nobody is teaching debugging like i said before and then finally like i think the reason that they're doing this is because they're trying to say it's so easy it's so easy you can do it you can do it but the other part of this realistic expectations that i mentioned is that there's not as many games jobs as there are people getting game job education and we really need to get around that problem as well because i think if people understood that there isn't a lot of jobs they might actually work harder at becoming better at the job

in order to go out there and become the people that we do choose to work in the industry so finally any questions because the only answers i have are get the hell off my lawn get off my lawn and yes get off my lawn [Applause] right uh so i wanted to draw attention to this title um seen this title before but i wanted to show it again because this is a rant session and i have to say that i find the rant format extremely challenging because even though there's a lot of encouragement out there for women to express rage and really put out what they feel angry about and own that feeling i'm really [ __ ] at rage i'm just bad at it i've never been that good at anger if it were a skill i roll low i mean i

do feel anger about quite a lot of things and quite serious anger and i think these are kinds of things that we tend not to talk about in this rant session because they're terrifying and maybe they feel a bit off topic and who really wants to bring down their day with this stuff i am going back to live in the uk and i'm not sure in two weeks whether it will be part of the eu or not that pisses me off okay but what i struggle with here is the effectiveness of the form like an angry rant is basically a bug report you're letting people know what is wrong with the relationship or the community or the institutional system that you're part of and maybe if you're a ranty person you get the reputation that you file all of your bugs is critical and there's a huge backlog anyway nobody ever has time to clean up so you know and besides you can hurt

people's feelings again a blank slide it's on purpose i didn't screw up it's because i'm bad at this right you can hurt people's feelings by being angry i'm not keen at doing that but there are times when it's necessary the thing is that the expression of the anger doesn't fix anything by itself it's not a patch for the problem and wherever possible i'd like to be suggesting possible solutions although for some problems i have to admit that i'm really out of my depth so last year i did the rant session and i thought okay i'm not good at anger but the rant is a genre it's a format you get up here and you do kind of stand up routine about something you'd like to see changed and everybody who's coming to the rant knows how that works and it's a particular context so this is fine um so last year i wanted to do a talk about how we need to do a better job of inserting nuance when we're asked broad questions about ai

that miss the point of what ai is and how it works and i wanted in particular for fellow devs to think about how we pick what to say about ai how we represent it to the public and even to each other within development teams but it was a rant so i picked a ranty title and the consequence here was that games industry biz published coverage of my talk with this headline which gets better if you read the subtitle and the journalists did get in a number of quotes where i think it's clear that i was talking to devs about the role of devs but naturally people honed in on the title so the next day i had a bunch of annoyed journalists on twitter asking if i realized how hard their job was and complaining about my contempt for laypeople which was not the point so obviously that was a relatively low stakes case where no one really got that angry or offended it was basically fine but it has me thinking about the way that our format shapes what we communicate and how there's an interactive fiction

author i greatly respect name porpentine who recently said that she writes fiction more than she writes nonfiction because stories allow her to be more truthful and actually didn't elaborate very much on that statement but what i took from it is that when you're writing an essay the genre of essays takes over from you a little so if you're writing an essay or an opinion piece you're supposed to have a thesis statement you're supposed to support it with arguments you're supposed to know what you think and it's supposed to be orderly you might have some nuances you might have admit some exceptions to the case you're making but structuring all of that information into an argument means that you're enforcing a hierarchy of meaning on the thing that you're writing you're saying some of it matters more than the rest and in fact one of the things that i like about writing is that it forces you to come to that kind of understanding to write an essay about something you're not sure about is a way to understand your own thoughts but there are things that are hard to communicate that way and the same thing is true for telling a story except that a story invites you to present struggles that might have no

simple answer or describe character experiences as they are without putting an interpretation or a priority set on them and of course there are a host of other genres and subgenres we can use depending on what exactly we're trying to communicate and how much we care about wearing shirts not only to entertain and amuse ourselves but to structure different types of information and different meanings now at this point in the in the rank you may be thinking i see the ai summit logo at the top of the slide but are you sure this is an ai summit talk i mean emily are you lost did you wander in here by accident from the narrative summit or like the game educators summit or maybe the advocacy track could you say something about machine learning now or very least procedural content generation and to be honest by the way the climate change pictures kind of ruined the mood but maybe some of you are ahead of me because what i want to think about sometimes is ai as a practice that lets us understand the world and communicate in new ways

simulations are how we write about systems how we imagine possible functioning worlds and ask ourselves whether we've imagined them right or gotten something terribly wrong last year i argued that we could do better at not exaggerating what ai is actually doing that we shouldn't present decisions as belonging to human level thinking when they don't we should be precise as we can about what it means for an ai character to think about something or prefer something or like something but to pair with that i want us to think about the practice of ai as an art form and as a means of expression so when we talk about metrics for machine learning models we often use words like false positives and accuracy that make it sound as though there are no human being was involved in determining what counts as a positive in the first place when in fact that is often the most important thing that is going into the system and if we are going to solve the really big problems it will be through the practices of data science and ai not because we'll reach the singularity or build an artificial general intelligence capable of perfecting the

world's broken systems but because the discipline of thinking towards good artificial intelligence requires us to understand the important features of our data requires us to seek and actively work against biases in our information requires us to decide what experience we want our systems to produce and how many failures we're willing to accept for which users and the thing about game ai specifically and finally i bring it back to actually this summit at this conference the thing about game ai is that in games we're allowed to talk about the fact that we're building aesthetic experiences on purpose we can admit that the ai we are building is not objective and it is important for us to tell the world how that works [Applause] hey everyone i'm ben i actually wasn't sure whether to put this talk here or in the simplest track in the book session uh like it's helpful advice but

there's a lot of [ __ ] words in it um anyway so here's my uh one simple trick for a way to improve your eyes ais is uh tell them to shut the [ __ ] up it's enough with the rants ever since i think maybe the game fear we've been giving npcs all these lines where they're just describing the situation they're in which is fine i guess even members of an elite paramilitary force sometimes just got to tweet about their feels but these barks are not helping you you've been doing them for all the wrong reasons like we're not usually making games to be honest where intelligence really uh organically comes up so instead we feel a need to just shove it in there like the player throws a grenade and so an ai character says [ __ ] a grenade i guess so that the player's supposed to think wow he comprehended my grenade what incredible ai listen man you saw me holding that thing for like three seconds and you didn't mention the topic then let alone try to take cover if i see somebody holding a grenade and like imagining parabolas at me

i'm not gonna stand there and see how the situation plays out and if they end up throwing it at me i'm not going to waste time yelling at the thing now one idea behind barks i will admit is reasonable if a little cowardly because players can misinterpret the ai right like a character decides to flank but from the player's perspective what's happening is the guy ran off and then later on somebody else magically appeared behind them and shot them and that's not fun whereas if the character says i'm going to flank left you know the players left while he's rather lost the element of surprise at least you get some credit for what your ai's doing also the idea of companion ai in a mixed world game prompting the player where to look that's a really good idea a really bad idea okay so true story i was playing a game a stealth game and i was hiding and somebody was looking for me and the character said the line and this is the line he's got to be around here somewhere what he's got to be around here

somewhere is not a sentence you use with grown-ups it's something you say if you're playing hide-and-seek with your five-year-old and you hear giggling coming from behind the couch it's not something you say if you think there might be somebody hiding in the rafters with a gun people your fellow developers are doing the their best to make the player feel immersed like they really are hiding in the rafters and then you got to come in and deflate that balloon with he's got to be around here somewhere of course what the guy is really saying is i'm in search mode so the player knows to stay hidden even in the midst of showing off his intelligence he's undercutting it by showcasing the stupid unnatural limitations that the genre places on him he's got to be around here somewhere he's got to be around here somewhere must have been the wind it's okay guy i'm back to patrol mode the real irony is that we think of barks as a one trip as a one simple trick sort of thing it's like it's an easy way to bump up the

intelligence quotient just by playing sound files it's like we're trying to cheat on a turing test like instead of having an actual conversation with the player the player can't talk and instead our guys are just monologuing about taking an arrow to the knee like if the player could talk that's what they would be asking about well guess what your one simple trick it's not [ __ ] working of course it's not ask any artist what's the hardest thing to draw and they will say the human body or in particular the human face not because the human face is uh inherently complicated but because we humans are so good at analyzing human faces at picking out small problems well we humans are also pretty good at communicating we know that people don't walk around saying must have been the wind to nobody in particular and we certainly understand the risks of sharing information with somebody who's trying to kill us and this isn't about having good writing it isn't about having good voice acting you can have those things and it doesn't matter even when it

doesn't sound artificial it still sounds unintelligent it has to because it is you cannot make someone seem intelligent and threatening if you make them complicit in their own defeat it's like what brian schwab said a few years ago about game ai's job being to lose with style [ __ ] you schwab lose with style there's no such [ __ ] thing a character losing with style means that their style is to lose don't patronize your player how is that challenging how does it engage the player he's got to be around here somewhere that style how do you imagine the player reacting to that well reasoned my friend i do have to be around here somewhere you truly are a worthy opponent enough with the barks enough with the insincere i'm going to get you play acting from characters designed and programmed to be shooting gallery targets if you want to show the player how intelligent your ai is make it outsmart them you want to

challenge the player engage them give them opponents who shut up and try to win thanks [Applause] all right i'm going to channel some anger might have forgotten to make it funny um i've been talking about this for a long time and for those of you saw my session earlier there might have been a bit of a disconnect because i took a lot of anger from that session and i brought it here so let's get started a couple of years ago maybe five years ago somebody came up to me and they were like oh yeah yeah it gave me i 90 percent that gave me isaac star and i was like what the [ __ ] are you talking about and then more recently i got given this quote ninety percent of ai in games is npcs and that's [ __ ] [ __ ] so

if you take a very myopic view on what ai and games is yeah okay we'll sit here and we'll talk about npcs and we'll talk about pathfinding and we'll talk about box not that i'm picking on you ben um who [ __ ] cares we stifle ourselves by focusing on npcs we limit our impact and if you think that 90 of ai in games is mpc's it might be because you've only thought about npcs this is what the games industry looks like 2.4 billion gamers in 2019 the sector revenue is 148 billion dollars half of that is going to come from mobile that means we've got a big secret here a significant percent of our industry doesn't even have npcs and our industry is an industry and our studios our businesses and our games of products and maybe

just maybe we ought to think about ai in bigger context than just [ __ ] npcs the ai revolution is here the ai as a sector is going to hit 168 billion dollars in the next six years um we're adding value across all sorts of sectors autonomous vehicles home automation you name it ai is adding value right now and in this city in particular ai practitioners are the gods of modern technology we have the capacity to rethink and reshape how we do things and you know what we've been doing this [ __ ] longer than everybody in this city calling themselves a machine learning expert we've forgotten more than they [ __ ] know because we've been doing it for 30 [ __ ] years so you can focus on your npcs if you want i guess if you want to chase a cool line in a review somewhere about oh and then the mpc said oh he threw a

grenade um like what's the point you have the capacity to add millions of dollars of value to your businesses so if i hit one more [ __ ] talk about pathfinding i'm gonna lose it so yeah that's it that's great all right so nicole lazaro and uh this is um yeah we're gonna see about this all right so this is my this is my rant um so i've worked on in games emotion for 26 years about emotion and interaction so that's where i'm going to channel we're working on creating an emotion engine for some of our games

and therefore my rant is about self-driving cars now i don't want to but i don't know what about this is and why why we are so obsessed as a as a as a silicon valley and as a planet with this concept of self-driving cars but it seems to be grabbing like a lot of lion's share of uh of you know of industry and what we're trying to do or what some companies are trying to do and is it i mean what is the value proposition of a self-driving car some people you know claim that we'll reduce traffic maybe it'll be you know a couple iterations down the road is it that we want to read our comics you know as we drive to work or as we get driven to work do we want a facebook on the back seat like michael jordan here or do we want the cheery face of a automobile saying hey i'm on my way on the on the grill as it picks you up uh by the way the i think that there's something wrong with her because she looks like she's in a neck brace um and i think that this this problem is like is this a problem to solve and that really is is

is my is the heart of my rant is a self-driving vehicle with all of the expense that goes into it all of the earth resources that will have to be created or mined in order to create replace the entire ecosystem of cars is that really worth it and is that the problem where is that a problem we're trying to solve and then the problem we solve is not this one either which is the on the ai aspect of like how do we keep them safe how do we make it more efficient how do we get people from here from point a to point b no you want to think about well what is the real problem that we what are some real problems that we have as a society and as us as interactive entertainers our mission is to help people imagine it's our mission is really to help people of that next generation come up with new ideas to be able to see the world in a new way so what i say is that the real problem is this you know we are you know we are a planet where you know depression is you know a worldwide you know epidemic we are we are losing connection with with each other with our loved ones we are getting sucked into things like

social media and because i think there is a opportunity because i am both a ranter and a solution provider is that because there is a connection between how we act and how we feel what we do in games and what we don't there really is a if you saw my talk earlier there is a really opportunity to create new new feelings new emotions new patterns of behavior the problem to solve i think in ai for games is not this problem either it's not you know the threat of life is just one emotion that we have and that one emotion that people experience it really is the relationship with others sort of a human-centered approach to uh to ai it's a people first like what could what sort of value could ai add to this interactive experience that makes us a better human that improves our quality of life that makes us smarter that makes us you know enjoy more time with our with our friends and you know i mean wtf you know where's the fun where's the fun ai right we want that that that um that emotion to to come to come forward and so with you

know the opportunities in narrative ai sentiment analysis and you know making players happier i think there's a real opportunity to create not just more believable ai more better pathfinding and you know more more intelligent barks i guess but it really is to craft emotion and allow players not only the opportunity to experience the emotion that you have but really give them the steering wheel to create the emotions that are missing in their life uh which is you know possible through you know the interaction interaction design and uh and through ai so with the four you know with four keys is just one lens into this space but we can actually through the actions in game help create uh different emotional states move them uh and they choose what actions they take so they could actually move themselves to different emotional states so much of width is rather it's not just really a choice between the red pill or the blue pill it's really a choice of a different order in a sense that we could provide games that give players their daily dose of dopamine oxytocin serotonin

endorphin these are wonderful um wonderful chemicals that feel good oxytocin is of course the huddle cuddle hormone but also it's also produced in angry situations as well social bonding social ranking is serotonin it's what drives the likes on facebook and endorphins of course mask emotions and so you can actually get quite a long way if you if you give players uh these different emotions in sort of every you know layer of your gameplay so imagine if your ai was really skilled at uh precisely targeting different emotional states and that through the player whether or not they were consciously aware or at least you know behaviorally aware of moving and using the game in this way or that way could really round out their lives you know really could improve the quality of their interaction with their friends could make them feel more associated with their with their fellow peers and that that you we could actually add you know these uh these mechanics uh to create different different emotions in the player at every level of the of the game and uh you know connect us

all all as a planet as a planet we can create meaning in games we can create significance the feeling that i matter and an ai that's trained to do that that that helps the player even to train themselves to give themselves you know encouragement and move through move through the world is a much better uh i think a much more nobler goal and a higher stretch and i think that's something that we're all very very capable in this room and so it's not so much you know what what tomb you know tomb raider lara croft raids it's really what makes what really really matters is uh how you feel after after the game you know playing with your friends so thank you [Applause] so some of these rants like that one were very feel-good type of rants mine will not be that kind um several what was it like three years ago or something brian

schwab did a rant where he's like everyone has like this point to make i have no point to make i'm gonna complain about five things that annoy me i could have easily made it 10 15 whatever but these are five things that have been pissing me off lately and that's that's the entire rant that's what it is so let's get started all right number one a lack of tools if i come onto your project and there are no tools whatsoever to deal with the ai that's going to be the very first thing i advocate for i was on a project once that was pretty far along and the only ai visualization was setting a break point setting a break point what dystopian horror is this like really i can't even see like you can dump a debug log to a console window in fact i'm so annoyed by this that tomorrow i'm giving a simplest my simplest trick in the book is how to create a really simple debug log and not just a console window but like mixing a c-sharp thing like you can do this trivially so if you don't have tools to actually see what your ai is doing then what's the point you're gonna step through like walking through your

horrible behavior tree that sounds terrible even on the sims 4 in the very very beginning when all the sims looked exactly the same and it was a green flat plane and that's all we had at least we had a console window where i'd spool out a bunch of debug logs of like why did this make this choice at least do something right don't [ __ ] us over on tools come on what are you doing okay number two and this one has gotten me some interesting looks single letters for template parameters did we lose a war like what the [ __ ] is this single letters for attempt okay t really okay sure it's type all right follow me here bad bad bad right these are all bad why is this okay it's a slippery slope right now there's this one i don't know what that means sure t is type but what is this or this or this like and maybe you think it's uh hyperbole right you're like oh come on rez you're just using letters in the alphabet well i mean this is eastl's hash table

right or it's insert you know it's it's hash table type i mean come on like i know what many of these do only because i know how it works on the inside but just looking at this i can't tell what any of these [ __ ] things do like really so please like name them something useful anything useful all right good god number three shitty interview questions yeah thank you you know who you are and when i first started interviewing i gave shitty interview questions right are you really going to have like what value is it for me to stand in front of a white board and like write out in a pen some algorithm that i could look up in two seconds in any book right like okay it's fair game to have me generally describe a star but to sit there and implement a star on the whiteboard with five high status engineers all staring at me that's exactly like my programming

environment at home i don't know about you guys like [ __ ] you i mean let's take something trivial super trivial binary search right super trivial if you had me go up and implement a binary search on a white board there is a at least 50 chance i'm going to have some off by one error certainly and what is that really showing you because this is probably how i would implement binary search in an actual game right i would call binary search across whatever that that structure is so what are you doing like what is that actually telling you about the quality of that engineer that you're hiring like really maybe a better thing to do if you want to check their coding ability is to give them a game design and say make this game design and then go away while they sit there with their headphones on and google in front of them so they can remember how some api works and like actually implement a thing in visual studio or whatever your your poison of choice is that's maybe a better representation of how you write code all right number four and i can say this one because i work in academia

terrible yeah and uh and i'm gonna look at you on this one as well uh schools that focus on high-level cs concepts rather than actual practical software engineering right it's true like okay great fantastic you can do a a whole proof on exactly why this algorithm is you know perfectly theta of n squared or whatever that's fantastic i will never ask you to do that in any job anywhere i ask you to understand the basic concepts of how that stuff works but like some of the really deep stuff especially to the exclusion of actual software engineering techniques so one of the classes i teach is assembly language and uh optimization you're welcome and there's a question that i give them i actually got this on an interview once and it's a really interesting question this is i think proves my point so bear with me here because you're all about to get an interview question ready you have this function

um this function all it really does is it takes in three parameters right has some takes in three parameters uh an array a size of the array and and a value that we're looking for specifically a sum that we're looking for so it's going to return true if any two values in that array summed together equal vowel to find it returns false in any other case that's it that's the question how do you write this function so there are many ways to write this particular function um they're sort of the brute force way right so we can do that one and you know that's n squared sure if we want to get into asymptotic complexity um which where we just churn through the whole thing over and over and over that's fun there's a sorting method right we can sort the array and then we can uh search we can basically take the uh the val define subtract the value that we're looking for as we walk through the array and then do a binary search across that array so now we have okay i'm going to call it 2 n log n um you know if we're going super proofy here then it would actually be n log n because multiplicative

constants and so on right okay moving on hash map that's another way of doing this right it's basically the same thing except it amortizes towards more constant time but now you have n in memory right who gives a [ __ ] because here's the here's the reality i can take that input array and change the values just some constants the values that are in it the range of the values that are in it i can change how big the array is and all sorts of different things by just changing a few of these constants i can make any one of these three in a real computer greatly vastly outperform any of the other three and that's the demo that i do to prove to them that like okay asymptotic complexity you know big o notation that's a place to start but in real actual computing it's completely different right because there's cpu cache and there's all this other stuff that comes into play there's what about all the memory uh what about all the allocations that could happen in the hashmap right does the hashmap have to regrow bunch of times we have to rehash everything all sorts of things come into play so okay sure write your proof that it shows me the exact perfect asymptotic complexity of any of these methods and i can show you a data

set that will blow that completely out of the water because that's the reality of how computing works okay um designers and i love you designers but designers who are unwilling to learn technical things that's number five we are in a technical industry right i have had designers who are like who are unwilling to learn how the ai works really i've given that i've uh around maxis and i can say maxis because all of the designers there were wonderful so i could say only positive things but there were many designers and i would say okay here's how the ai works and i would draw like math on the board and i would watch like the eyes go more and more glassy as more like greek letters started coming out and it's like okay let's distill this a little bit but to their credit each and every one of them stepped back and said okay let's really understand this let's explain walk me through the step by step and i see a lot of designers especially new designers who come through

and they're like you know no that's that's that's programming's problem right our job as programmers as ai developers is maybe not even to make the next cool thing my job might hopefully is to enable designers to be creative but in order to give you those tools you have to meet me halfway right i can't just miracle this thing in front of you so you know and i'm not saying that you have to learn all this stuff you don't have to understand any of the stuff any designers are there any designers in the room okay like three oh yeah a handful right i love you guys i really do and you don't have to understand anything that i said on the previous slide at all i don't care about that but i do expect you to understand the basic idea and be willing to learn very basic things and here they are if you're a level designer especially basic vector math basic vector math not crazy basic and the love of god probability theory if i will i will just and i'm going to lose it if i see one more designer that doesn't understand the basics of

probability theory right and just basic probabilistic math and that sort of stuff because that's the world that we live in anyway i don't know so that's me thank you okay so to kind of end this for my rant i was maybe it's not so ranty but i wanted to really think hard about moving the ball forward about pushing game ai really to the next level and really ben already ranted about this two years ago and he had some some great things to say he said what are you doing to save game ai because no one is coming to save us that was and then he further went on to say a little bit more inspirationally but we need to build it into something worthy of notice we need to get over our ideas of what is and is not possible

we must do things that we do not know how to do we must create experiences that we thought were not creatable and we need we need to push the ball forward and i i started thinking back to when good old behavior trees first came on the scene what what happened back then and planning happened just around that same time and so as another old man on the on in the talk i think i will re regale the the tale that uh what was happening at the time was games are starting to get more complicated more npcs more oh i said npcs more characters and state machines weren't cutting it and independently people were trying to solve this problem it was sort of a crisis um and so behavior trees from from halo 2 planning from fear or gop so what this really comes down to is if you want to move the ball forward it feels like there's needs to be incentives right incentives to to make new things

happen so i mean we all know about deep learning what the heck is going on with deep learning right it's in the news all the time we're hearing lots about it um we keep wondering how it might affect uh all the parts of our industry but with deep learning what's really happening is the incentives are right and it's not necessarily the the game industry but what deep learning is about is about predicting things it it makes the cost of predicting things cheaper and so harvard business review in an article thought that deep learning could actually deliver something like almost 6 trillion in annual revenue to businesses which dwarfs the game industry and so this is and so in the article it said follow the money right this is what's what's happening and so where where in the in business is this happening well it's retail it's retail with serving ads and promotions to the right people it's it's in operations if you can predict better you can reduce inventory costs which can save a lot of money and

manufacturing if you can predict which machines are going to go or fail right you can keep keep everything uh going now just very recently alpha star was developed by deepmind to play starcraft 2. it was a deep neural network trained with supervised learning reinforcement learning and it did beat some top players and it was an impressive feat a little less impressive when you find out what it took right it it had to train on half a million human games that's a lot that's a lot a lot and the agents that they trained started playing against each other and experienced up to 200 years of real-time starcraft play which kind of makes this like well how do i ship a title with any of this if i have to collect all this training data and let the and then be done with the whole game before the bots can play each other too right and and they're like is this really the biggest problem trying to

beat the top players right and this wouldn't have made um right starcraft 2 a such a better game that it would have sold more but what does deepmind doing right well they don't care about games right they they like games but they're they're using games as a platform to prove their tech so then they can apply it to the six trillion dollar industry right where there's a lot a lot of money so we we do when so thinking about incentives well how do we move the ball forward right we well we're gonna need to create some incentives it might be might be what we can do so so ben gave his rant two years ago and i you know and and his incentive sort of was his incentive strategy was uh shame right shame you shamed the audience which that might work but it probably won't it'll make him

feel bad for a little while okay so what are some better incentives right so well just trying to make a ground breaking game it might it is an incentive right when we saw it with spider-man earlier this morning and we're gonna see tomorrow the division two right groundbreaking games do push do move the ball forward and and that is that is one way and that is great how about ai summit talks right maybe maybe they spur some innovation we're going to have an experimental game ai session tomorrow which you should totally check out and hopefully that does help move the ball forward and and we're doing what we can at the ai summit maybe we should just get a billionaire to pay all of us that might yeah yeah that might uh create some incentives uh so it's certainly on the table if we have any takers um a game jam or a hackathon i haven't really heard too many

game jam or hackathons completely centered on game ai so that might be by might be some incentives or just maybe a well-designed contest so there have been some well-designed contests that have moved the ball forward this is nathan sturdivant he worked on the or he created a grid-based path planning a competition this is not a memorial picture he didn't i was i was reviewing my slides and i felt slightly bad [Laughter] he was not born in 2012 and he did not pass away in 2014. he is still alive he he was able to get real maps from real games uh a bunch of starcraft maps dragon age origins because well he had worked on the game and warcraft 3 and uh you know he had those maps and he could use those maps in that contest some real world examples

and because over those short few years a lot of neat algorithms were tested and they could be compared against each other and we could really understand you know what's the deal what what should we pay attention to and it it did spur things i worked on jump point search plus with goal bounding and i gave a talk about it here in 2015. and it it that contest kind of helped me you know work on that problem and even be able to compare my solution with other people so that that's something um this is ben weber he is also currently alive [Laughter] so in 2009 he started the uh starcraft ai competition which was which was wonderful um but recently over the summer he wrote an article called the end of ai competitions and what his he kind of cited some some issues with what's going on is is with a lot of games uh we still have closed apis which make

it very hard to to program what you need to program if you want to make um competitive ai within a game that's a shipping game um with deep learning and a lot of these new uh techniques that are kicking some ass uh the compute is very expensive so um luke told me that uh so ben works at zynga and and luke told me that uh ben uses way too much uh compute and zinc oh he uses a lot a lot of money goes but rece the problem is researchers can't do that students can't do that normal people can't do that they can't pay thousands of dollars a day or something to pay for compute or for a project and you need access to a lot of training data right for for these these techniques so it isn't so there it seems to be this problem that is is and then and then deepmind came along and who's going to compete with that who's going to compete with hundreds of the top top ai researchers uh for you know deep learning in the

world um so here's what he said he said in a way my initial call for research on openai problems failed um it was an open problem but i didn't fully encourage participants to be open in their approach to the problem so sometimes people don't share what they they did accomplish or even if you they do share it it's very hard to use um have you heard of kaggle i gotta pronounce that right it's not kegel it's kegel um so this is a this is a well it's a website and a group where there's hundreds of competitions with real world open data sets to solve data science challenges and they give big money like fifty thousand dollars to the winner and what happens is they get corporations to come up with a problem uh give away their open up their data sets uh and then uh see if you know people can apply machine learning to it and come up with something that can help them make their business better and uh google has purchased uh kegel so uh um as as they do also own

uh i believe deep mind um but but that but kegel has shown the power of certain machine learning techniques and it's only like by having these kind of contests out in the open that you really see and can test some of these things so competitions do seem like there's some kind of incentive there um that's interesting so it would be interesting to think more about um competitions for game ai but what are some good incentives you know what i think people already love to make new things we all we're all in this because we love making new things sometimes maybe the right incentive is just just knowing what to make right lots of us just don't maybe have the right ideas so maybe the first step in moving the ball forward is is making a list making a list of really great challenges uh right we need to start with a list challenges aspirations goals right and as just kind of recap what ben

said um you know build game ai into something where they've noticed get over our ideas what is in is not possible do things that we do not know how to do create experiences that we thought were not creatable so right we we don't want to maximize winning right so the whole starcraft competition is sort of um sort of flawed uh we're a little interested in maximizing performance a little optimization there um we definitely speed of development we want to do more with fewer people right we so we can make uh more interesting games we want to basically maximize entertainment which is a very hard fitness function right that's very hard to measure that is that is we're in trouble is what we're we're in trouble no we need to max but but if you think about it as maximizing wow moments right that would be something so what are these things i tried to list them out and i was a little disappointed in myself but we'll see what see what i have i want to author behavior quicker than i can do with

behavior trees right behavior trees as ben said in the previous can't be the end of the line right this is not it it wasn't invented 14 years ago and that's it although in a way play devil's advocate behavior trees are a programming language and we use programming languages that are old like c plus there's reasons right they're solid they're dependable there's certain reasons so um but we do want to author behavior also that's more nuanced than i can do with behavior trees and i heard the word nuance from from one of the speakers so there's something there too but and and be able to just script wow moments faster and easier because sometimes that is hot uh the way to do it is hand authored um and create innovative tools to create interactive dialogue or just all tools so another shout out to tools so i really think that all those kind of land in the in the realm of improved tools and languages right so if you could create incentives around that if you could create contests or even just list out the challenges i think

there's something there um let's hear some more so use game ai as an innovative gameplay mechanic you don't see too many games that do that there's a couple like dominisla's third eye crime but we you know to be able to see more like that it would move the ball forward better ways to converse with the npc it's it seems troubling that uh we can't really get farther on that better ways for the npcs to express themselves and heck npcs that grow as a person over time why not but we need kind of more loftier goals right you have to kind of make them a little out of reach or else you don't you're not reaching and so those kind of go around improving ai and game design right this means like ai programmers need to talk to game designers more like a lot more i was just at a our function party last night and talking to an ai designer and i found out that they in seattle have a local meetup and our ai programmers used to have a

local meetup in seattle and we didn't know about each other it was just very weird so now we got to get we should try to get the designers with the programmers and just start talking so i think my call to action is that we need to define the challenges right that's the first step that's the first step to moving the ball forward we need let's inspire each other to create those wow experiences because and then and then the end of this is simply well this is not the end we've got to keep the conversation going but this is the end this is the end of the first day of the ai summit so with that i i want to thank you and tomorrow is a big day we have a lot of great sessions so let's give a round of applause to all of our speakers