Meet your new and improved teammate! Human-directed AI agents and gaming

[Music] good afternoon everybody you've made it through most of PG connects pretty pretty amazing event so far I've heard tons of good stuff I hope I can live up to the standard that's been set before me at PG Connect we are primarily interested in so we're an artificial intelligence company but we're deeply interested in games right so we ask ourselves why do people play games why do we why do we get involved here and I think it always starts right with a narrative with a story so people are always going to get involved because there's a core story that that speaks to them and on and then there are layers

that evolve around that story right then you've got challenges and you've got surprises that happen right once you once you work your way through a certain period of the game you figure you've got it mastered then something unexpected happens and the game evolves and changes and becomes more challenging so there's there's a sort of opening up of the game as you as you play within it you begin to achieve a level of mastery at the end right there's this achievement where you go hey I've pretty good at this now and and this for us is is sort of fundamental to the to the work we all do now you know then we ask ourselves well why do we build them and I think as people we're storytellers right were we've got something that we want to talk about something that we want to express something that we think needs representing in the world and and we're excited for people to hear that story or let that story unfold for them and see how they can connect with it see what it means to them and so now as we look at

AI you know this too has been a story in gaming right the the world of AI in gaming hasn't hasn't always been a rosy one you know so like all good stories that began a long time ago with this overwhelming enemy right and that enemy was the bots all right everybody's had problems with BOTS right you face them in some game you've been screwed over because you were at that point and something was going to happen and then it all went horribly wrong right so when BOTS you know when we Piku bots in gaming right and people like Oh a Rai means BOTS it has right but at the end of the day what what causes problems with a story in a game is when a bot masquerades as a human when it takes on the role when it when it sort of surreptitiously acts under the guise of a human and sneaks into the game and distorts the mechanics right so when it's masquerading as a human you can no longer you can tell by its

behavior that there's something wrong here but the fairness goes downhill right and the your enjoyment of the game is is materially impacted and your achievements are taken away it goes to the top of the leaderboard and all of that stuff that makes the game valuable to us as players starts to disappear right and ultimately it hijacks the story in it and it ruins the the environment so when we say hey we do AI in gaming get a visceral reaction understandably because we've all been in that place where where that's happened to us then the enemy transformed and and and has evolved right because gaming now that artificial intelligence has moved to a new level different things are happening and and I suspect that a lot of you may have seen what happened with at the International this past year with dota 2 we're open AI

brought a whole new level of competitive bots and I don't you know this is just a small snippet of play where I think all of us were probably shocked by the ability of the bots to engage in teamwork the ability of the bots to or the willingness of the bots to run and just sacrifice themselves for the team right this is not something that we as humans are great at you know if you're playing along on a team of five people nobody wants to be the guy who jumps on the grenade but to these BOTS were really really fine with doing that right they didn't have a problem engaging in very inhuman behavior and it made them very successful so and in the bots because they learned the game in their own way started to play different strategies started to kind of unlock strategies that players hadn't encountered before and that was part of the reason that it was successful now since this this particular bot had in the sort of little

brain cell the LS TM cell that kind of figures out how things unroll over time that had a thousand 24 neurons in it you know within weeks of this then they had expanded to 2048 and then there the 4096 and guess what like it became even much more performant so so there's a there's a lot going on with with these BOTS but the really fascinating thing is it's not what it used to be right it can't just imitate and do a few hard coded actions it can do a whole lot more so now we need a hero right we need something new in gaming that sort of aligns these forces and and our view on that is is language is the key to that right we think that AI changes or sort of will unlock a lot of new dynamics and the way it's going to do that is because we're going to be able to control the AI right and and the story behind this kind of started with I've got a son who's almost seven now and as I started to look at

his future right and say well wait a minute AI coming up and replacing people and to over taking all these tasks and beating humans it's not a world that we think is great right fundamentally we don't think it's great for gaming we don't think it's great for people so we started to say well wait a minute you know if all this power is there then let's give it give people the ability to to shape it and what happens is power you know everything becomes better right we get through bringing these new dynamics I think we're going to see some really fascinating things if all we're going to see people interact differently with games and have and discover things about themselves and about the world that they didn't that they really didn't perceive before and that the covers a range of stuff so we think that in these human and AI interactions right once you have language as a as a sort of bond between them the agent can become much more aligned with the way humans see the

world they can start - if you think of things like Diablo right where you've got the little guy who's trailing along behind you and he's gonna tell you something so he's he's necessary to the game but he's actually a drag on play because he's not good at anything right and you'd wish for somebody who was and and you feel like you deserve somebody who's going to look after you right rather than this guy who kind of does random stuff at the back so we think that having intelligent agents that can not only fulfill a certain static role but evolve with you and sort of come to serve you better we think that's a really interesting dynamic and I think game developers with their own stories are going to come up with ways to use that that are that are fascinating they're going to lead to really deep deep experiences in the game the next angle is this new interface of voice right if you look at the way people are playing fortnight now and ever you know how much chatter is going

on this is a big deal right people are interacting using voice massively now right they're used to it because of Alexa they're getting into it because of fortnight so your players your potential players are out there using their voice all the time but there's not really been a way for game and voice to work together because Alexa sucks right that's not you know doing you know Alexa the problem with Alexa is every command the result is always a little bit disappointing right the thing about a reinforcement learning agent is the result can actually be better than you think right because the agent is gonna have its own it's like okay you want me to do this and it's gonna try to figure out how to maximize that that to is a really interesting dynamic and we were looking at I was listening to some some great esports people talk in the other room and they were talking about how important the story is right of what's happening and this is you know there's an opportunity here for this interaction to become more interesting so we see how these teammates can can in boost

engagement by following their leader right so being you know like this guy in Diablo instead of being this kind of static uninformed guy you can start to shape more of the game start to engage more with and lead players on a more interesting journey also the intelligence in these agents and the the fact that that intelligence evolves means that your your player gets to wrap the story kind of up in their own you know they get to take ownership of the story right they get to perceive it through their own eyes and kind of shape it and and we think that leads to a lot of opportunities as well as just the you know the affection right in there's there's a there's something that's very there's a reason that Tamagotchi worked right all those years ago you know people like oh this thing needs me I have to keep looking after it otherwise it'll decay or die or whatever there's something to that right we like to be needed in by by something and and these agents it may sound it sounds a bit

bizarre to speak of an agent in that way but but the fact is we form bonds with with the things that service now this is an example of one of the things we've done if you look at the little blue guy there on the top the player is saying heal me so the agent focuses figures out okay who needs help and runs around and actively targets and says so we've taken essentially this is the starcraft sc2 le environment we did we turned it into kind of a hack-and-slash maze game and we put this agent inside it trained it and and you know then we put it you know we sort of stuck language on top and and got it to understand the way we've players think about the world and all of a sudden you've got an agent that can respond in real-time to human input now if you don't give it any input it'll do its best and it'll just sort of make autonomous choices but it actually performs better and you're much more

likely to succeed in the game if you actively coach the the agent to play alongside you so will you complete the maze without coaching the agent probably not right because it's going to do its best but it may not be in sync with what you're trying to achieve but if you work together it can here's another example and you probably won't be able to hear the audio but in the next one you've got the agent who then is asked by the player to block out so all of a sudden you'll see him shift from attack - suddenly blocking out so this is and for the player all of a sudden you've got somebody who's who's acting on your behalf and who's creating really interesting interactions right who's who which is acting it's using its own intelligence right this isn't you weren't controlling every motion we think that unlocks a lot of stuff and we're humble enough to know that you know the game developers who have their story to tell are going to take this in a lot of different directions but we think the fundamental empowerment of saying look you can take an agent you can train it and then you can tell it what to do is going to lead to to some

pretty remarkable player experiences most importantly because it makes a player feel awesome right it's fun to be the leader all of us love to be to have people who will you know who will dive on the grenade for us or follow our lead or or take our take or our mentorship and and that for me is a is a fundamentally important thing to tie into it's good to be needed the other direction and you know I'll be very frank we're just beginning to explore these these various dimensions I don't know if you guys have seen the unity people they did they have their machine learning interface and they're just sort of conventional reinforcement learning agents but they did pop oh the Corgi right so giving non-player characters intelligence we think if you take that but then say look like hard-coding a reward function is something none of us should have to do right so the idea here is to take an environment like that put

the agent into it let it learn itself and then take this semantic this language wrapper over the topic and constrain the behavior so if you want Papo to go fetch tell him right if you know in i was told but in some of the conversations i had the these these couple of days in korea apparently pet games are a big deal so you've got these little pets that follow you around the things well but there's only so much you can do with point-and-click but if you can speak to a pet you know there's there's a lot that you can probably unlock as well as saying you know doing the same thing for agents and saying in your design process however you are constructing environments ultimately I think you're going to see people like us or us be creating tools that say live within unity that are like oh here's what you're trying to do can I finish that up for you right do you want to do some more that so the goal would be again for the agent to intelligently adapt to what it's seeing and to try to replicate that and and produce interesting results we think that leads to better workflows

right if if a game developer can focus on story then then you're gonna have you know you're gonna there's got to be better created or more creative output because you don't have to spend your the proverbial glucose on alignment of things and and things like that and as you saw in the those couple of Clips the agents can actually be pretty dynamic and effective this last thing is is something that you know we we sort of gave some thought to pre-show and have found that a lot of people seem to agree with us that 50 percent twitch viewers number comes from somebody who works at Google on a lot of these eSports live streaming and partnership things and and and it was a fascinating number to me I think I heard this about six months ago and they're like 50% of people who are watching these games don't even play that game alright so it's not like they're trying to get better it's just these some of these players are just

super watchable right super interesting and by the way a lot of the games have super big hurdles right if you want to start playing the game that it's like okay let me get started right first of all tutorial and then this and then keyboard and mouse and the first its immediate pain right so but if you say okay let's put an agent in between and say and start getting involved via the agent go straight from twitch to coaching an agent around the environment so say oh hey I'm playing around I'm having some some I'm engaging with the game I'm starting to enjoy it and then they can go deeper right but to start out with this as a transition layer again these are theses that that we believe in but it's starting to you know this seems to be such a big pain point for people that they're saying to get involved in a game is just is just really tough so so we think we can do that as well as taking something like Starcraft and just saying look you don't need to write rewrites our craft like Diablo has just had their Big O they moved it to mobile he's freaking out if you look at you know

game like Starcraft right it's it's very watchable it has huge viewership in various markets but again that hurdle to get involved in so massive you know we think say look take Starcraft make it playable on mobile using your voice alone right you tell the agent what to do all of a sudden you're getting involved you're you're sort of making your way through the world and engaging and again it's up to the developers to really figure out where the how the players are what the players are going to think is amazing in terms of experience but we think that that unlocks a lot of a lot of stuff so as well as human AIT modes right we're looking at FPS type games where the player plays and has a wingman next to them and says okay take that hill I've got this covered or I need cover fire I'm gonna go do this right - you know and that may create more watchable content it may again just kind of enrich the whole ecosystem and flow of things in this market so you know whereas the

old world was you know Bostaph eyeing you know BOTS and gaming was was always a bad thing so so our view is kind of the old world is all Val Kilmer you know you can be my wingman anytime and we are more aligned with Tom Cruise say you know the human should be able to say you can be mine so that's it thank you very much [Applause] yeah ultimately if you know it all comes down to the training process and how you engage with the bots so so if you're playing with your bot and you say do this and then do that and then do that and don't do this then there's no reason

that that can't become you know it again that a training process so yeah you can ultimately behaviorally shape these these algorithms because they're you know the way they learn from from past behavior that sort of shapes the choices that they make in the future and you saw that actually with open AI if if you you know with those dota 2 BOTS they were you know in the first couple of games they had then they went away and they're like oh no it's doing badly against Roshan so we got a so they they actually went into the game and trained a bunch of stuff overnight like Oh get the player in gear you know let's fight Roshan a bunch of times and then the next day it was completely different behavior so so it's very much and and they were doing it again they're super talented technical people over at open air it's a great team and so they're doing it tactically we believe that that should be accessible to to everyone so that's kind of our view is this is actually a super interesting toolkit

right that just sort of lifts the gaming level very exactly and that dynamic is something that we think people are really going to get into because then it's it's almost minecraft e'en right because then you it's a whole new dynamic to explore right even if your world is not changing that much the way you interact with your bought and and what you achieved with your agent in that world becomes really interesting right and so even in a world like Minecraft right where there isn't a specific stated reward phone you can have very specific goals for your agent and it can become great at building you houses right so you could end up you know put it to work kind of thing so we think it's an interesting dynamic it's it's funny in the early

days of conceptualizing this we were looking at very much saying well you know you could actually put these agents you could train a bot and then put those weights on the blockchain right and and just sort of have this whole and just say hey if somebody wants to use this bot which is you know here's my here's the score it's achieving in this game if you want to play with that game so one could write because they're digital assets at the end of the day so and I think it opens up a lot of stuff like monetization in games so if you want a you know the clout there could be a cloud 9 bot right that you that would have particular talents and skills so I think for game developers again it's always this battle to take free-to-play and turn it into add layers that are relevant to the player so I think that space is really interesting because then you can upsell upsell upsell and you know and and give the player a really interesting experience like wait a minute it's not doing what I thought oh yeah well it's a cloud 9 ball it's not going to write it's got its own

[Music]