Ethics in Artificial Intelligence

[Music] thank you for coming out to this panel this is something that a lot of us are really excited about getting to discuss especially here at gdc um so a few months back when i had pitched the idea of doing a panel on ethics and ai here at gdc i actually got a lot of feedback from game devs saying ethics and ai but we're at a games conference if this was a social media conference or a conference that was maybe focused on machine learning then sure but why would we talk about ethics at a games conference and hearing this feedback from game devs was what made me realize just how important having a panel like this should be especially as we're starting to see the spectrum of game ai grow into so many different areas over the past few years when social media platforms were being built ethical and privacy concerns were not their focus

and look at where we are today when machine learning was brought into the education system ethical considerations were not made and it caused major issues because of biases that had been programmed into their systems with big data comes big responsibility when our panelists actually got together the other day an interesting statement was made and it was that discussing ethics had always seemed very taboo and we wondered why is it because companies and individuals are scared to admit that they aren't necessarily the experts in a given field that they actually need help that maybe a system game or tool that was built by a company um was coded with biases that could be offensive or even dangerous to some people but where one would rather ship a product than actually have their product be scrutinized and redesigned for improvements through ai we are able to do amazing

things but without proper considerations around biases and ethical implications things can easily go south so today i'm joined by some amazing developers from different areas of expertise who work with and use ai across a number of very different areas and have all been advocates in their respective fields around ethics and games and ai we're going to cover quite a wide range of topics today from biases and game ai to data and privacy concerns and areas of ai that transcend from traditional traditional video games like xr and digital assistants and then we'd like to spend some time where we open it up to the audience because we really want to see what do you all think are areas that we should all be addressing when it comes to ethics and ai so hello my name is alicia liedecker i'm one of the advisors here at the ai summit and during my day job i work on xr

as director of developer experience at magic leap prior to that i was lead ai on many of the assassin's creed titles so we're going to start off by having our panelists if you'd like to start emily talk a little bit about yourselves and some of the work that you do sure i'm emily short i'm chief product officer at spirit ai and what we do at spirit is middleware for games um so that includes a product called character engine which does dialogue for npcs and how they can respond to natural language input or other kinds of input from players and the second product which probably has the greater application in this area is ally which is a community moderation tool that looks at toxic behavior within communities and helps give community managers an opportunity to see in a triage dashboard who is causing the most trouble in this space and what are the biggest concerns that we should be

looking at moderating so we're not just waiting for things to be reported by players but we're actually able to surface issues in the community and obviously that both of those products and especially ally raise a lot of questions and things that we need to think about about how do we train to look for these things what data are we using how are we tagging it how do we decide what's offensive what's what's racist what's appropriate and then how do we make use of that information when we have it how do we protect the privacy of the clients and of the players that are making use of the system and then when we're applying characters that can respond to risk input in interactions how do we make sure that players who are interacting with those characters understand that they're interacting with an ai and they form an appropriate rather than an inappropriate kind of connection with that character thank you celia hey so my name is celia hudent and i am the least knowledgeable person about ai on this panel my background is in psychology actually

i have a phd in psychology i was i'm specialized in child development and development of cognitive psychology and i've been working in the game industry in the past 10 years i started at ubisoft in france i'm french and then moved to ubisoft montreal i worked at the playtest lab there and also worked with a rainbow six franchise i moved to lucasarts uh working on star wars games like 1313 that never saw the light of day sadly and then moved to epic games in 2013 to be director of user experience at epic because now i'm specialized in game ux this is how my background meets game development and so i worked on all the different products that i picked so undrill engine and of course fortnite i left epic late 2017 and now i'm a consultant in freelance and also wrote a book called the gamer's brain and so i'm very interested into understanding how we can use psychology

in developing products for good or evil so all these questions are really really interesting to me awesome thank you timoney hi i'm timoney west i'm the director of augmented and virtual reality research at unity labs my background was originally in a product design in social media and others and i've spent my entire career trying to figure out how to get data from people largely personal information and then give it back to them in a way that makes sense and is useful for them when it comes to spatial computing that has been an even bigger conversation because we are literally creating tools that let you record everything about your house everything about the way you move or the way you're moving your device and then try to put that boat back into the engine and then into your game or to your experience in a way where it makes sense uh and actually adds additional value so i fundamentally believe that if we take

in information from these devices and from our users we have an ethical consideration to give back more than we've got and that is especially when it comes to having you know devices that have 10 different cameras watching your every move and listening to your every breath something we really need to take seriously so i'm glad to be here talking about it so i'm luke dicken i'm director for central and strategic analytics you might have heard me talk here yesterday i will try not to shout at you so much today um so i'm zynga uh and we are kind of well known as a data driven games company we've kind of come to prominence in maybe 2009 kind of era uh you know there are quotes flying around that oh we're not a games company we're an analytics company masquerading as a game studio um so we've been kind of collecting a lot of data over the years um and i think that we we do a pretty good job with it so i'm really excited to kind of come here and talk to you about the way that we do that awesome thank

you um so sorry luke we're actually going to start off by talking about npcs i know yesterday and i'm going to start shouting again about this um yeah that's fine let's show it you know um so my first question to the panelists is um what are some examples that you've seen in video games uh where you've seen biases that might have been programmed around characters and systems ai anyone want to start you try uh yeah i remember um when i was uh playing watch dogs um they there was a lot the ai was always uh you would see like a woman always getting violented and in trouble and as a player so you're incarnating the um a male character and always coming here to help out the women and the women always need someone to get out of that violence and so the the women are in that game are massively

portrayed as getting beat up and needing some help um i always like to diss navi from ocarina of time how many people play that it's not even fair she's just clippy i get it but um it was an early attempt to have you know the user sort of guided and and given context as they go through experience but she was extraordinarily annoying she just wanted you to stay on task and the great thing about ocarina was the first time you could really run around in 3d and hyrule right so that's what you're gonna do and she was just there being annoying telling you to go to the next temple and you're like no navi i just want to ride my horse leave me alone anybody else i mean one that jumps out to me i think is um you know if you look at sort of systems like in rim world um remote has a whole kind of deep simulation system but a lot of it kind of gets represented as

very um it's an example of a way that's very easy to get mpcs doing things that are very um kind of uh stereotypical because it's leaning into things like oh this person has a misogynist trait this person has a misandris trait and it's kind of uh it raises some interesting questions about like does that actually need to be part of the ai system and what we're actually kind of representing or is there something kind of i guess it comes back to the question of taboo right like if it's taboo for us to talk about it here is it taboo for us to like express it exactly so it was interesting for a real world because that was one programmer one ai programmer who decided to program what they thought the rules were for how men and women should behave in the game and how sexuality um should behave in their in that game um what ended up happening is the community ended up going on reddit and basically demanding the programmer to remove some of those biases and the only reason that that person

made changes was because the community demanded for it so what do we think that we could actually do to help put in place processes or how can developers be more mindful about coding systems without biases that have been put in place well i think some of some of the approaches that we see brought into narrative context around having sensitivity feedback having people comment who belong to particular groups that might have a little bit of more understanding of kind of what does this portrayal mean for us what does this kind of representation likely to convey about us have a look at the work and give some feedback and intentionally seeking out sensitivity players has been really useful for story games and i think it's it's entirely possible to bring that kind of approach also into looking at systemic representations and i would add to that it's not necessarily the case that systemically representing negative attitudes on the parts of of characters like having a racist

character is not necessarily inherently evil but i think it's it's we need to be careful about how we represent and unpack that um and so having people be able to intentionally seek that kind of feedback about their system is quite important actually that is the curious thing about being the creator of a system if you're trying to mimic a real world system you may or may not agree with because just inherently in creating and defining the system it sort of feels like you're giving it the thumbs up even if you're not you know like like because i am replicating this i feel like the sort of implicit uh acknowledgement that the system is here even if i completely disagree with it it's just an interesting tension there's no real solution there i know when we were on our call the other day we brought up some interesting discussions around just what the role of a programmer for systems and ai means and actually about the education of programmers i think a few of us brought up how who here actually who's a programmer has

done some type of ethical class when you were in university a lot of us right or quite a few of us i remember personally for myself that it was kind of that class that you didn't take seriously and it was actually very focused on business rather than actual ethical um areas that we should consider as programmers so what do we think that we can do on the education side for programmers and designers maybe don't even call it ethics it doesn't really matter what the ethics are that's like a social thing you know it changes from country to country or even regional region but just being able to create or have a class that teaches you the underlying socialization systems so that you can confirm your own bias and know that it's a bias right recognize which part is is the socialization aspect and how it differs across different cultures and and then it takes out the is this good or bad and it's just like this is a tendency that you have acknowledge it and then see if you want to build it into the system you're designing or not

yeah maybe maybe we should call call it unconscious bias more than ethics because ethics is a set of values that we decide and who who's deciding how we decide it's complicated but if we talk about unconscious bias and we explain that as humans we are fallible and we all have biases and it's okay we need to admit it and accept it um but then we need to work around it because we can't even if we know our biases we still can't get away from them so we need to design the environment to go around them the um the example of i'm gonna try to do that fast but the example if if you want to hire more women you it's we are discriminated against women even even women against other women and so if you take the example of the orchestra i think it was in boston they wanted to hire more women because they figured that how come we don't have that many uh musicians in our uh in our in our group um that are women

and so to avoid um being discriminated against women they started to do blinding um uh auditions so they were a curtain and they couldn't see and also later on they added um carpets because they could still hear the high heels of the women and but after they um added the carpet and the curtain then they hired i think it was plus 30 to 50 more percent 50 more percent women uh in the orchestra so we are biased it's happening we we have to acknowledge it and we have to understand how it works and how we perpetuating them and so in ai or in any system if we don't take that into account we are going to um perpetuate all these discriminations i'm just going to give one more example from ai um but i love that example because it's pretty clear um if in turkish there is no gender when you talk so yes she is a doctor he is a doctor it's

gonna be the same phrase you don't have he or she in when you talk in turkish so if you google translate uh he is a doctor or she is a doctor you will have the same phrase that i'm not gonna say because i don't speak turkish at all but you have the same phrase for both of these different phrases in english now if you take that phrase from turkish and you put it in google translate and translate it back into english then you have he is a doctor now if you take the phrase in turkish that says he or she is a nurse and you put that in a google translate you get she is a nurse and so where there's no gender in one language uh through uh ai algorithms uh we it's gonna perpetuate some discriminations so this is where we're at we need to take that into account and if you want more equity we have to find a workaround so actually i agree with what you were just saying but i want to make the case

for ethics is about more than just the biases biases are very important but i think as a as a culture we need more training and ethical thinking and how it works in general um and maybe that's everybody needs to watch the good place a lot more but i feel like one of the things that's happened is that because we have for many reasons and you know historically a lot of this kind of thinking about how do we figure out what is good and bad has either been in the realm of fairly academic philosophy or it has been expressed within a religious context and moving away from those contexts especially as more and more people like don't necessarily share the same backgrounds we need to have ways of talking about these problems like how do we decide what is good and bad like that's a it's humanities you know we got to still have them in schools so i think it's it's broader than just identifying a set of issues it's it's about like building that whole framework of philosophy yeah i mean that

yeah cecilia was talking about like all we're doing is measuring the vector differences between words that are close together and that is how you get she is a nurse and he is a doctor but that's how you get he is a doctor we're just you know something's running through a bazillion words and trying to figure out which words are close enough to each other that you can predict which word comes next that's how it works right so if on top of that we want to say but don't say he or she that's where we need this ethical framework of like okay go back through the data set again and now do basically kind of a ethical fix but the interesting thing of the data so yeah where we get that list what is the list who defines the list where is that who validates it how do we get it part of our process but we already have it right i mean well is it in your pocket in as much as it's everything yeah so i mean it's really easy for us to kind of step away from like fiddling about with with demographic

words in particular yeah but like everybody in this room knows what an imbalanced data set is and if you have an imbalanced data set you know that you need to do something about it so like if you can take that that kind of tool set across everything and then start applying that to places where you're not necessarily applying it right now is that what a fix looks like just how do you account for cultural differences there are countries where women cannot be doctors i mean i feel like if we had an actual plan here we'd be off doing it yeah fair enough that's true um so moving on from npcai there's a lot of different areas and different types of games that use ai from social games to online games celia i know you had some areas you wanted to discuss around like how we could be using ai data better to improve those types of games i don't know if you want to talk to that uh sure um no i think the the only thing that is is just to go back to my previous point

we our society as is not we don't have equity um and we i mean a lot of people want equity i think if we can agree to that then we need to figure out what is it that we are uh showing you know what are the biases that are perpetrating in our games that it's not just our games not to be fair uh it's perpetrating in everything social media in in movies and books or whatever but we are also participating into that because we are part of the the culture and so a lot of people play games now so what what do we want to reinforce there are we going to reinforce discrimination towards women and towards people of color so what what is it we want to actually how do we want to participate because we are going to reinforce some discrimination uh or we can reinforce a better vision of of the world so since we are participating even if we don't mean to

what do we want to do how do we want to participate so coming back to sorry online online and social games and i think like it's mostly a few of you here especially luke at zynga as well and then from fortnite um they're very different types of games that aren't just about the biases um for characters but how you're using aai data to influence those games in different ways what do you guys think about that or what are some initiatives that you've done on your sides so one of the things that we do is we definitely don't feed demographic information into machine learning algorithms partly because that would be wrong uh it's partly because we don't trust our demographic information uh it turns out that a lot of millennials lie to everybody they can online um so as a result of that like it's we know that it's not necessarily the most reliable data so we don't actually use it as a basis for anything

but that's an example of like how you can be a little bit more kind of buttoned up about this stuff like you know machine learning in general takes the the view that hey throw all the data into the machine and see what comes out it's magic um but you can be a bit more intentional about it and you can kind of um be a bit more structured and kind of you know we gather a lot of kind of in-game behavior data um you know as you're playing through the game we fire off counters for probably too many things frankly given what's in our database um but at the same time like we can use that to actually back into a model of your player journey and then from there kind of understand that like i was talking about yesterday kind of what you like and dislike about the game and what you're what's resonating with you that feels like um there's like two different ways you can go with that right there's the the kind of black cat okay let's just milk you for as much money as we can and then throw you to the curb um there's also the way that you can kind of approach it which is like let's

actually work with you to provide a good experience and i think i think some of it is intentional right like if i come at it saying hey i'm just going you know i'm an evil corporation and uh my comms team is gonna [ __ ] hate me saying this stuff um but if i'm you know if i come at it with with that hat on then that's probably not super ethical whereas you can you can just change the framing and use the same kind of approach but do it in a much kind of better way and this actually came up this morning i was talking to ingrediesto who's our head of monetization at unity and we have the ability to uh map and track player behavior and change the game on the fly as a result and the reality is we could be manipulating the heck out of our users constantly so the ethics of how long and how addictive

area but then we gamepl need to have an ethical base on which to make these decisions because what is the cutoff is it a thousand hours of gameplay like in a row if you spend two grand on one mobile game do we cut you off do we should we have the ability to cut you off it should be a percentage of income maybe you're rich in two thousand dollars is nothing i don't know but obviously it's not good for anyone to waste their life or their money on these on these trivial things so anyway it's something that we're thinking about quite a bit who are some of the people that you're working with to help synthesize that data because like celia was saying coming from a psychology background it would say i would think that working with experts on that field is who we should be really like synthesizing and going through the data with rather than just the product owners or the programmers on that side um so have you done any like consulting with with game companies um that are trying to figure out what to do with all of that data and then same question for you

luke how are you synthesizing all the data that you're collecting from zynga i mean one of the things that we do kind of poorly is the psychology side of things we are very stats oriented um and i think that's partly because we actually don't have a user research group we have a like a kind of marketing and consumer insights kind of focus test group and then we have the analytics team and if you have an analytics team they like numbers and they don't necessarily like people so they're going to focus more on the numbers but it's definitely something that we could probably do more on um sorry um yeah so i so i do a lot of the consulting now and usually people ask me to help them uh offer a better experience to their players so that there is that empathy that that that's that's willing for empathy um but then you never you never know how things are going to be used because even if i if i explain um things like um behavioral psychology

and how we react to uh rewards especially uh in a variable schedule of reinforcement just like in loot boxes and explain you know this is uh this is exciting in a lot of cases that's why we love to play dice or if you have critical hit it's exciting so most most of the time you know in games these things are exciting now if you apply that to monetization form of loot boxes that you actually pay money on it and it becomes a little bit of a problem because they are using something that is engaging that we know is engaging to make money now is it a bad thing to try to make money when we see all the studios that are closing and it's actually very difficult to survive when you make free-to-play games i don't know um so it's it's it's hard because now a lot of people talk about how psychology is used for for evil we can use it you know to for dark patterns but we can also use it to nudge people and to make people make better choices for

their health or financial decisions so this is this is where it's not clear um there's there's more people outside of the gaming industry than now asking me for consulting uh to for questions around inclusion and diversity um more than the game industry so far so um so we keep kind of bringing up data and ethics and ai is kind of nothing unless we are talking about ethics and data collection and privacy um emily i know when we had the call both you and luke had talked about different standards and regulations that you had used to put in place in your products and in your games can you talk a little bit about that sure so in in the eu because we're working across sort of multiple continents in the eu we have gdpr which for people who are not familiar with this is a regulation about how companies are

allowed to store and process and apply personal information that they have from their players and users and it means that you have to be very specific up front if you're collecting data you have to let people opt out of having their data collected or used you have to let them know if somebody places a request they can actually as a matter of transparency ask what data you have on them and then as a corporation you have to keep track of like what kind of status do i have relative to this data am i just storing it or am i a processor of the data um so it's it's a huge sort of piece of bureaucratic overhead which made a lot of people very frustrated and very uh it took up many many man hours um and woman hours person hours to um to make this thing work um but i think that's the only way that we get around uh the sort of very strong incentives that exist for corporations

to just consume as much data as possible and then do whatever the heck they want with it and one of the other really important pieces about gdpr is that it has a really serious fine attached so it's the larger of uh 20 million euros or four percent of your global revenue which is enough to make a difference to a google or facebook to actually make them pay attention to this kind of thing and i think that's something that we need to think about we can identify a lot of issues and problems and things that we hope that you know people who are in a position of power within individual institutions are going to take seriously we hope that they're going to apply these ethical considerations and we can talk about that and we can try to create you know cultural standards and norms about how they should be behaving but i think the business incentives in some cases and especially around big data collection and big data use are so strong that we really do need the effect of government regulation in order to keep that within parameters that are

acceptable and for the audience was gdpr that was just what i was expecting does it stand for oh uh general data i actually don't remember the gdpr you just have been chanting gdpr for so long i don't actually remember what the ackerman acronym stands for general data protection regulations right and are you using the same at zynga i mean you have to yeah it's an eu mandate so if you don't want the fines you have to be compliant with this thing um so yeah i mean we had a pretty big initiative internally partly because of the very significant fines but also it is actually a i don't want to say it's the gold standard for for what privacy and kind of personal data regulation should look like but it it feels like a really good first swing at kind of government legislation around this stuff i agree yeah uh unity also supports supports grdp um actually when you were talking it made me realize something i hadn't before which is we talked and when i was just talking with ingrid about how we should at least think about

whether or not we have a responsibility to cut off players who are spending too much of their time or money uh somehow ads gets like a free pass on this i had a friend who told me she stopped using social media and she stopped buying as much stuff because every third instagram photo is actually an ad for something you probably were considering buying just bought or probably will want to buy and we don't really talk about the ethics of that like how much money how much more stuff can you buy in a year on the internet before the great adds god in the sky says no more that's enough hmm i mean i read run an ad blocker so much of the time that i managed to yeah i mean can we try this but yeah i mean it is it is insidious because facebook so so this is this is like my personal story about how much i hate this right like i changed my facebook status to engaged and that same day i started to see facebook ads for weight loss like wow and it was it was completely obvious like what was going on in the creepy little mind of the algorithm back

there so um but anyway um i i don't know i mean i think that the the ads question is even harder than the question of where do we cut off people in free to play because in free to play you at least have the record of like oh this customer spent this much money and it's a lot maybe this isn't a good idea um and so you could have kind of internal standards of you know how much is how big is the whale allowed to be basically um so you can think about that kind of thing but when you get into the ad space it's very hard to know what effect the ads might be having on the person i didn't sign up for any of these weight loss programs as it happened so unsuccessful there so you know it's kind of tricky to know but it's a valid question yeah so but the thing is what is it that we're value what is it are we measuring what is it that makes your investors happy it's like how much money you're making right what are the profits you're making so if this is what what we're measuring and we're not measuring if people are happy using your product or if you give them a better life that's the whole question about

um what uh tristan harris is going around with the economy of attention and and you know what what is it we're measuring what is it we want to offer to our people um because we don't like facebook they they measure how many uh how much you engage where with the advertisement they don't measure if you actually have a better quality of relationship with your friends so what is it we measure because this is what is going to drive the algorithm or whatever to um optimize it uh so we know that for example like the reason why clickbait is working that well is that outrage is making us react and this is something that makes us like spend a lot of time on twittering and i got a little excited about something so if we measure if the thing that we want to optimize is people clicking on the link then all of a sudden their algorithm of course is going to favor all the big claims and the stuff that gonna actually make us fight each other and so it has a terrible impact on our relationships on society so what is it we want to measure what is it want to optimize is it always making profits and making you

know the best the most clicks on those views yeah actually exactly like it doesn't matter what ethical standards we have if some pm's kpi for this quarter is to increase activations 25 and you're like well you know doesn't quite match up that doesn't matter they don't get their bonus or they they lose the respect of their peers if you want to go to a little bit more high level they're seen as a bad worker like we need to have what people do for a living and what they're rewarded for financially and and you know socially match up to the ethics or they're just not going to implement it which is what we're seeing today yeah i mean i seem to be up here making big statements but i think the part of what we see is that it is hard to debug the system the systematic incentives without sort of taking here are the results this is the kind of results that we're seeing is ai is allowing us to take what we were already doing much further than before and we're

seeing bad results from that but what that is telling us is that we need to debug capitalism like it's not just it's not just the machine learning that needs to be fixed right yeah i mean it's a system we can manipulate it just yeah turn in a different direction everyone gets to make a lot of money you know yeah yeah well but anyway that's it's gonna go way off data um timoni you mentioned this in your introduction but to do xr experiences it's even more important today to get as much data about the real world around you you mentioned that what you want to know is see what the user sees hear what the user hears but there are a lot of discussions that are currently happening around what does that mean and what do we expose to developers and to users so what are your thoughts around that collect as much data as possible so there's too much and then no one can do anything with it honestly i tend to be pretty lazy fair about data collection i briefly work for the state department and it was very

cough gasket and i walked away thinking there's this is not going to turn into a police state they're not that organized but and if we get to a police state anyway that's always a question right it's always like oh well people have your data and you don't know what they're going to do with it and if in the future they just you're you're suddenly on the bad list because you did something in the past that was recorded uh you're [ __ ] okay you are but if we get to the point where we do suddenly turn america into a police state we have a lot of other problems as well and i don't know if data collection is top of the list necessarily that being said i do know people in the audience who have been on lists and therefore going to the airport sucks for them or because of their name because of how they look so i i do want to be sensitive to that um there are a lot of inroads right now into new ways of anonymizing personalized data the biggest problem right now is it tends not to be performant because you can only perform calculations on encrypted data which doesn't work well on mobile devices which is basically what ar hmds are for example but i think as computers get

faster and smaller we'll continue to see inroads and being able to basically obfuscate parts of the data set so that you can have a personalized experience but it doesn't tie back to you personally but then there's a whole other question of you probably do want a digital identity that's tied to you personally as you go from space to space or room to room and other people's homes and you want to see what they have or they want to give you permission to see the digital goods in their in their house but that's that's kind of for a few point do you end up with the same thing if you walk in to go shopping for a wedding dress and you walk out and you see a digital ad about weight loss that says this way so yeah and digital ad blockers um kind of riffing on the xr side uh we're seeing a lot of talk i think every platform is talking about doing um digital characters in the real world um also with spirit ai like your whole focus is about how do you create these interactions and build like

ai systems to interact with digital characters we're also seeing where how many people in here have alexa or google home at home quite a few and essentially that is also a digital character just without the visual aspect of it there's a lot of talk about kind of the humanizing side of things and again what do we do with data and how do we design these digital characters to not have these biases similar to what you were talking about with watchdogs so what do we all think about that i think what what we're seeing at especially with some of the clients that we're talking to are use cases like there is the use case of it's just a digital character or an assistant in your home for adults but there are also especially a lot of use cases that are even more sensitive than that so cases where um people want to create an educational character or a toy you know it's an educational game series that where this character kind of lives with your child for a period of years and it has dlc

that like teaches your kids stuff and remembers things about what your kid likes and of course that like runs smack into like all of the sort of child data protection issues but also a lot of sort of subtler things about like what does it even mean if you've got a sort of an educational program that is shaping itself in response to the child like what are your responsibilities about what you teach them and don't teach them and it it gets very diamond age and it's kind of strange um but that's an interesting area and then another big sort of point of use is in cases of people who've experienced trauma or who are suffering with autism and have or whose parents kind of want to help them get through uh learning to interact socially i probably shouldn't say suffering with autism that's not a good way of putting it but so in cases where the finding is that some people in some situations actually find it easier to talk to a digital character than to another human but then

you're again creating a context where there's the potential for the ai to do a lot of good right there's the potential for the ai to make somebody comfortable talking about something that they don't want to talk to a human being about because they're ashamed or because they're uncomfortable for whatever reason so that is that's high value but it's also it's a high-risk high reward kind of situation because it's such a vulnerable space you could also do considerable harm so i think there again a lot of the kind of impetus is on making sure that the people who are working on these materials are coming to it with the background of kind of educational experience and psychological experience that it is not just a product being formed by an entrepreneur who like came up with this because it was an easy story to tell to vcs like it needs to come from some place of much deeper understanding than that this is where i always get into we need to tell the computer no and the computer needs to be able to hear that and react to it and record it

for future context right like now today if siri starts playing music or alexa i'm like no no not that song no you're all wrong no not amazon music whatever but this needs to be the case for all especially in xr when things are going to be coming up to you or trying to attach themselves to you or pin themselves to places in your home you have to be like no that's wrong no that's wrong no that's wrong we're already starting to be able to do this with ads in a really systemic way because the ads follow you all around the internet so oddly it's like kind of a good use case but yeah that we need to be able to tell the computer now which means that we have to have that in every piece of software the listener that listens to the know and records the now um there's also the idea of as we start seeing that there's more and more happening all around us do we have a responsibility to start explaining to people what is an ai and what is not an ai i was at a conference once and we were talking about alexa and a few people that were there were telling me that

their little kids started yelling at their parents because the parents had said alexa play song x and the children yelled back at the parents saying you didn't say thank you to alexa and basically the people had said that children are growing up where they feel that alexa is part of the family um that it's a real person and as we're starting to see these digital characters come into the real world around us with xr to these um smart home objects and really ai everywhere what do we think we need to start thinking about like how do we explain to users this is an ai versus this is not real i think it's it's actually i didn't look this particular point up before this panel so i may be misremembering but i believe it's the case that the state of california law is that if you have a chat bot or something similar to it that if the user asks if it is ai it has to respond correctly like it has to admit that it is ai you cannot have a character that is going to say no i'm

just a customer service agent um you know i'm tim and i'm here to help you um and i that's clearly kind of there there's a particular sort of customer service context in which that could be important is like you know if as a human user i really need to make sure that i'm getting accurate information back from this company that i've got somebody that i can hold to it or whatever that so there's kind of a specific local case but i think more broadly it is important for people to understand kind of what they're interacting with and they the thing that i keep thinking about in this context is that many years ago so i made a game very early in my career that was a conversation game where you're talking to this character and um it was a very like low fidelity it was all text you were interacting with it by typing so nobody was going to be confused that it was like physically a person but at one point i got this email from this guy who said like thank you for making this game galatia like i keep it on my phone she's become my best friend and i talk to her every

day and i felt like thanks but i'm actually really quite concerned right now um and so like that that seems to me to suggest like some other areas where we need to be careful right like you know it's is it okay to be using ai characters to resolve loneliness is that maybe that's a good thing maybe maybe it's a little disturbing and sorry um so i agree with all that it's just because my child development background um i just want to mention one thing children have a tendency to animate things that are inanimate uh so i'm not i'm not so sure about that part do they really not understand that uh alexa is not a real person because children will have a tendency to say oh you kiss me good night you have to kiss my doll good night as well so i just want to make sure that we are not saying that kids are not able to discriminate between the two i wonder if we can come up with a new way of

human i mean we do this all the time right to your point or you know you if i make two circles on a smiley face everyone's like a face we're really good at pattern recognition i wonder if over time we'll just start to create this new space for these digital creatures there there are several excellent companies that make ais that help people through depression and suicide and have great search are great great results and i would not want that to stop that's awesome right and clearly they've defined this relationship to this digital being that works for them knowing that it's not human because it's very clear when you go there and i love that i love that use of the technology what i don't like is when people kind of humanize the sort of malicious intent like oh it's all going to be skynet and they're going to destroy us and we deserve it by god you know now that's just you projecting so yeah i maybe in in teaching people about what ai is we can start to come up with the language that allows for that middle ground where this is made by humans and it's designed

to be interacted with by humans but it is something else yeah i mean i think if you present it as like this is an extension of the people or people who created it and they do you know want you to recover from whatever you're you're dealing with or whatever there is a personal connection being made through the ai but it's not yeah yeah yeah so that kind of transitions into our last question i want to ask everyone before we open it up to q a um we're all passionate about ai and the ethics around it because i think we all inherently believe that ai can do a lot of good in the world around us um so what are some recommendations you can give to the audience ideas um areas you wish for this whole community to start driving ai towards what's to start oh i can start yeah look looking to look into psychology and try to understand how we influence people we all influence other people and we are influenced by our environment so we need we need to understand that

better understand behavioral psychology better not just for to make more money and to hook people up but we need to we need to understand how we can use it to favor equity and to make people like feel better because also it's at some point it's going to be a good business drive to have a trust relationship with your customers and that you treat them with respect and make them happy in the end and i'll just like make them pay pay pay because if this is the only metrics that you have then of course you're going to favor that so look into look into these things around to understand it and again don't this is not about oh you're a bad person we all we are all uh biased and we have to be in peace with that so we can actually move move on and um and solve the problems that we want to solve so i i would add on to that um because it's really cool to be like okay well now we are in the psychology portion and we're gonna now think about that but bring that lens

to everything like literally every decision you make make it intently around what celia's talking about because it's really easy to like sideline it and go cool we're gonna make a whole bunch of systems we're gonna throw a whole bunch of stuff out and here's the kind of like ethics bit bring it to everything the ethics module um i think another area and this is this is a little bit of a tangent from from some of the things that others have suggested but but i think the fact that machine learning tends to take our biases and exaggerate and expose them and that like causes embarrassment sometimes and that's a reason why we need to be careful about the balance of our data and so on but it is also a way in which the process of working in an ai gets us to think about the systems that we're part of in general um what is embedded in the world around us and in the data around us all the

time that we ignore out of being used to seeing it that way or sort of a general level of comfort or having incentives not to notice it um so first of all sort of a practice of interrogating the things that we build with ai and saying like all right well what is this expressing back to me and not only is it broken but also what is that telling me about humanity and the systems around me and should that perhaps be addressed in some way um and then the there that there's tremendous potential in our ability and this this does get back more into the kind of games and simulation space again um to build models of how we think pieces of the world maybe should be and explore whether we're satisfied with that you know potential way of doing things so that we can then move towards more just remediation in the real world i just have three things to write down to google later the first is homomorphic encryption which i sort of vaguely referenced as a way of anonymizing data

sets the second is differential privacy which is another technique used to do basically the same thing and then finally and this kind of builds on what everyone is talking about i highly recommend shane parish's farnam street which is an online rationalist community that has a wide directory of different systems of thought and mental models and collection of biases so if you want to delve into how you can start thinking more clearly and accurately there's a lot of online communities but that one is a great place to start my final one would be consider accessibility and how do you create systems that can ensure equal opportunity and equal accessibility of people getting to try your systems whether this is from a video game and making a video game where the system is not just about killing and shooting but how do you make systems that allow the users to do many different types of interactions to having digital characters in the real world not everyone has access to the hardware

not everyone is going to have access to systems so thinking about how can we make and build this community to be accessible to everyone so we've got nine minutes left we'd love to do some q a please ask any question this is a list of references that our panelists had put together of books that we recommend reading a lot of articles that dive into things from machine learning to standards and regulations so please take some pictures but um we would love to hear from you over here i can't see does that kneel yeah um some of this echoes earlier gdc's or perhaps it was cgc cgdc back then um compuserve and genie had the same issues they called it credit card meltdown that is it uh ethical to let someone for example a sailor on a nuclear submarine who spent six months underwater comes home and melted his credit card with online charges playing games on genie and compuserve um and my suggestion there is that you'll know

we've done this if we're not back here in 10 or 20 years doing the very same set of questions um it's and and perhaps there is even more history for us to go mine along this stuff you've all talked about the idea of labeling data in terms of the happiness and contentment that it provides for people so can you say a little bit more about any ideas that you have as to how we might go about that can you can you repeat that go about [Music] sure so so you've talked about the fact that if we label our data sets instead of with the amount of attention grabbed or the number of dollars we we get from a person instead we label our data with the satisfaction and happiness that it may have created for the user of an app or a game or a web page or an ad or whatever

could you say because it's easy to be glib about that and i feel the panel is maybe an opportunity to do a little out loud thinking about how we might label our data for happiness so the question was in short how do we label our data for happiness sorry i'm supposed to be repeating questions yeah i mean that's it seems like probably more of a reinforcement learning kind of thing perhaps where it's not so much that you start out with labeling i know that these things are going to lead to happiness but i'm detecting something but then that leaves the heuristic completely undefined so i mean i don't know is the short answer but i think it probably our best attempts to work that out i think i'm i am definitely thinking out loud so thanks for permission to do that um i think it's going to be very much on kind of a case-by-case and product-by-product basis like

what kinds of things in the context of this particular situation um would equate to happiness um and their i mean there are all sorts of things that you can do with sentiment analysis and stuff like this like what like how much expression are you getting back from the user right so this starts to get into territory where where actually have done some prior thought about it because one of the things that we're interested in with character interactions is being able how do we tell from inputs that we might have from language from facial expressions from gestures that the user is in a particular mood but something that you had entirely trained to um you know be reinforced if the user is smiling and like let's all smile at our computer all the time so it's super creepy so really that's not really the answer but but i think there are some things where basically if we're in what we really need in order to train towards that is something where the user or interactors

experience is pretty expressive because otherwise we're training against inputs where all we have is like oh you you completely watch this entire youtube video end to end and that makes me think that you would like to watch these other creepy like even more politically reactionary uh videos from end to end and we've seen where that leads so i don't know but i think it's i think there are sort of things to to delve into um especially like starting out in the spaces where we have a lot of information about how the user is reacting like xr so i would probably say and i'm thinking out loud as well uh but you know adding on to what emily's saying there like the youtube example is a single kind of axis of thing like i think that expressive things like happiness and sentiment and that kind of thing are going to be like multi-dimensional so if you define it in terms of so as i

was kind of sat here thinking about it um you know retention for us might be a good one as a proxy fair for happiness except it's not really because if you take that to the ultimate extreme then the ai system optimizes for sending somebody to your house and saying [ __ ] play the game um so you know i think that if you actually track kind of multiple axes then you end up with a more intricate function that allows you to capture something but that's not super actionable i mean there's also immediate pleasure versus long-term happiness which i mean as humans we're not even particularly good at working that out so that's why we need nudges yeah psychology but also look at what tristan harris is doing this is exactly these questions at least exploring the ethnicities um because like for example if you're tinder uh what are you gonna measure i measure the time that people are spending on dinner but ideally you want people to match with other people and and have fun with them and so therefore not being on tinder so what is it we're

measuring what is it we're trying to accomplish and how we can uh make sure that we we can have a business and still provide what we want to offer to our to our clients that's ux you want to actually offer the experience you want to offer and not just have like a business orientation uh we're going to go to the next question we have i think room for two maybe three more hello uh sorry if i started i'm a little nervous oh no being really realistic it will be probably a long time before i we have like like global regulation in ai ethics uh so how can we go about justifying uh like for producers and managers that we have to care about these things uh like for example in the facebook ad about weight loss uh how could we justify that we shouldn't show that ad even though it's if we show we know that it can perform

well maybe in in a different realm uh like in the room of diversity for example um there is a lot of research about uh why it's important to have like diverse things and like uh so we can use this information to justify to the managers why it's important uh so are there is there any studies being done like oh we if we are more ethical we can also perform better and how so that's a great question we're asking um the the platforms for ethics and ai just doesn't exist today and this is why we're talking about it and trying to see like what people can do so two questions there one what can we

do to help inform producers people on teams to think about ethics and then what are some of the trainings that already exist that we can share with managers um i'd say there's more studies on how diversity improves monetization and kpis than uh than ethics but i think you kind of get one from the other or at least the brightening of that yeah i mean there's a lot of horror stories out there right now about like machine learning gone bad right usually as a result of a lack of diversity exactly um and i think you know finding those kind of articles is a really good start for for this kind of work we should do it more yeah also maybe also about having a great diversity and and your team is going to help try to figure out how this this system is going to work and to try to come up with all the different how all the different ways it can go bad uh if you only have like people that

look like you uh on the team it's gonna be very hard to come up with all these examples um so that's that's a way i said to you and i would say for us it's talking about this not just that advocacy conferences but talking about this with the people actually working in the field so they start being more aware um last question nicole i think yes thank you great great panel really awesome that it's here at the ais summit and uh my question is just an extension of the idea of like uh well and in the context of facebook you know i actually have a gender blocker i i declare myself as non-declared uh and what i found is that i got less wrinkle cream and dresses and you know i mean you're going to get zuli zulu ads no matter which gender you choose um but i thought that was a way of me as a user saying look i you know that's not what i want to see and i will go out intentionally on the net and like go to some very you know specific sites in order to kind of crack the you know the code that

might be you know that's uh the preferences that the ai is is um is assuming about me so my question is you know is there a lot of talk about you know getting giving users control over their over the platform can we have a reset button we've talked a little bit i've heard a little bit last year about having like a reset button not just for ads in general but you know like my my social media feeds i want to you know be able to like hit a reset button and just you know start from scratch like i may go down a really dark path and am i going to be able to like my way out of that dark path and i've got multiple personalities i mean not as a not not as a not as a medical condition but just simply you know i you know i run a studio i'm a designer i'm also an engineer i want to be able to and i'm lively environment so i want to be able to switch and i don't have those kind of controls either so there's this efficacy of just being like one thing for everybody and that's you know just like a fire hose without that flexibility so are people talking about the ethics of user control certainly they are um the the idea of

especially the right to be forgotten is like how people talk about like your permission to have information about you removed from the system and there are specific applications of that under gdpr but that could apply in a lot of other cases as well um yeah i know we don't have a lot of time to answer so yes especially in europe there's a lot of people thinking about that you know there's a lot of committees uh in france you have uh the cnil that's looking at all these questions but yeah in europe you can ask for your data to be removed so i'm not sure yeah i mean it's not necessarily easy it kind of depends on what but i think yeah i mean there's i think but as people become conscious of that like building products towards a point where it becomes easier to opt out of things um or say you know what i like i i would like to change my gender presentation now and that's across the board so let's acknowledge that like all of those kinds of applications yeah i mean there's also there is a flip side to that is the ethical

consideration of people using that kind of tool to remove things that people should know about you know we there are a lot of kind of right to be forgotten cases where it's a politician who doesn't want some image to be circulating generally of them with some high profile donor or something you know there's a there's a counterpoint to the ethics and i think there's use cases on both sides yeah but i think it's also not just the right to be forgotten in terms of what i publish but as a consumer i've got a right to you know adjust you know the difference if i want to be able to go to a horror movie or a wrong romantic economy or something like that i don't have that flexibility in social media right now i don't have like a wide variety of things it's just one thing and again you can navigate yourself into some like very specific like feed content and it's very hard to get out especially if you're not a developer i think we're going to have to continue the discussion out there we are already over but thank you so much thank you to our panelists