A Practical Guide for Doing Ethical Player Testing

[Music] good evening it's 5:30 so we should get started thank you for coming to the last session of the day I really do appreciate it mm a couple of reminders first set your phone to vibrate you'll be receiving your valuation over email so please remember to fill that out and if you have questions they'll probably be pushing us out because this is the very end of the day so there's a wrap-up room or wrap-up Corral that's that's right across there so I can answer any particular questions that you have my name is MIA can solve oh and I'm gonna be talking about a practical guide to doing ethical player testing why should you listen to me I've been studying game players for about the past 17 years now and I've used a variety of methods to do that I'm

mostly interested in you know who plays why they play and how how player culture adapts and evolves over time and so I've been getting at that mostly from a qualitative perspective so a lot of observation both participant and non-participant observation I do a lot of interviews and focus groups and I've done a bit of Survey Research but I don't get so much into like experiments and like big data analytics overtime you know like I've investigated a number of different things and with respect to players so my first book was about cheating and games and how and why people cheated how they defined it what they did or didn't do according to their own definitions I've looked at a casual MMO that was online for a short period of time called fauna sphere most of the players were older women and so there was an interesting dynamic going on there and when most recent book is Atari

- Zelda which is about Japanese video games and what happens as they traveled to the west including how Western players take them up and make sense of them so I teach at Concordia University which is in Montreal and I run the em lab where we we play games we study games who study players and we try and innovate in better methods for studying play so what I want to do during this time is give you an and I wasn't going to include the first point but the track administrator said we really want a history of how this evolved so I was like okay so you're gonna get a very short short history of why for example universities require researchers to go through like ethics training and get ethics approval for their projects usually of course it's because somebody messed it up for you I'm going to talk about some important concepts to keep in mind for testing some questions to ask yourself when prepping for testing I have a short section on VR because it's just becoming

more and more of an important thing to think about and hopefully we will have about 10 minutes for Q&A at the end I should point out that this is not a talk about improving your game design through player testing there are many other people or you're probably more of an expert there there was a great talk yesterday about evil data and its relation to play testing and how to get better data or you know like sift through to get really you know ask the questions that you really want to ask instead what this talk is about is ensuring that your play testers are getting the best possible experience that they don't leave feeling vaguely bad or mistreated you know that they didn't there was something that happened that they definitely didn't sign up for and so you know they're happy they're engaged and so they are giving you the best possible data to work with and also if you're hoping to work with them again that they are willing to come back and and test again and not just you know run

screaming and telling everybody you know what a horrible experience it was so a caveat you know it would be easy if this were a list just a simple list of do's and dont's instead most you know researchers will say that you know thinking about ethics with your research is like a series of questions that you have to ask yourself and sometimes there may be a definitive answer but times it's really not that there's one right or wrong answer but it's that you've considered it and what are the consequences of going one way or another okay so you know these are some guidelines for you to consider some you know hard-won knowledge that other people have found so a brief very brief history of human subjects research I'm just going to talk about two studies and you may have heard of them already all right well if this will advance okay and these are from social science medical research has its own outliers in terms of very bad practices but in social

science in 1961 Stanley Milgram did a famous experiment where he was interested in how obedient people would be to figures of authority and so he recruited folks and said you know I'm going to you you pick a piece of paper and you can be either the tester or the testee and conveniently everybody was the tester they didn't know it the test II was just somebody who was in on the study but the testers were told you're gonna be watching them and they're gonna be trying to like it's this test to improve their memory they have to they're going to hear words they have to repeat them in sequence and we're trying to see if shocks you know if pain will increase their ability to you know engage in in memory recall or you know increasingly motivate them and over time there were more and more words and the shocks got stronger and stronger and Milgram actually thought that most

people would drop out like they would refuse to do this like that they would refuse to engage in the shocks now of course these people did not know that the shocks were not real the people who were being shocked were actors but disturbingly they found that many many people were perfectly fine going along and shocking people more and more I mean these people were like screaming one guy was like banging on the wall you know in apparent distress and so they said you know wow this is really showing us some interesting things about how obedient people are to Authority this is important as we go along but also another important element is that the people were being deceived but they didn't know you know of course that you know these other people were not really being shocked which may cause you some psychological distress that carries over but also the people who were doing the shocks were not really told that they could withdraw from this study there were four prompts like if they were being reluctant the person who was watching them said you must continue and

then it's really vital that you continue and there were four going up - like it's absolutely imperative that you continue this study so even if these testers felt uncomfortable it was a situation they mostly felt like they could not escape from the second experiment is the Stanford Prison Experiment which happened ten years later Philip Zimbardo ran it and he was curious about the psychological effects of becoming a prisoner or a prison guard he didn't know if it was the situation or you know individual psychological traits within people that would lead to problems in prisons and so he recruited a group of college students and they were randomly assigned to be a prisoner or a prison guard and it was set up to be a two-week experiment they set up a mock prison in the basement of a university lab on campus the prisoners interestingly they were not told exactly how this would start but they were arrested at their homes one day driven to prison they were processed given a jumpsuit and they were

given a number and told you know like you're not going to be referred to by your name now you're this number for the duration and if you're familiar with it you know that things quickly went downhill the prison guards wore reflective glasses so you couldn't see their eyes they were told it was perfectly fine to psychologically you know harass the prisoners they quickly and readily fettle fell into this role the prisoners also fell into their roles where some of them the ones who tried to fight back where harass you know and further abused many of them just fell into silence and depression some of them actually had breakdowns and had to leave early another important element the subjects were told that they could not leave voluntarily and Zimbardo himself was not a neutral observer he couldn't really step back and see he was playing the role of the prisoners superintendent it was actually his girlfriend at the time who is a graduate student who stepped in and said hey this is kind of

problematic I think you should really look at what's going on here and they shut down the experiment after six days also another important element that led to further review by universities is the fact that they never actually talked to the people involved like the subjects until almost a year later to see like hey how are you feeling are you okay it's anything going on that might have caused you distress on going after this so those are the types of things that led to universities saying to researchers hey you know we're gonna have some protocols and guidelines if you're doing any federally funded research and actually if you're a researcher in the US or Canada even if you're not doing funded research because your university accepts funds you have to you know get ethics approval obviously you may not have to do this but many of the standards still apply for you know like what are the what are the principles and best practices for engaging in research and one of the the

big major one that I'm going to start talking about is consents you know what it is that people are agreeing to and so in the Canadian guidelines they talk about consent has to be free informed and ongoing and by free they mean voluntary so I'm going to go through each of those three categories of consent so voluntary obviously means no undue influence or coercion the easiest example of this is you know if you you force somebody to play your game you know I force somebody to participate in a research experiment with me you know I maybe I threatened them you're going to fail my class if you don't maybe you say you're going to be fired if you don't do this obviously that is they you know they cannot give their consent you know a little in a more of a gray area is undue influence and so one of the things to consider is is there a power relationship going on between the investigator you know you who's setting up the test and potential participants now it doesn't mean that you should

never like it's it's never okay if you find somebody who you're in a power relationship with just that you should be thinking about you know is there some way that I might be you know unconsciously coercing them or convincing them to do something are they really free to say no I don't want to do it right if you're not sure if it's a power relationship think about you know do they trust me you know and am I really pulling on that trust to kind of get them to agree participants should know obviously with informed consent what's expected of them during the test now this does not mean that you have to tell them every single thing about what's going to happen every element of information you know involved in your game but that they should have some general idea you know of what they're signing up for so you know what's the game what are you asking them to do you know are you asking them to test like the first few levels are you

asking them to go through the entire game is it characters that you know you're asking them to like react to like or is there a character that they prefer is the storyline believable you know whatever now it may be the case that you know there's something that you don't want to specifically tell them you know how long is it gonna take you to get through this level because once you tell people something then they're unconsciously thinking of that but you can frame it in a more general way right so at least they have some idea they should also know like how long they're going to be there roughly you know is this a 30 minute test is it a two-hour test what's gonna you know how much time do they really need to budget how will they be observed and recorded I'll get into this in more detail but just you know like what are the elements involved in what what you're doing in terms of watching them and also that it's okay to change their mind so at this stage you know after you've explained what's going on maybe they're gonna say like oh it's a survival horror game I really don't like those this is not for me okay you

know it's fine for you to back out they should also be told you know what some of the risks are and if there are any potential benefits involved I'll get into this in more detail also that they're under no obligation to participate and again that they're free to withdraw at any time and so you know it's really in terms of your being open and honest about like saying hey this if you're really not into this maybe you know this is not for you and I should also know you know like what's gonna happen like if you're recording their data are you gonna identify them in any way like if there's a video camera you know are you recording their face their facial expressions and is this something that you're going to keep or are you gonna like maybe gray out or blur out their face or something there should be obviously some contact information so they can ask you later if they have any questions or concerns you should talk

about how you're gonna actually use the data and so like is this something that it's just you who's looking at personally is a member you know is it like your team is that everybody from the studio are you gonna use it oh in a GDC presentation you know and you may not know immediately like how far you will take it but you know you should think about that and say like it may be used in these particular spaces maybe you're gonna have blog posts or something about you know like testing and what's going on you should have information about like any payments or incentives and also you know you should give them a reasonable amount of time to go through this so that they don't feel rushed and they can also ask questions and this can be these things can all be things that you can either you know print out so they can read quietly or you can explain it to them verbally either way is fine consent also has to be ongoing what this means is just that it's not enough to get somebody to say

like are you okay with this I've explained everything they're like yep and then then it's over you know then you're in like prisoner territory it's like they said what yes once and then no matter how horrible it was it was like oh well you signed up for it too bad you should people should feel like they have the opportunity to leave at any time if yet for whatever reason maybe they don't like the experience maybe they're feeling sick you know you know if they're not having a good time they're not giving you good data and so you need to think about like how what are the different ways that you could like remind people during the play session you know it's okay like we're gonna take a break now if you need to go now that's cool or whatever if you have multiple aspects of the test are there things that maybe if people are uncomfortable with one aspect you know that you could omit that for them but have them do others or maybe you need people to complete the whole experience and just be upfront with people and say like if you're not if you're not

comfortable with everything then this isn't for you so again they should be told that they can stop not just leave but stop at any time and the environment should support this so it's one thing to say this and you know you may not have control over where you're doing the testing you know maybe it's in a room down a corridor with no windows and you know they're you're sitting in front of the door and I mean that's like a very you know tough thing to do like people already is those experiments show kind of obedient to Authority like my students you know if I sit at the front of the room and the doors near me it's very difficult for them to like get up in the middle of class and walk out right like that's not a cool thing to do they don't want to draw attention you know and that's another way that like the environment can unwittingly shape people's expectations you know so if it's possible for you to think about this and alter the environment so that people feel comfortable that's one thing you can do if it's not then you know it may be a case of just reminding people more frequently and there should be multiple

ways to withdraw so maybe you know they can tell you or maybe you know like I said it's a break and they just kind of disappear so you can use incentives you may already use incentives probably for most testing and I've found this with testing you know playing the game is often incentive enough for most people you know I ran a study where people were playing games that had already been released the Walking Dead series from telltale and also Dragon Age Inquisition we asked them to come in for three to six session of 90 minutes each and we didn't have any dropouts and we didn't offer any compensation they just kept showing up because they enjoyed playing the game the one thing to think about in terms of ethics is you know if you're offering very large incentives because then it becomes about people just signing up for

the money or whatever the swag is that you're offering them so deception you know I mentioned you should be upfront in telling people what it is that you're asking them to do but you know is it ever okay to deceive people is it ever ethical to deceive people and yes it can be you know if as I said there are things that you're trying to measure and if you told people exactly what it was you know that's obviously gonna it might change how they react but the big thing just to think about is what's the risk involved in the deception and does the benefit of the deception outweigh any potential risks okay so this is where you're in the balancing act of you know are you just doing this because you didn't really think it through or you know is there like something really important that you need to get from players and if you told them ahead of time you know it would not work

also another thing to keep in mind is you know you've been working with this game you're close to it you know it so intently people coming in may have no idea really like what it is about and so for them many surprises are going to be you know magnified probably the majority of the time the testing that you do under university rules would qualify as a minimal risk the university speak for defining minimal risk is probability and magnitude of possible harms implied by participation in the study is no greater than those encountered by participants in those aspects of their everyday life that relate to the research it's amazing how convoluted universities can make you know simple statements so if this is something that people would probably do in their everyday life then its minimal

risk right so if your game is rated you are targeting it like efore everyone a t14 is probably something that they would do in their everyday life and so you would be qualified as minimal risk there are a few times when you go beyond minimal risk minimal risk doesn't mean that you have to ignore all of this but just that you know the stakes are a little lower if you're beyond it then you do have to think a little bit more carefully you know and you might want to consider some of the resources I suggest at the end beyond minimal risk can happen when suddenly you're involving vulnerable populations which is how we talk about them so children those people with diminished capacities so people with like cognitive impairments you know where consent may be something that you have to really think about how they can understand that and also indigenous peoples so indigenous peoples in Canada have this you know very bad history of being exploited and abused they were tested on by different people just as

like a convenient population to see what would happen and so now they get extra protection as a result I just want to talk a little bit more about children though because that's probably the population you would deal with so obviously with kids you know it's not just the child but the you know their parents or their guardian who would need to consent but also think about the age range you're including and why because kids aren't necessarily you know an amorphous group a seven ten and a thirteen-year-old are all going to be really different in terms of what they understand how they think about the world how they can communicate back to you what elements of the game they might think are interesting or exciting or scary and so you know think carefully about how to you know which age great which age ranges you want to include and I would say this this is a common recommendation for like researching with children I would say it would apply to

anybody you know as the prior experiments have shown adults sometimes uncomfortable with you know saying no I feel uncomfortable doing this kids are even more so so are there ways that like with nonverbals you can figure out like they're not having a good time you know are they closed off you know they started laughing now they're not they're looking down their body language is closed up you know that can be a way that a kid is signaling they're not comfortable anymore I'm not going to talk anymore about children but if you do research or play testing with kids I would say check out this talk from the vaults in 2014 Gareth Griffiths talked about testing Skylanders with kids of different ages and he's got some really good information about like how to do that so you've identified who you want to test

with and what you want to collect now you want to think about how you're gonna store this you know how are you collecting your data you know are you audio/video recording are you going beyond that are you just taking notes but how are you storing it is this gonna be secure in any way is does every member of the team have access to it which can be fine but you just need to know like how far you want it to get out how are you identifying participants you know are you using their names or are you like taking off their names and then assigning like a pseudonym again what are you doing with the data and also what happens to the data when the study is over when your play tests are done when the game is released if you're like me and your pak-rat you tend to just hang on to everything you're like oh I don't know when I could possibly need this again but I might but you know it might be better just to erase it and get rid of it so that you don't worry about it getting out anywhere and related to that you know how are you going to

identify these people there are different levels of control you can have so you could keep people anonymous this usually works with like online surveys where you don't even know who these people are but maybe people come through who never physically identify themselves to you you know and you've got this data that you just is like literally unlabeled more commonly you know you know who these people are and then you have the choice of saying you know we're going to keep this information confidential like your personal name and you know address or we're gonna identify you a lot of times people offer them the choice like do you want to be identified or not chances are you're not using this outside of your studio so it's not going to make a difference but if you did want to use it you know in public facing materials this is something like to ask people are you okay with this so VR there's a study that came out last year talking about research ethics in VR testing that said you know VR poses

risks that are novel that go beyond the risks of traditional psychological experiments and isolated environments and go beyond risks of existing media technology for the general public even though VR feels pretty new you know there have been waves of interest in VR and some researchers have been studying VR for quite a while and so there are a few studies that we can draw on to kind of get a sense of how people might react with VR systems in 2002 there was a study of the virtual pit so they had subjects put on a head-mounted display and stand on the ledge they were asked to lean over and drop a beanbag into a pit now in reality they were standing on a wooden platform that was an inch and a half off the ground and they were told this like they knew it as they were getting on it but still even though they cognitively knew this was going on their bodies felt that it was a real situation that they were really standing over a pit and so their heart rate increased their sweating increased you know some of them felt very uncomfortable so even

as you know they knew in their brains this is just a test this is not real their bodies reacted like it was real remember the Milgram experiment with the shocks that I told you about somebody redid this experiment in VR in 2006 but what they did was they asked people to administer shocks to a virtual human so they had some people who saw the virtual humans some that were just it was a text communication but the inform the the testers knew that this was a fake person right and so but they were told well we're seeing how the application of pain applies to learning with virtual humans I don't know how that would apply but this is what they were told and interestingly again even though they knew it was fake they reacted bodily as if this person was real so two people you know they had to read words and then the virtual human had to repeat them back and if they got them wrong they

shocked them to participants emphasize correct answers you know like these are the you know they're like reading them out kind of like hints hints and you know asked when asked about it later they were said they were trying to help her viii repeated the question after receiving no response so the virtual humans quiet and they're like hey come on come on let's go let's go even though they knew she was fake they showed increasing frustration with wrong answers the researchers concluded people behaved in a way that only made sense if they were responsive to the virtual character as if she were real they said humans respond realistically at a subjective level so they were like justifying it verbally afterwards physiologically you know they could tell their heart rate increase they were increasingly uncomfortable by the way they also kept going up with the shocks even though they said that they really felt bad they felt bad in doing this to a virtual person and behaviorally as well like their physical body they could tell that they were uncomfortable doing

this so you know they said this could be evidence that again we are obedient to authority figures also maybe a matter of participants being willing to put up with their own discomfort for the sake of honoring their agreements to be a participant in the experiment and I just want to underline here for the sake of honoring their agreement to be a participant in the experiment so we already know you know like people want to help you you know there's other research that they'll you know make themselves uncomfortable with VR even again they felt you know fair amounts of distress and they put aside their own distress because they really wanted to help people the question is you know are you really getting good data if people are uncomfortable with the experience if they're not having a time on the positive side of things Nikki and Jeremy bailenson have studied what they call the Proteus effect putting people in VR and for example like giving them an avatar that's taller than themselves or better looking and like asking them to negotiate and they find that they're more confident and

able to better negotiate and that actually this ability carries through after the VR is over they don't know how long it lasts after the VR but it did last you know after they continued after they entered the testing so what this means is with VR you know there's a bit of a tautology like the central the one thing we say is do no harm to people but even though we know a little bit of what's out there you know and researchers will say there should be no foreseeable harms with the research that you're doing we don't know what necessarily the foreseeable harms are with VR people are reacting differently than they do with standard like research and so with VR it's more about again carefully considering like having a statement to tell people you know immersive VR can have lasting behavioral influences some of these risks may presently be unknown it's just being honest with people like we really don't know what might happen like that's the truth also telling people that they may

have powerful emotional responses to game content whether or not they believe it's real you know it may be one thing to say like oh that's you know like I'm used to scary games but it's another when you are in the scary game and you know we know that people's bodies react powerfully even when their brain is telling them it's fake its fake its fake you know torture in the virtual environment is still torture you know if you're perceiving it as such also with VR there's the chance that you're getting a lot more data suddenly it's not just a video recording but maybe it's itraq eye movements bodily movements you know you're tracking their kinetics over space and time and so you know you need to think about what it's fine to collect that data but how are you storing it and how long are you keeping it and what might you reasonably erase at the so that you know people keep their privacy and also just beyond testing

thinking about doing a game development in VR we don't know the psychological impacts of long-term immersion in VR I mean the games that you see now are fairly short experiences obviously testing is a pretty short experience but you know like it's still an open question in terms of what long-term exposure might be and also like what is the potential for abuse of avatars you know that look like their users not just when a person is in the virtual space with their avatar but if they have ownership of an avatar you know is there the possibility that somebody might be able to take it or copy it and exploit it so that's the background so now you want to create a protocol for playtesting and I'm just gonna go through here with a game or two that I've created with some questions based on all of this you know to kind of guide you know like thinking about my thought processes as we went through and also

some additional ones like for Imaginary features that we would add so in 2010 I was part of the gambit game lab at MIT we had a group making exa I led the Y's kind which was just a puzzle game that was on Facebook and this is what it looked like and just so you know I when I put up this screenshot I realized that at the bottom where remember the old Facebook games where you had the faces of all of your friends that you were playing with and I had all the faces of all the friends I was playing with I was like oh I didn't ask them if I think their faces could be up there so I have cropped that that's why the aspect looks a little off so yeah didn't want their faces necessarily associated with it so what it was I testing for so with this game you know we did a couple play tests over the course of the summer that we were making the game and at first you know it was do players understand what to do in the first level because as you saw it's a puzzle game you twist tiles so that your little avatar can make

their way through you can make puzzles for other players and share them and it's timed however long you know it takes and you so you can beat people's times and share other puzzles so do they understand what to do do the instructions make sense and how long are players taking to get to the end of the level because at first the first few levels we created for people so that they would kind of learn how to do it so with recruitment you know we could ask so who's the player base how can I recruit people from that group without exploiting any trust or power relationships you know we were lucky that gambit had a reputation for doing play tests regularly over the summer so we had a good pool of people who were happy to sign up and attend just for the pleasure of playing the games in terms of incentives and rewards there was the playing of the games and then each team with a game usually had cookies sometimes we would compete with each other for like better cookies or chocolate or that kind of thing and so

we didn't have a you know relationship we were necessarily exploiting and where was I asking them to play tests so these were labs with several doors you know it was easy to get in and out of them people moved freely there was no way to really kind of trap people you know in a particular space how will I handle obtaining consent so with gambit at the time there was a blanket consent that people would fill out when they came in to test the games if I'm doing other testing of other games you know then I can have like a sheet of paper with all of the information on it to hand out as people start that's usually the way that I like to do it rather than verbally explain it because then people can just kind of take their time and think through it and how do I ensure testers feel comfortable enough to leave if they really want to quit you know we did have people observing people playing exa but they tried not to be really intrusive they were sort of standing back but we

would tell people a couple of times you know like you can leave whenever you want there's also in this game like once you solve a puzzle they're fairly quick it's not like you're the game itself makes it difficult to just sort of get up and leave deception is there anything in the game I want to be a surprise to testers there was nothing here that really stood out is there a way to let testers know there's some material that might they might be really bothered by in this game no there's another game that I'm working on on my own where as the geek of all geeks I'm making a game about being a department chair at a university and you know there are multiple problems that you encounter and you know one of those problems could be dealing with sexual harassment you know and I want players I don't necessarily want to tell them in advance like every single problem that the player character will go through but I might you know say in the consent form like as an administrator you deal with

multiple problems these could include sexual harassment student complaints shrinking budgets whatever you know let me know if you have any questions about this so that somebody came up to me and said hey what do you mean by this I could tell them like here's what the scenario is are you okay with that because really you don't know when people are coming in what their backgrounds are right like you don't know in their history what kinds of bad experiences they've had or maybe that you know their friends and family have had that they've been a party to maybe they just had an incredibly bad day you know why make them feel worse by you know springing something on them but in that way by sort of generally alluding to a situation I'm not informing everybody you know of what's coming but the person who might have a problem with it you know then can talk with me and I can give them more information so that's where the you know is the surprise really worth it or not now if I were doing a vr version of this and this has just gathered from research from play

testers play testing groups who are working the VR now but you know having a kill switch so that everything quickly turns off so that people if they're uncomfortable and upset can get out of the situation especially with VR because there are so many parts involved it's not just as simple as taking your hand off the mouse right having a protective bubble around people's avatars whether it's with because they're playing with other people or even potentially with NPCs so remember the virtual human people felt bad about hurting the virtual human so it may be the case that they feel bad about being attacked by an end piece and again protected populations which I can deal with in the QA if you're interested so it might be in useful if you're doing VR research to find information for people like if you're dealing with certain issues or certain problems that you know like you know sexual harassment or racism or whatever

like to have information available for your testers to know like if you're bothered by this here some people that you can talk to you so other questions you know I would ask as I was doing my play tests so how am i recording with both games I wasn't actually recording any gameplay I was simply standing behind people watching them taking notes and then I would also ask people to fill out a brief questionnaire at the very end you know asking them about specific elements of the game because of this there was no identifiable information like linked back to people what we would usually do is say like tester you know one who sat in this chair female you know and maybe ask people for like an age range so that we could you know kind of get the data and sort of understand maybe how it broke down so I did not share that with anyone so I didn't need to worry about that again we kept the

data we created a Google spreadsheet with it and then at the end of the project got rid of it and yeah it's gone now if we were doing a multi play test you know there are questions to ask so do players you know like in terms of what am i testing for suddenly if you've got players not just by themselves but may be competing or collaborating with other players do players understand how they can interact with other players do instructions do the instructions I'm giving them make sense maybe you're testing out harassment tools and so you're thinking about are they using them you know and what kinds of toxicity am I seeing just a note here based on some research that I found about harassment a lot of times when you ask people about harassment like have you been harassed they will say no that doesn't mean that they weren't harassed sometimes especially like with teens they don't see what I

would see as harassment as harassment they would term it drama so if I say have you been harassed they're gonna say no but it doesn't mean that there weren't problematic behaviors so it could be the case that you just asked generally are their interactions that bothered you you know were there negative things that happened you can also ask you know as a way to kind of offload did you notice negative comments about others in the game which is usually an easier way to get at it right because people are usually much more willing to say oh I noticed this directed it so and so rather than like I was harassed they don't want to be that person so putting it all together if you want to do a consent form this is just some language that would distill it down and there's slide after slide I'm just going to read them through quickly so I understand I've been asked to participate in play testing of X I love the Y's kind I mean I can solve oh and

these are the kinds of consent forms that like I would have to give to the University to show I have been informed that the purpose of the play testing is to determine whatever it is that I'm testing for at the time I understand I am being asked to play a game that isn't finished for about 30 minutes I understand that the computer is video recording my gameplay I am being observed by someone while I play who is taking written notes obviously at the time we weren't doing that but that's something else that you would add I understand that I am free to withdraw my consent and discontinue my participation at any time without negative consequences going to risk there might be certain risks these risks include feeling frustration or anger while playing or hearing other players saying negative things during gameplay which might upset me this would depend on you know which way you're going with the data but I understand that my participation in the study is

and you can also offer people here the choice so they could circle the one that they want it to be you know confidential the researcher will know but will not disclose or non confidential my identity will be revealed in study results so sometimes you can say pick the one that you're comfortable with and then I've carefully studied the above and agree understand this agreement I freely consent and voluntarily agree to participate in this study some additional resources for us-based developers the guidelines the Belmont report and some other training modules are at these URLs and then for Canadian based developers there's another website and for European based developers there's a third website European researchers usually have slightly different guidelines to think about there are stronger protections for

privacy with people working in a European context there's also slightly different guidelines in terms of they may not universities may not always make such a heavy hand in terms of like asking about these matters they may just rely more on researchers who are doing their own work but depending on which context you're in it can be useful to see kind of what the standards are and then again if you're interested in VR based testing this is what I was drawing a lot of the material from this study from last year real virtuality a code of ethical conduct recommendations for good scientific practice and the consumers of VR technology it's an open-source site so you can read the whole article and it's also got lots of good links to other research other player based research using VR that goes back you know 10-15 years or more and that's it we have plenty of time for questions so we have a mic here if you would like to come up and ask thank

yeah hi there my name is James so I have a question regarding recruiting for recruiting the testers and when does it go from targeting a type of tester to almost like discrimination do you have to worry about for example like sometimes I just want beginning beginner players right so you can target that but then let's say I want disabled players is that more in order sometimes as I say female players or female pleasure of this certain ethnicity when do you start getting into discrimination well what is what is the goal of your targeting I guess yes why are you targeting particular maybe for disabled players is to test certain accessibility features let's say right okay in that case like at when we were at gamba we had you know a game that was made for visually impaired people and so that was what they did they were looking specifically for blind people so it's looking for your primary demographic at that point

right so it's you know acceptable to do that the question it would just be not who you're including but like if you're excluding people from your study why are you excluding them and it may be perfectly acceptable to say like I'm just interested in this one segment right now but to understand like why you're doing that yeah so like newbies like maybe that's really important to say like I really need to understand how people who are new to games or this particular genre understand it because experienced players they're gonna go through it too fast like that's not the goal of your research so yeah that's perfectly fine alright thank you so in the context of games that are taking high amount of risk to physically are more so psychologically affect a player's mental capability is there additional legal obligations that you

should look into like following up with therapists afterwards things like that and the contents if there is a psychological breakdown during the testing that's a good question is especially if you're dealing with like the protected populations so like yes mmm like I mentioned you know kids are one also like I've had a couple conversations with people at the conference here about pregnant women with VR because of problems with like motion sickness like that can be problematic so that the large studios will regularly ask women II like are you pregnant or you think you're pregnant because you shouldn't you know we don't want to test with you if you're pregnant while playing because there could be an implication mmm without knowing the details you know it's hard to say for sure you know one thing that's possible is to consult with somebody if you have like a university nearby like with a psychology department or there's always an ethics board at the university to say

like hey is this something that that we should you know think about and you know they if you give them the details they'd be more able to say with certainty thank you no other questions okay thank you all for coming have a good night [Applause]