Assessment Games

[Music] my name is Tom Smith I am a game designer at ROBLOX I have been a game designer for many many years I worked in board games I actually uh wrote the first American rule book for Settlers of Katon and then board games had a golden age so that's entirely me uh I got into console games when they started doing 3D and that went pretty well I got into mobile games early early days worked on Where's my water and some other cool stuff uh that did pretty well and then I've recently moved into educational games so I'm predicting big things in the future and uh the big thing that I've been doing for the last maybe five years is assessment games which is why I'm here and what I'm going to talk about so before I go into that though um I'm also work with the Education team at ROBLOX I work on promoting Roblox as an educational platform form encouraging

people to bring cool educational games and make them on Roblox so that's a whole another thing if you want to talk to me about that later I'm happy to talk about that but first uh who here raise your hand if you have a good sense of humor if you have a a strong sense of humor if you're better than normal okay so uh hands down now who here can tell me what the funniest letter is out of 26 uh what is it no it's actually B sorry um and finally would anyone be willing to tell a twool line joke a really Qui J to the crow right now anyone okay go he's got aing Wheeling out ofs says what the deal with the steering Wheeling out of your the PIR says I nuts ah and that's one that as a Roblox employee I can the trust and safety people will not allow me to repeat so uh quick joke okay so these are all really bad ways to measure someone's uh sense

of humor so if I was looking to measure someone's sense of humor there's not a normal test I can pull off the shelf and apply so I might ask people to self-report on what they think they're good at I might ask them some questions that I made up that I think are relevant to someone's sense of humor the letter thing I actually did find B is a really funny letter when I was working on a Leisure Suit Larry game and um anyway that's a whole another story um or I might ask uh for people to demonstrate their skill um so these are all ways that you can assess things and when you're looking to assess something that is not normally assessed you have to find these sort of creative ways but the creative way that I really do recommend is assessment is assessment games games can be built as assessments games have always been assessments um when you build a game you teach someone to do something and then you assess how good they are at it so when you play Mario you are learning how to jump you're learning how to jump on mushrooms you're learning how to knock things into the sky and then the game is measuring how good you are and if you do well enough

you get to go to the next level um it's an assessment um when you're playing Beast War simulator you're being taught a very complex economy that isn't a real economy it's an Economy based on honey and pollen and things but then your assessment unlocks new areas unlocks new powers um combat games have always been very assessment based uh Spider-Man puts you into a situation and won't let you out until you've demonstrated the skill it won't let you move to the next level it won't let you unlock the new next story piece until you show that you can knock people down with webs which isn't a very practical skill but it is a skill um and this is such a thing with combat games that the idea of the boss as the final exam is a very common Trope it's uh literally a TV Trope that the boss is the final exam and that is an assessment of the skills that the game has taught you um and so when you're learning these sort of skills when you're learning how to you know avoid a natural disaster because there's a volcano and a hurricane and a a flood all happening at the same time in a Roblox game um you're

not learning traditional skills this isn't something that a teacher would use in the classroom but you can use a lot of the same principles to build an assessment that does assess normal traditional things so there's ways that you can take what what we as an industry have learned over the many years of figuring out how to make engaging fun assessments and apply that to the real world apply that to real problems that um are are useful and practical and you know teachers have known this for years decades um games in the classroom are often used as a way to engage students with a subject matter to maybe you know have some sense of how they're doing a formative assessment of how they're progressing in their learning uh progression and sometimes been used as like more formal assessment but less less common even in uh traditional classrooms and then people have tried to you know take this into the digital age and build uh video games around this but a lot of them are relatively simple measuring things that we've already

measured I mean Math Blasters isn't really teaching you a lot it's really assessing a lot it's assessing how good you are at simple math problems and this is great and you know like there are other ways of assessing these things though we you know multiple choice questions maybe aren't perfect but they do a decent job of letting someone know if you're good at math or good at English or good at you know these sort of traditional subjects um what I'm interested in is subjects that are hard to measure in these ways there's a lot of skills a lot of uh subjects that were really interested in and that we really need to know in order to move the workplace forward to move education forward we want to know how good people are at communication how good they are at systems thinking how good they are at creative problem solving and these are things that you know people have tried to build in the sort of traditional way and they haven't had much success so that's where games can come in and that's what I've been doing so we build games that are used as part of the hiring process at ROBLOX uh there are these games are just one step in a

larger process you're not given the job based entirely on the game but it's part of that process and we're not the only company we're not the only uh group doing this there's other games other people that are building games that are meant for assessment um I'm not going to speak to these exact games because I don't know exactly what they are I haven't worked at these companies but they're out there the it's it's it's a small but growing segment of the industry and we started as a uh startup we started as a company called embellis we worked with McKenzie the consulting company and we built some things that are still being used um for their hiring process um but then we ended up kind of splitting up between some of us went to Mackenzie and some of us were acquired by Roblox and so now um today if there's uh there are certain jobs at ROBLOX not all jobs at ROBLOX that if you apply for you'll get a a screening call with a recruiter and then we'll send you a link to a Roblox game and it looks something like this this is a piece from the one of the tutorials and you'll play a game and it's it's a game it's you know we give you a set of controls we give you a

world that you're you have to understand what's going on and the systems that are involved there and a goal and you're trying to do something and so you know we're not building a little multiple choice mini game we're building a you know pretty much a full puzzle game strategy game type thing um and this is something that at the end of you know 20 30 minutes of playing this we can say something definitive about your ability on a specific cognitive skill so we have assessments for Creative problem solving for systems thinking situational awareness um we've talked about some other ideas that we might do in the future but um we build to that specific skill and so we don't reuse our assessments our assessments are specific to that skill so if we want to build if we want to assess something different we build a different game and it's a it's a completely different game but with the same principles it feels like a game it's 3D it's you know got a lot of different inputs a lot of different options it's it's a very open uh experience and we've been doing this for a while now um at roblocks alone we've

done you know tens of thousands of Assessments and uh we've gotten some good data we're very happy with how it's turned out I am not a uh cognitive scientist I'm not a psychometrician I'm a game designer so I have to trust these people who tell me that it's good so um our our reliability coefficient of 0.93 I am told is very good um these are things that some of you probably know better than I do in this audience so I'm going to trust my experts here that's that's one of my lessons is trust your experts and because of this uh Roblox has kind of broadened its hiring uh funnel so uh you know while since we have introduced the assessments into the hiring funnel we've also made some other changes it's not entirely due to to the assessments alone but we've been able to open up and hire for more colleges hire a more diverse uh pool of candidates and so we're really happy with that result we're really happy that this has made uh the hiring

process at ROBLOX even better um and that's really been the goal and we feel like even in a couple years we've only been there a couple years it's already we're getting really good results we're going to keep tracking the data and make sure that that continues on and that the employees that have come from this are doing great work but um we're we're really happy with the first couple years of progress so most of my talk here is for me to talk about how we do this um The process by which we go about this and especially from a game design perspective what challenges uh we have building assessments that are also games um so mostly this is me complaining about how hard my job is um but to start um if you're like me this is me um one of the things you like about the games industry is you get to work with a lot of really interesting diverse intelligent creative thoughtful people and so that is something that I really enjoy I this is these are some of the people I work with um and these people have a lot of different skills a lot of different expertises a lot of different

perspectives they are really creative people really technical people really thoughtful people and uh one of the things that we've been learning as an industry as as the game industry I've been working games 20 some years um you know this cre in order to get a good product out of these very different viewpoints you need to create a complex system a process that allows everyone to work together everyone to communicate everyone to uh work towards a shared goal that has all these different intelligent perspectives in it and if you think this is easy I will refer you to basically every other GDC talk ever um because this is hard uh these people have different very different worldviews um but if you can make it work it's great um every game that is shipped is in my mind is a success because they've worked through these problems figured out how to make this happen and now in my life we've introduced a whole new set of uh perspectives so we add the academic perspective to these already Divergent viewpoints and it's it's great for me I enjoy it because I love working with all

these different interesting intelligent people but it does mean that now we've got some new uh new people at the table that we have to account for in our process and in our systems and if you've worked with systems you know that when you add new things it's not an additive process you really have to kind of re-evaluate the whole thing um every change you make to the system is going to Ripple through everything and everything is going to require new uh processes new thoughts you don't you're you're not adding you're rebuilding you're you're starting over and thinking about everything from A New Perspective so that's one of the things I really uh encourage here is don't look at assessment games or even educational games or any sort of serious game as something that oh I'm going to make a game and also do this other thing it's a whole new new Beast that you have to figure out on your own and so we still use a lot of traditional game development tools and processes and techniques and again like I said I've been in the industry for a while we've learned some things like you want to start with a good core Loop you want to start with a good core piece of gameplay

that is fun and engaging and that is still true for us um but you know I've worked in freet tolay Mobile where your core Loop needs to monetize well it needs to engage people it needs to you know have people returned for day two and day 30 and day 60 um these are not things that we think about for assessment games we don't monetize people at all there's there's no money involved uh we don't retain them at all if they when they play our game the goal is that they play it once for 20 minutes and never Play It Again so our day two retention is zero and that's what we want so we're looking at our core loop from a very different perspective our core Loop is all about are we going to get the scores that we need are we going to measure the thing that we're trying to measure and that's another key lesson throughout everything I'm going to say today is scores our number one priority for us because that's what our our goal is um not traditional games we still want to be entertaining and fun but number one priority is can we score what we need to score similarly prototyping is a great tool that I've used in all sorts of

games um it being able to quickly build something to answer a question about your game is so much better than sitting around an arguing about it in a room for a week um and I've I've done that but um I found prototyping a much more efficient way of doing it and for these assessment games that's even more true we really need to be putting things in front of people as quickly as possible because everything that we care about our scores is happening in their heads and I'm going to talk a lot more about that in a minute so I'll I'll leave that for later slides documentation um something that I did a lot when I started in the industry it's kind of tapered off a little bit but now I'm back to it um we need to document everything because these different perspectives a lot of the people that we're working with don't know traditional game development and so when we describe the game systems they need to extrapolate the scores out of that they need to extrapolate what the player is going to do out of the player actions and before it's built they don't know what those are going to look like so we do a lot of documentation we do more documentation than most game companies I've been at in the last 10 years and finally play testing really great tool

the best tool for all game developers to know if your game is working also super critical for our assessment games and honestly I think for a lot of serious games um you need to put it in front of people to see if you're getting the effect that you want uh medical games Health games educational games uh even more important than Entertainment Games and I'm going to talk about this in more detail going forward as well so uh the rest of the talk is me explaining our process and a lot of this is traditional game development I'm going to assume that you have some familiarity with traditional game development I'm not going to spend the time I have uh walking through that again refer to every other GDC talk ever um but I'm going to try to point out places where it deviates where we had to change the system where we had to change the process and why and especially um places where the game design has to account for different things so that's going to be the rest of the talk um and the top piece here is kind of a a

walkthrough of that and I'll kind of keep that as a guideline for what's going on because these are live active assessments that we're using in a true high stakes way for people's jobs I can't always give you all the details of how we made those decisions I can't explain exactly how the current active assessments work um so I apologize for that where I can talk about those I will um if I can't I'm going to substitute in a a pretend sense of humor assessment because I can kind of walk you through the steps that we take with a fictional example um please don't go building a sense of humor assessment I don't think it's a great idea um but if you do come back and tell me because I'd love to hear how it works um okay so our process starts uh like you can see here with the uh what I call the request phase the like let's figure out what we're going to make and so you really have to figure out what you want and from a variety of different perspectives um yes people of the right age will will get my joke here um and so you have to hit all the sort

of traditional game development needs what's my budget what's my schedule what's my team makeup um all of those questions that are true for a traditional entertainment game are also true for us as an assessment game but then we have to figure out a lot of other things we have to figure out you know what is this going to be used for uh why am I assessing the this am I assessing this for uh the hiring role like we are at ROBLOX am I assessing something as part of an educational process because there's a big difference between an assessment like ours that is really about the assessment where the output of it is I I have a you know in our case a binary decision to make uh based on that assessment but you can also use assessment in a lot of other ways and I just uh while I'm talking about my experience I think a lot of the lessons that I've learned apply to educational games need to be doing assessment all the time to understand if people are absorbing the information that you're presenting to them and you know even if you're not building an educational game and the classroom you often need to know you know are the

students learning the thing I I'm you know I'm sitting up here talking to them I'm lecturing them I'm having them do little activities are they actually learning it um you need to assess that in some way just to know if the class is working and so games can be used for that games can be used for Career Training games can be used for um you know entertainment even with in an entertainment game like I said before you want to do some assessment and you want to think be thoughtful about that assessment and make sure that that's because you know even if you're you're not making someone choosing someone's job or choosing their education even if all you're doing is choosing if Spider-Man gets to go to the next uh boss fight it's still a decision that you want to be thoughtful about and I want to really be clear too that like when I talk about assessment it's very different than educational games um we really don't want to teach you anything uh we want to know what how your thought process works we want to know you know when you walk into the assessment how good are you at this thing not if we train you can you become better that's a wonderful thing and I think there are people doing that with games um but that's not our particular

role so we're looking at pure assessment versus uh teaching and there's often a lot of overlap like I said teaching when you're teaching you want to assess you want to understand if if the learning is happening and this is uh this is basically the difference between formative assessment and summative assessment so summative assessments are you know what do they know and formative assessments is how do I help them uh progress in their learning so our my job is currently all summative we've we've talked about do you know potentially in the future working with some job training type things that would be formative but that's not what we currently do and then once you kind of understand you know here's what my goal is my goal is to make a hiring decision my goal is to help my students progress in the class um you also need to understand what is it that you're looking for if you're teaching a class you'll have a curriculum you'll know these are the things that I want my students to understand um but in our case we're not looking for a specific class we're not looking for a specific set of knowledge we're not we're not building coding tests that test how you know how well

you understand C++ or Lua or whatever we're building cognitive assessments that are based on skills we're trying to measure these things that are are a little harder to measure in other ways so we're measuring like I said systems thinking creative problem solving something like that and and that is really important to understand what you want to measure because it's kind of tempting to just say well I'll just measure intelligence I'll just measure you know general intelligence how smart the people are and we'll hire the smart people um and it it seems like a good idea but it it really doesn't work very well it's hard to Define exactly what intelligence is it's hard to measure it in a very general way and honestly for a lot of like what I do that's not what we want we want to know are you going to be good at this job not are you smart in general so we're going to break down what exactly are the skills that are used in this job how are they used in this job and then try to measure that so we really have to break down intelligence into different skills and then Define those in excruciating detail

and that's really the next step is once we agree on hey we want to build a assessment with this team with this size with this cost uh we want to use it as a hiring metric and we want to measure things that are relevant to that job uh then we get to start moving into the need finding phase uh which for for in my example for uh my sense of humor assessment that would mean going into you know starting doing a lot of research so look at what people have written about a sense of humor look at what how people have defined it traditionally um look at experts in the field see what they've said before um and you know this is my examples are a sense of humor but for any skill there's there's people out there who have thought about it who have thought about you know how exactly this works uh creativity systems thinking everything and see if there's existing measurements um for a lot of things a lot of especially traditional assessments there are some good standards that you can use the the the

middle Slide the middle piece here the N Next Generation science standards ngss uh is a really nice breakdown of exactly what students should know at each age in each field of science if you want to know you know what should eighth graders know about biology it's a good standard to work with um it's not the only one there's other other people who break it down in different ways but you want to kind of work from that knowledge base of people who have uh done this before but at some point you're going to hit the limits of that you're either going to be looking to assess something that is not doesn't have these sort of traditional measurements you're going to be looking to expand it in a way that's specific for the role that you're hiring for or the class that you're teaching and then you're going to need to get some experts you're going to need to have people who understand the thing that you're looking at better than other people um subject matter experts so if you're teaching science and you want to build an assessment for that get some people who understand science if you're uh looking at cognitive skills you want to get some learning scientists who know how those skills work and the important thing one

thing that I really want to stress is you don't want these people involved just once you want these people involved all the way through the process this is where I'm talking about updating your system and adding New pieces to the system that require you to change the whole thing because you want these people involved from the beginning to the end games are an iterative process uh you're going to find things halfway through that require you to rethink how everything works and you want those people there to be doing that thinking with you um so yeah don't like hire a consultant as a onewe thing where they write up something and throw it over the fence to you and you never hear from them again um it if your goal is to really do assessment or education or anything really well it's not going to work um you can get a really like a simple sprinkling of Education that way but if you really want to be like creating something that has education at the core that the heart of it is education um you need that involvement all the way through beginning to end and so in our case when we're looking at uh applying our assessments to a new role a new job the first thing

we do is spend a lot of time figuring out exactly how that job works when we first got to Roblox we spent months just interviewing people and talking to people and talking to managers and talking to hiring people and talking to people who do the job and like how do you use uh you know creativity in your job what what are the skills that you look for when you're you're promoting someone um you really want to understand what the need is um and not just say oh well you know General intellig or oh this seems like it's probably going to requ this you really want to understand the nuances of how it's being used and so this will give you sort of a a profile of these are the skills that are relevant to this position and these are the skills that I might want to measure and the first thing to do is not go off and make games for all of those some of those things you can assess in traditional normal ways and that's usually going to be a lot cheaper and a lot quicker and more efficient and not scare people so um if you can measure something without a game Do It um where

we usually throw we usually get involved as a game based assessment is when there's something that's important to the job that is hard to measure in other ways so that's where we've identified things like systems thinking and uh creative problem solving and situation awareness as things that a game can assess better than other things and it's worth the effort to go through this whole process and build this whole thing and so we kind of focus in our attention on on those places where it's most necessary and then once we do that we we spend a lot of time thinking that through like exactly how is it used in the role is it used you know on a daily basis is it used occasionally is it used uh in combination with other skills is it used by itself is it used uh you know with time pressure we we really want to know um exactly how we want to build the assessment so that it reflects the job and it it's a a meaningful uh a meaningful assessment of what the job actually requires uh one of the questions we always ask after any test is you know did this

playing this game feel like the things you do at work because we we do a lot of testing with existing employees and that's something that we we look at closely to see if there's that correlation and then the outcome of this is going to be uh evidence statements and these are uh concise you know sentences that say if you do this you are demonstrating the skill um and this is something that isn't you know we didn't create these These are something you can find all over the educational field this these specific examples are from the ngss the Next Generation science standards that I mentioned before um so you want to kind of create something like this based on your need um so the skill that you're looking to measure the job that you're looking to hire for um and make these as specific as possible so this is something I cannot share about our actual assessments so here's like for the for a sense of humor is a sense of humor about can I evaluate whether a joke is good is it about whether I can tell a joke is it about you know can I define why a joke

is funny um these are all different sort of subcategories of the skill and so being able to break down the subcategories and then explain how to demonstrate those so um answering questions might be a way to demonstrate that I understand what is funny and isn't funny um actually performing might be necessary in order to demonstrate that I can tell a joke and get the timing right and all that so breaking down the skill in this level of detail is really important and this is kind of the the goal of this whole phase this whole need finding phase is usually our uh our scientists our academics going off on their own breaking things down in this level of detail and then we get to the point where they have broken it down and they switch it over and pass it off to people like myself the game designers and we look at what can we put on a screen that will evoke this thought so if I know that one of my evidence statements is you know can they tell if a joke is funny what can I put on a screen that

will make the player think about what is a funny joke or not and most importantly click on something this is uh the most important thing because if you're playing the game and you have a thought and you go through the thought process and don't do anything that I can record in Telemetry is useless to me as a assessment I need you to click something and that's really like the most important thing and the bane of my existence uh in my day job is uh getting people to actually click things because when they do then you can actually go off and create scores you can go and say oh this click means you were demonstrating the skill and so that's worth this value the the actual math is much more complicated than this I work with psychometricians who are experts at turning result turning actions into scores and measurements and um they're going to laugh at me putting a plus four up here but the basic idea is that the click equals the score and then throughout the 20 minutes of you playing the game I can add those up I can compute based on all the different actions you took ideally different

actions that are reflecting different uh evidence statements reflecting different aspects of the skill that I'm looking for and aspects of the skill that are relevant to the job and in the case for a hiring assessment so um this is this is the goal of the concept phase and the thing that I do that uh is kind of difficult is figuring out how to get something on the screen that makes the player do the click that is related to the thought and so we'll start just doing a whole bunch of brainstorming we'll get a big list of crazy ideas and you know we'll kind of do our own brainstorming process we'll look at existing games how where have we seen the skill demonstrated in other games where have we seen mini games that that we think reflect the skill if we're looking at a specific skill we'll break down you know what what games uh include a sense of humor what games are relevant to a sense of humor and this will give us some ideas we'll look at you know other ways non-game uses of the skill and see if we can gamify them see if we can turn them into something that works

um we'll look at existing assessments if there are ways that people have measured this skill before can we make that more engaging and more game likee and so we'll come at it from any perspective we can very open broad game brainstorming and then we'll just start pitching ideas so uh you know we we put a joke on the screen and we ask you to measure it and you have to say if it's a good joke or a bad joke uh we give you half of a joke and ask you to finish it we ask you to complete it maybe that's better I don't know a card game where you're building jokes out of combinations of of nouns I don't know um you're walking through a 3D environment and the doors you choose to go through determine what is funny and what isn't um this is probably a terrible idea but um it allowed me to put a picture of a duck in my slides so that's good um you have to build a factory where you're manufacturing jokes I don't know um we just create as many ideas as we can and do a little bit of self-evaluation and come up with a big list um and we're really looking for that uh you know screen to thought to click process so if we can if we think

there's a possibility of that happening it'll be on the list and we'll keep it around and this is where it gets a little tricky because you might think oh games have all these great examples of people demonstrating all these skills but a lot of times people demonstrate skills in ways that don't work for assessment so um if someone's playing chess and the player makes a move they they move a pawn to A4 or whatever um they might be being really thoughtful about that might be a you know skilled chess player who's put a lot of thought in it or maybe they're just moving things at random and they don't know what they're doing um as a assessment I need to know the difference between those two things and like if they are being thoughtful how thoughtful are they being are they predicting moves ahead or they predict how many moves ahead are they thinking because each of these might result in a different action by the player and you know three of these may result in the same action and I don't know which one happened so there's all these different thoughts happening and all these different clicks happening and I need need to be able to break those apart I need to be able to say which corresponds to which so I need to

break it down so there's one thought and one action and this is especially difficult because as a game designer what I have learned sidm likes to talk about a game as an inter series of interesting choices or interesting decisions and that's great advice I I I think he's a smart guy um there's a lot of ways to make a decision interesting one of the most common ways to make a game decision interesting is to give it these layers it's part of the sort of standard game designer toolbox is to say this decision involves five different factors you have to think about the longterm and the short term and you know what you're doing here and like if it's chess I have to think about my defense and my offense and the long-term position of the board and like that's an interesting decision chess is not a bad game if anyone here designed it good job um I I think like that is good game design and that's something that I value and it's not good assessment design because I don't know

which of these thoughts motivated the player's action as an assessment I really need to get that onetoone correspondence and this is true of you know almost any game has these layers has these different factors that go into the decision if I'm deciding where to collect honey in beor simulator there's lots of reasons why I may pick a certain field over another one if I'm mining in Minecraft there's lots of factors in you know why do I want that uh Sapphire block there why am I mining it is it because I want to build something out of sapphire is it because I want to just turn to the right and move through that area it's hard for me as a assessment designer to know what that thought was so again I have to take all these different thoughts and really turn them into one and there's a few different ways I can do this I can break up the action into multiple actions I can say Okay now click here and you know I'm going to I'm going to take this click and break it up into three clicks I'm going to take this Rule and break it up into three rules I'm going to take this action and break

it up um I'm going to take a decision that you were making in our really well-designed card game and say now there's two options instead of one now you have to decide and you have to tell me what you're thinking when you make the choice and sometimes this can kind of work um you could do something where like after the player makes the decision you uh do a follow-up question you prompt them oh you know tell me why you made that decision um as as uh some of you might have noticed this is not good uiux this is not a good process if you ask me after I take an action why I took the action you're breaking my flow you're pulling me out of the game um we often have a battle between uiux and scores in our games um we often have to decide whether to get as many as much information as we can to get the scores or to make a good efficient clean uiux and usually the uiux loses because scores are our first priority um you know we still want to be an entertaining and engaging experience but

if I have to sacrifice a little bit of that in order to get the score I need um we do things where we have information and we have you have to click to get additional information um as a uiux designer that's terrible but then we know whether or not you read that thing we know at least you looked at it we don't really know for sure that you read it but we know at least you had the potential to read it and that's information that we can use in the assessment you can also fix some of this sometimes with design but it's also hard so like if I was uh building Minecraft from the beginning I might say Okay sapphires only used for building it's never used for crafting it's never used for other things the only this block I know when you mine this block you're going to use it for building except you might also just be trying to get through there so I don't really know that so maybe I have to design around that maybe I have to say um you know the spawn logic knows where to put Sapphire so it's not in the player's main path um but you can see see how this starts to change the game um if you're taking a game that already

exists and trying to add uh education or assessment or things like this um it doesn't really work because now I've C changed things to to the extent that those ripples are going to cause me trouble again this is why you want to start from the beginning start with your core Loop and design around uh your assessment or whatever it is you're you're trying to do from the beginning because this is going to start to change everything about Minecraft um and this is going to make a whole different game so I've gone through and I've kind of broken up and I've I've decided that I've got these ideas that have a single thought relating to a single click I've now got this nice big list of games that I can start to think about and I can start to evaluate and so we go through a process of sort of evaluation here and the first um danger sign is if it doesn't have good scoring potential that's something that we you know always talk about a lot and we we evaluate all of our Concepts first according to this and we look at our evidence statements

we see which games have the best you know thought to click ratio not exactly measuring it like that but yes we basically are looking and then that allows us to trim down the list pretty quickly because there's going to be things that are better and worse um and then the ones that are left we can start to look at additional criteria um the ones in the middle are kind of traditional game development criteria that we look at is is this game going to be really hard to build is this game going to be you know difficult for accessibility purposes is it does it feel like the right sort of game we we we know that our games are often um the first game that someone plays as an applicant to Roblox and so they're really thinking about Roblox and so we want them to reflect the values of Roblox well and so that's something we look at and so some games we can trim based on these but then the last two are actually really important and they're more specific to assessments so it's really important that our assessments are not biased that no one group of any sort has an advantage in our assessments and I I would not claim that we can remove all

bias bias is a really difficult thing to uh pull out but what we're looking for our Target is that it's fair that there's no group that is is performing better than other groups and so we check our data meticulously constantly before we release once we release after we release and if we see any signs of any um irregularities any group is seems to be doing a little better we immediately take some action and do things we just were testing out a new idea recently we had some for this and so we went back and redesigned it and now it's doing a lot better so this is something we take really seriously as a game-based assessment um we also have to worry about gamer bias because we don't want people who are used to playing games to have a a preferred status in our assessment we don't want people who aren't familiar with tradition traditional game mechanics to not get a job so you know that that idea I had about going through doors probably we'd have to reject because that's a popular game on Roblox and if you've played that game you're probably going to be better than someone

who isn't but even a lot of sort of traditional tools in the game designers toolbox we have to be really careful about we we try not to use avatars because Avatar movement Avatar control is something some people are more familiar with than others we don't want someone who isn't familiar with that to be uh disadvantaged in our assessments and you know Gamers think that like wasd controls are easy and understandable and they're not like a lot of people who aren't used to those are going to have just navigating through our worlds so we tend to do more UI based uh interfaces and UI based navigation um you know we we built an assessment based on uh uh modifying a factory um and so we looked at games that use factories and made sure that we weren't doing the same sort of thing we weren't giving you the same abilities we weren't giving you the same goals uh so even if there's some similarities we're confident that they're not uh ADV giving an advantage to people who are really good at factorio other than you're you're probably pretty smart smart if you're really good at factorio that's a really hard game but this means as a game designer I

have to like set aside a lot of my traditional tools I have to uh look at a problem and come up with a way of solving that problem that makes sense that is logical but isn't something that is something that that is in so many games that people are going to have an advantage doing that and then finally the last big factor that we evaluate our Concepts against is uh the ability to build variations and variations are really to us because we have to worry about cheating we have to worry about people trying to gain an advantage in our assessments um based on things other than the actual skill and this may sound kind of silly and paranoid but like when we were doing this back with McKenzie there's a bunch of websites where people were trying to tell you how to beat our assessment um and you know we were just one company and it was still it's still a thing um and so you know in the traditional uh testing world and the world of the SAT and all that it's a big problem it's a big uh thing and so you know at some point we hope to be that big and uh we want to be ready for that

and have the tools in the game already to prevent that sort of cheating and the cheating can come in a lot of different forms there's like the direct cheating of like the answer to question seven is B um this is usually pretty easy to just create a few different forms and then um if I tell you the answer it might not be the answer that you get um but sometimes you can also give guidelines give heris STS give strategies and this is something we kind of worry about and we're we try to be careful with that the strategies that I can communicate to you are not going to give you a huge advantage and really if we can make those strategies um more and more complicated we can get to the point where the strategy that I'm telling you is basically the skill that I'm looking for so the the the goal is always that the way to cheat at our game is to learn the skill and we're fine with that like if that's how you cheating good job um and so this is a problem with traditional assessments um they create different forms every every year and change it year-over-year and you know

they you might get a slightly different question a slightly different phrasing of the same question um and this is something we do as well so we create different forms for our assessments and give people slightly different versions um to make it harder to do the cheating especially the first type of cheating but you have to be careful with this because you don't want one person to get an addition problem and one person to get Vector math um and so you have to create a lot of forms and you have to make sure that they're all equivalent that they're all the same and normally as a game designer if I'm making a whole bunch of different things to put in front of people um I'm going to randomize it I'm gonna have uh you know roll the dice it's fun your result is a little different the enemy conflict the set of enemies you fight might be a little different their health might be a little different but this is another tool in my normal tool box that I have to throw away because we need really close equivalence we need you know everyone who is getting an assessment to have the same Fair advance the same fair chance of success and if there's even a slight slightly harder version or a

slightly easier version then that person is not going to have the same experience they're not going to have the same opportunity to show their skill so you know Randomness like real Randomness where there's variation in the outcomes is just is something that we have to throw away um equivalence is totally important because we want everyone to run the same race we don't want one person to have Hills and one person to have curves and one person to have a straight line um so to do that we have to have a a concept that is going to support a lot of variations ideally variations that are easy to generate um we recently had an assessment that we thought was going to be easy to generate and it turned out that um we needed to do a lot more animation than we thought and so uh apologies to Dustin who is our artist who I made added a lot of work to because we couldn't figure that out okay so all of these criteria are used to thin narrow down our set of ideas and the goal is to get down to maybe two or three ideas that we think have a good chance of uh measuring the thing that we want to measure and then we move into

prototyping and we start looking for the clicks that we are are going to assess based on and really what we want to be doing is watching the thought process to make sure that that's happening um but there's no good way to do that um so the closest we can get and the best practice we have is listening so rather than trying to read their minds we ask them to tell us what is going on in their minds and we listen to that so we spend a lot of time doing thinkal louds where we'll put the game in front of people and we'll you know be on Zoom listening to them and we'll ask them just tell us everything you're thinking tell us what you're looking at tell us what um why you did the thing you did tell us what's what's going on in your head and so we can get them to start verbalizing and we can see if those thoughts are occurring ideally we'll get them to verbalize even their deeper strategies and the sort of undercurrent of why they did the thing they did um and this is great because it allows us to verify that the thought is happening and then we can see if the click happens in correlation with the thought and that's really the heart and

soul of our assessment games but at this point we're too early um we don't have the game yet so we can't put something in front of people so we take the ideas that we still like and we start prototyping and this is something that like I said before I've used in all sorts of entertainment games um but here our questions are a little different um we're trying to just get at is the thinking occurring and um a big advocate for prototypes as throwaway work for prototypes as getting to a an answer as quickly as possible in whatever means possible so we will literally do paper prototyping on pieces of paper um we will do uh prototyping in Google Slides where we have people dragging things around and uh you know simulating the actions they're going to take uh with crudely drawn shapes um minations is a tool I've used um if you're not familiar with it it's a really nice like game economy simulator that uh we've used to prototype uh some more complex systems

um Roblox is actually a great Tool uh we built spaces in roblocks and put it in front of people to see how they would react and one of my my favorite ways to do this is with a little bit of puppeteering so we'll put something up on the cloud and have the uh player participating in it and then when they want to do something they'll verbalize that which is a great way to get them to verbalize and then have the designer in the same uh application in the document in the cloud make the change as if they were you know operating under the game logic and so it's a way of pretending like you have game logic before you've written any code and it's a way to kind of give the player the experience they're going to have and see if the thinking is occurring and if the actions are occurring um really really early which is uh really important for us because that allows us to kind of identify if the thinking is occurring and if the clicks are corresponding to the thinking so we use prototypes to really narrow down to one idea and once we do that then we start Game Dev development we start moving into the actual development phase where we're getting stuff

done and this for the most part is traditional game development so not easy um still challenging but uh can use a lot of the sort of standard tools and standard practices um we still do a lot of think alouds we're doing constant play tests with people to verify that the thinking is still occurring the way we want even a small change in the UI sometimes can change the way that someone is thinking and the way the way that they're reacting to things so we are basically you know keeping everyone involved all the way through and involved in discussion every week we're looking at new think alouds new play tests with new features and we're having a discussion with that full team about what we're seeing and what we're how we think we can improve Things based on what we've seen um and this is really that that system that I was talking about before in action one thing that we Implement early on in the development phase that is an special problem for us so I want to call it out is the tutorial um tutorials are hard um game game development in general

has gotten better at tutorials over the years there's some best practices there's some good GDC talks um and a lot of what I would distill that down to is learning that tutorials are not separate from the game so it's not that you do the tutorial and then you play the game it's that you need to integrate them you need to have the player constantly learning new things and playing around with those things and then assessing how good they are bouncing back and forth and different players can take different paths through this um and that's a nice tool in the game designer toolbox that once again I don't have access to because in an assessment we really need to make sure that everyone starts at the same place that everyone has the same information before we assess anything and then we start assessing things and we don't change it as as they're once they've started assessing so we really need a clear line between tutorial instructions and the test you don't you know learn halfway through the SAT that suddenly everything is not what you thought it was and uh you have to approach things differently um because

we really need that to be a fair measurement so we need this clear line um between tutorial and assessment and this is something we've been experimenting with and trying to get the right balance on because as a game designer it's really important to me that the player has a chance to play with the system a little bit before they're assessed and so we don't want to just have a bunch of you know text explaining the rules and then throw you in the game with no practice with the UI with no practice with anything so we've tried a number of solutions this is something that in my mind is a problem we're still working on um we're every new assessment we're we're trying we're making some improvements to the tutorial some of those work some of those don't so this is something that um is an ongoing struggle for us to let the player kind of play with the concept but still keep it fair so that everyone has the same chance of success okay so at this point we're back in a normal game development process we're we're writing code we're testing it we're doing the think alouds um and

then at some point we're kind of done building the game and now we're just iterating we're constantly making changes um I call this section out just because it's been really important for us to remember uh as game developers that you know iteration is usually takes as long as development like getting the game right is often harder than just getting it working um and so this is normal game development but I still want to call it out and then we' go through a normal QA process we wrap things up we get things ready to ship but an arc shipping is still testing sh we start with uh pilot testing where we're putting the game out maybe to a few hundred people and really at this point we're just checking to make sure things work we're looking for any sort of red flags any bad signs in the data we're making sure that the clicks that we expect are happening and happening in the right way um we're looking that the the things that are happening on the screen are getting the result we want that the scores are happening and that you know we're not seeing any weird biases um we're all the things that I've

talked about so far are happening and that's really the the pilot test and then the field test is basically a bigger version of that where we're actually now looking at the scores and are the scores doing the things we want are is there any bias on any one individual score is there any bias across the whole thing is the are the the scores actually resolving in a way that we can definitively say something about the skill and there's usually this is uh where the psychometricians get very involved and do their magic in ways that I don't fully understand that they're very smart people you might think okay at this point you've got all this Telemetry you know every click they've done you know where they've moved the mouse you know what they've hovered over what they've looked at um why don't you just look at everything and just extrapolate out what works and what doesn't like you know throw it through some AI see like you know if they they move the mouse to the left here that means they're going to score well if they click this three times that's going to mean that they're going to score poorly um we've tried this it doesn't work um the scores that we've created that we've crafted that we've thought about those do pretty well

the scores that we kind of retroactively try to apply it it it just doesn't work there's not a good correlation um we're going to keep looking because maybe at some point there will be something that will help us realize something we didn't think about but um just using sort of like big data processing methods it's it's not very effective okay so once we've done all this we then go operational which means the game is out there the game is being used people are uh actually being assessed and actually having a result based on it it doesn't mean we stop we're still checking the data we're still looking at things we're still creating new forms we're creating new uh versions of the same assessment so you know in some ways we're alive Service uh to use the game development term but at this point it's actually being used and we've now finished that assessment we're very happy so um to wrap up and summarize first the most important thing about an assessment game is the assessment you need to get scores you need to get clicks that relate to thoughts that happen because of what

you've done in the game um everything else is secondary um and to do that you need to have a team that is built around your experts you need to integrate and build a whole new system a whole new set of Gears that have everyone uh from beginning to end which means you have to throw out a lot of your traditional tools as a game designer um this may sound sad for people like me but um honestly it's part of what I like about this this is it's fun for me to have new challenges I've done this for a number of years I like uh solving problems that I haven't solved before and to do this you need to look at how people are responding to your game so you need to put it in front of people ask them what they're thinking um it's the only way to know that your game is actually working and then finally tutorials especially hard think about your tutorial be careful with your tutorial make your tutorial early and often so that is assessment games uh thank you for coming uh I work on the Education team like I said I also teach

at uh the local Community College um these are all things I'm excited by if you see me around the rest of the talk please accost me and talk about these things and I think we have a few moment a few minutes for questions so if anyone has any questions about anything I said please go up to the uh microphones and talk and then I will go to the wrapup room after this if you don't get your question in thank you hi hi yeah you mentioned briefly some metrics that you guys are using to like track the progress and track the success of this program you guys are employing I guess like what is your key metric to determine whether this is a successful thing to roll out to Roblox and companies beyond that as well I to be clear our goal isn't to roll this out to companies outside of Roblox our goal is just to use it in Roblox um reliability is one of our key metrics that's one that we look at really closely um we just look at score performance like each score we're looking at is this does this have a good distribution um we want

scores that are are tell us something meaningful about the range of candidates we get um we are starting to look at now that we've been at ROBLOX for a few years we're starting to look at long-term uh success rates so people who have gone through and gotten a job uh are they you know performing well um there's a lot of different ways we can measure that um and honestly uh the nuances are going to be up to the scientist that I am not an expert on so what I've just said is about as much as I can say about that thank you hi hi um for assessment games do you especially care about making sure that your play testers are fresh and if so how do you find fresh play testers yeah so a lot of times what we do is we're using uh recent hires so we use people who are in the role that would have gone through the assessment uh recently because we can usually get a Goodwill from uh people inside the company um and yes we need them to be fresh we don't want

people who have seen that same assessment going through it multiple times because yeah they're going to have uh I mean ideally they're not going to have a huge Advantage because our assessment should be based on the underlying skill not on learned uh qualities of the individual assessment but generally I mean roblo is a big company we have thousands a pool of thousands of people to pull from so we can usually get enough people from that um back when we were a smaller company we would uh go through a little bit more effort to get in uh people that were close to the target audience because it is really important that the uh the testing pool match the uh candidate pool so does that answer your question yeah thanks okay thank you hi hi um what made you choose contexts that are sort of um not as directly related to the like it's it's not a sit at your desk at ROBLOX simulator that you've got like a factory some kind of robot thing where you do

need to tutorial eyesee them how did you choose where to go on that spectrum of like most like their job to most Fantastical yeah we've had a lot of discussion around that we want something that we feel has the right level of sort of near transfer where it's similar to what you do at the job but not exactly what you do at the job but we also need a a a metaphor a theme that doesn't have people a lot of existing knowledge that doesn't have a lot of bias towards oh you know we've used uh the natural world before because that's something a lot of people have a small amount of familiarity with and are comfortable with uh at ROBLOX we're using a little bit more industrial style things that feel kind of you know Hightech and that's kind of the feel that Roblox has but um mostly we're looking at things where there's not going to be that that bias and we're not going to have um pre-existing expectations we actually when we were doing the natural world we had one biology student take it m and uh they did terrible because they kept expecting things to match reality and that caused trouble so that's part of

why we I think we moved away from that as well is we're looking at things where you can't come in and already know think you already know the answer so yeah thanks thank you hi hi how do you manage retesting in the case someone applies six months later or um we do identify that and we try to give them at least a different form um in some cases where we appropriate we would like to give them a different game that measures the same thing um which uh right now we don't have uh a full Suite of games ready but we're working towards that thank you so yeah but at minimum we try to make sure they're not going to have exactly the same answers so hi hi uh what's the role of sound in an assessment game is it the same as traditional like providing information feedback setting mood or is do you just try to leave it out entirely so far we have left it out okay uh for the last uh couple years I've been trying to get it in and I have failed so

far so um it's usually on our list of things we want to do but on the lower side of the list and then we don't get quite the because honestly one thing long term I think it's going to be important for accessibility as well so I think having the ability to have the game uh narrate to you for people who have visual challenges um that's something that we definitely are cognizant of and working towards but uh we have not yet gotten partially too because we don't we don't want to assume your technology we don't want to assume that you have good speakers and audio and we want to make sure because of the the fairness quality of that going to be hard for us to ensure so we want to be really careful when we do uh add it makes sense thank you thank you uh how do you deal with subjective kind of assessment like you said you know you're ultimately prioritizing things that are scores does that mean it is it has to be number um how do you kind of confront kind of that boundary of soft skills um let's see if I understand the

question correctly I think um we have to boil everything down to a number in some way because that's kind of like our final output is going to be uh numerical so we do we can batch things in different ways so we can look at like different qualities of the action that you're taking uh if there's more nuanced actions than just a single click that corresponds to a single number we don't have to have everything be a single direct correlation we can build more complex relationships if we need to um it also means that there are some skills that are challenging um we've spent some time thinking about how we would do a communication assessment and it's tough um it's something we'd like to try at some point because we like tough problems but um there are skills that are inherently more difficult because of that right I I would I asked this partly because because last year I was working on a on a game that was about assessing creativity and I was doing all that work

to try and help deconstruct the into the points you're talking about yeah but finding kind of a threshold where we either need an expert assessing the answers yeah or we need you know massive numbers um to really like spread it across and I was kind of struggling with that with that challenge of do you have experts who are coming in to look at the score as well like do you how many scores does a game have within it is it just one score um so a couple of nuances there you can't have experts generate scores for you it's just complicated and expensive so um we've we've thought about that in some cases but usually at roblo we're looking at you know thousands and thousands of applicants so it's not very practical to have people look at each but for like for the SAT for the writing sections they do that um in terms of uh like number of scores uh I probably can't get too specific but you know

we're looking at uh at least dozens of scores on any assessment when we're asking you to spend you know 20 or 30 minutes and generally we're we're targeting a much higher number with the expectation that a lot of those are going to not work out the way we expected so we generally you know start with a very as large as we can really and then assume that some are just not going to work and some are going to collapse into the same score basically where you know these are directly correlated so we really can't treat them as two scores they have they have to effectively act as one score um and then at the end we boil all that down so far we've generally been boiling that down to a binary decision because we're doing a hiring process um we have investigated more nuanced um output feedback to provide to the hiring person or to do for uh for uh other forms of of job evaluation um it's challeng in because you know we we're always pressured to keep them short as possible because people don't want to spend hours uh you

know doing things when they're applying for a job um but that that limits how many scores we can get yeah thank you hi are we out okay um I will go to the other room if people want to keep talking I'm happy to keep talking and if you see me around the hall please harass me happy to keep talking thank you [Music]