Improving Critique of Game Projects with Expert and Peer Feedback

[Music] thank you my name is Marta Pema this is Jessica hammer we've split this talk into two parts and I'm going to talk about expert feedback and Tracy Chi is later going to talk about peer feedback I was asked to kindly ask you to turn off your mobile phones and any other noise making devices and I would also like to ask you to fill out the evaluation after this talk if it's not in your inbox it's in your spam folder and it's a way of criticizing us which is kind of making everything into a nice circle I will spend the next slides introducing myself so I am a person that verse a bit too many heads at power at times I have an I'm a co-founder of a small indie studio called broken rules that has recently made the game called old man's journey has anyone heard of ole miss journey yes actually quite

successful game and we're very proud of it but I wasn't very involved in the development of that game because I'm in the moment heading a games program at the T University in Copenhagen which is one of the I think oldest games programs on the university level in Europe at least here on this image you see students working on a game that is actually nominated for or it's not nominated got an honorable mention in the ITF student category which makes me very proud and I'm also teaching mostly a project-based classes they're actually so whenever there is a class where students make an actual game this is what I am teaching and this is also why I came to the University with an industry background when I'm not doing that in my in my abundant a spare time I'm actually trying to push games a little bit forward with the Covenant game collective very Co organize events and/or help events to be more interesting or just weird games projects

now and then and try to just break games until we can reassemble them to be something new but let's talk about critique in design education I guess most of you are educators or somehow as educators interested in the critique process so I'm not going to ask to answer the question why it is important you all know why it is important I'm just going to frame it a little bit with design philosophy and one day I will submit the design philosophy talk to 3dc because I think it's actually a super interesting topic so my understanding of design as a process is that you you work on something and while working on it you realize what you're actually working on this wonderful illustration is based on the PhD dissertation of Henry kindred and he bases it on John Dewey's the pragmatist philosophers notion of doing for the sake of knowing so you execute an action in order to learn something about something and that is of course also the problem with design process is

it generates they generate new problems while you're working on them and critique is one of the ways to constrain the design space to help staying on focus and to serve many other purposes in the design process so then one of the problems or one additional problem with design processes is that in hindsight they look very linear but they are not while you're in them so this is the development of the player character in a different stages of the game and the other moves it was made by protein rules which is actually an interesting story because I was one of the supervisors of the team when they were still students and later from the company with them or they formed it alone and then I joined up with them and it looks like a linear development but there were so many stages of errors and dead ends until we arrived at this final character that the design process more resembled this image by then cook from spry Fox who has an

interesting article I associated to that image his blog is anyway fantastic and so in the design process you actually expand the scope of what you're working on and then you contract again and then you expand again and then you contract again and in the purpose of that is of course that you accumulate more and more what he describes here as good stuff now I've never seen any product that ends up with only stuff in it but so it's a bit romantic maybe this image but in general this is the iteration the iterative process you're working in and critique comes in at several places so while you are expanding one of the purposes of critique of criticizing the designer that is at work are the designers is to give them new perspectives for example because you bring an outside view to the table then but I do a lot actually because students are at university and they never get the chance to play in such a safe environment again and they are out there in the real world they

cannot experiment that easily so I try to push them a little bit at least outside their comfort zone because I think that is actually very enriching for them and and is one of the purposes of Education and of course by I just having more experience than them I can also contextualize or help them to contextualize their work and to learn about what in other environment they are working then value while their design process is contracting you can help with critique to get students into a position where they have an easier time making decisions design decisions they learn things like throwing away stuff that they have worked on which is one of the most important lessons you can learn in design and you can support that by criticizing them and of course coping and framing and just generally keeping the direction is a super important aspect of the design process that you can support by criticizing the student and finally of course you also have to

motivate them now and then because as we all know projects take a long time and you lose the motivation when you have a week where nothing happens and then critique can also be something purely positive or mostly positive that was motivating students and then one of the final purposes of critique is that also it enables students to reflect on their design process and to understand how design processes work in general for example design processes are messy as I said before you have to throw away things that you have invested time and energy into you go off on a tangent and and then I have to return to the main road that you were actually driving on and so on it is never a very linear or really only in a very boring case is excessive is it a very linear process and that's completely normal and something that students have to learn and realizing that leads to a process of reflection that makes them more aware of their own practice turning him into a reflective

practitioner as a term that was introduced but not trained by the way in his fantastic book and finally of course just by presenting their designs over and over again and in unfinished states and being aware of in what stage of design they are at any given time they get better at in their communication especially if you make it very if you consciously this frame how they are presenting their works and that is a very important skill to learn and there are different ways of talking about of setting up a system of critique so on one hand you have P critique which is Jessica is going to talk about in different forms then you have plenary critique which is a mixture of peer critique and expert feedback so if your student is presenting to the class and gets feedback from you maybe the TAS and other students then suddenly you have a mixed mode of critique and then of

course there is just one on one or one on n supervision and there is creating which is another very distinct case in that it has well of course students can I guess in your universities to take and file complaint so it's not like rating is only in one direction but in 99% of the cases it's a one directional process and it's a very the bluntest form of critique I can imagine actually I could give a whole talk about criticizing grading but it's something we have to do still and it's still a form of feedback a form of critique for the students work so yes to summarize in the beginning you support the student to open up to the science base then during development you support that it gets becomes more focused and after the project you actually support the reflection on the process by the student now in my next slides I think I'm made too fast in my next slide I will actually go and go through a couple of challenges that we all regularly face in the critique in

the process of critique and offer some kinds of solutions to them but the important thing is while I try to make when I try to make a mapping between one challenge we face and offer one advice of how to deal with that in the end of course you should follow all their advices or at least come up with your own reflective process of criticizing students because they they don't work in isolation and some of them are actually the answer to a couple of challenges and others are actually not just a part of the answer to the challenge I'm presenting so it's not that as straightforward as I as it might look on the first glance because slides make it simpler make it appear simpler and I will use some anecdotes from my own practice in order to underline what projects I was facing so this is a play on on battlegrounds I guess held a couple of people have played it I've heard it something like thirty million or a crazy number and it came out about

a year after a group of students I was supervising was presenting their idea on how to fix the problem that games have different flavor multiplayer games play very differently depending on the amount of players in the level and their solution was actually the opposite of player on battlegrounds and the solution was to expand the level the more people were trying the server and they would write procedural animation algorithms to actually expand in a useful way which is technically a very interesting challenge but it is just not solving their initial design problem so their problem of the game not getting or the game they are having to adapt to the number of players in order to stay interesting was exactly not soft but an approach and I had I somehow completely failed at keeping them focused on the problem they were actually trying to solve I had I

was facing challenges and one of the challenges was they had different goals to me they actually I thought their goal should be to solve the problem of scaling multiplayer games but they thought they had actually come up with a particular solution for a particular use case and that has to be great it wasn't and I like I would have to criticize myself for that but it is one case of students that are just that were just not receptive to what I was telling them but other case other challenges we are facing our for example students that are mini maxing or a Maxine meaning their efforts so they try to get the maximum return on their investment with minimal investment of course or they try to just get through with the minimum effort I guess you all had students like that and it's not easy to work with them what works for me in any case but a particular in that case is actually

getting them truly engaged by asking them questions about what they're working on in order for them to not be satisfied with just delivering or not or maybe they are not even able to find out what they have to deliver to you so they cannot just sneak through with minimal effort if you do not give them the opportunity to find out what what the power is they have to over but instead ask them to get thinking on their own then for students that haven't done anything they think there is one thing of course as I said before design process is a messy sometimes actually in having not done anything has very good reasons you need a week to just a week to pay you cannot come up for use now I don't have an have a week for you just cannot come up with any new ideas it's a very normal thing in a design process and your criticism of student projects should reflect the fact that sometimes doing nothing is a very productive thing

and sometimes not coming up with the correct solution is still a part of a process that can lead to the right outcome so you'll critique your feedback to any work this turns to should kind of match the development stage or the design stage that they are in in any case with the students that worked on this entire player Nobel grounds idea I think the mistake I made is that we didn't talk about the core of their problem explicitly enough so just drilling down to what is the problem they are actually trying to solve it was just it did not happen because I was not reflective enough in the process of criticizing them by myself so I should have actually asked them the right questions that would force them to actually confront themselves is what the problem they were actually trying to solve well we have evaluation systems that n mark is huge winnable evaluating everything especially education and of course

students are allowed to give feedback after the process so in this case after they had finished the master thesis they filled out in a relation form and one of the feedbacks I got from them was that I kept telling them the same thing over and over again and what I told them was that if they want to solve the problem of an expanding level what they should do is make a super simple 2d prototype with barely any gameplay was just the bare necessities and just try out if their solution is actually the solution to their problem they didn't instead they started with 3d and had all the problems the 3d games bring with them despite the fact that there are a lot of well-known big triple-a productions out there has a lot of small games they started out a simple 2d prototypes so obviously I was not very effective in communicating to those students one of the things was that it was very obvious that their team leader took any criticism that I presented extremely personal which maybe I could have

mitigated by separating much clearer between criticizing the product the process and the designer I do not know if it is even a good idea to ever criticize the designer their students but what I do know is that in that case I should have criticized the process most because the product was something that was so they had invested so much it was so dear to them that it was like criticizing the designer but the process was something that we actually could have agreed on together and that we could have fixed in order to fix the product also so I was kind of targeting the wrong I was just targeting the wrong thing in that case and that made communication very difficult and then if students a lot of students so we as I said and my universities in Denmark you're an English language program we have students from I don't know how many countries and most of them are not English native speakers that's a huge exception and it's somehow sometimes

they are very bad at articulating what they're doing and you have to have a lot of patience with them sometimes I am bad at articulating my critique but what I have learned is that it is very hard to to formulate critique so that the positive and the negative comes an equal fooling because it's not about not only about what they say it's more about how it is received actually which is difficult rhetorical exercise and finally a very simple trick if students are easily the motivated or also if they're easily offended is just saying something nice first or maybe even send Wittering the critical part those things work let's quickly talk about an example of a creative process of critique was actually more fruitful and it moves I mentioned it before the game done by broken rules as their first game while I was supervising them and data later trying to deser as a partner in the company they had one issue where they wanted to make a mouse based mouse and keyboard controls whereas I wanted them

to make only keyboard controls because they made a platformer a platformer should be played with a gamepad easiest equivalent that you have on a PC is the most straightforward equivalent is the keyboard and not the mouse well um in a way I was very right I was very clear and told them in a very concrete way that they just should do that instead of having the mouse because it will be easier for people to play funnily later they reintroduce the mouse controls for when they ported the game to the and suddenly their old control scheme was actually exactly what they needed for that port but in that case I could convince them because I just demonstrated to them that this is the established practice for platformers and they believed me on them believe me they believed the numbers they believed the experience out there then a trick if you if you have a tricky that you can always take if you're criticizing students is

just taking a step back and kind of aligning yourself with the student on a metal level I sometimes do that even when I'm grading that I actually agree if have very different projects in one class and have to create them according to the same standards then my standards are I agree on individual goals per project with the student in an explicit session supervision session where we agree on that so aligning goals can be something that is done in a very explicit way and then finally it should all fit based on moles and Gardens we will to sense and friends especially Danish students seem to be very fond of that they like the formal structures of courses to be very clear at the same time of course you lose a bit of flexibility that way so it's not something about universally recommend but in the end the process that leads to your that that encompasses your critique should at least be clearly communicated to the student well I always said that I

have a very distinct relationship to the game and the other knows because I supervise it and then later I became financially invested in it in a way so the fourth challenge in critique is of course yourself in this case it was so that I actually got sold involved in that project that it was very clear that criticizing the game was criticizing my own supervision practice it was criticizing myself which is something that can happen very easily and the only solution that I actually have is that I'm neither is very conscious about that and that I'm very that I'm very actively trying to give students a lot of freedom in order to prevent that I get too attached to any project and because I as a teacher I think I shouldn't claim too much authorship over it it bent okay with moves but that is I think not the normal case so it's difficult unless you explicitly set up a project to be a collaboration with students of course but that was not the idea back then it should have been their game maybe even more and finally I'm standing here telling

you how to criticize student projects and at the same time I have a huge program impostor syndrome I feel like Who am I to tell you anything and I don't have a clear solution for that I'm pretty sure that they were enough talks about that there have been some at GDC and I did a quick Twitter survey to find out if xkcd on GDC slides is okay and was like 84% agreed that it is actually totally fine to have an xkcd joke to end your slides but there is no solution for for imposter syndrome the only solution is well obviously I have been teaching and supervising game projects for must be 18 years or something like that so I guess I have learned something along the way I just have to trust that and you have to trust that too okay so with this we have covered at least some aspects of the expert based on critique and I will hand over to Jessica who will talk about

peer feedback hi so um I'm Jessica hammer and I just want to say that one great approach for dealing with impostor syndrome around providing feedback to students is not to do it make students do it instead and that's one of the hidden benefits of peer feedback in the game design classroom just for context about Who I am I'm an assistant professor jointly appointed in the entertainment Technology Center and the human-computer action interaction Institute in Carnegie Mellon University and I'm also a game designer and what I want to talk about today is these two formats for critique that involve students providing feedback on each other's projects and I want to start by talking about why peer feedback is valuable the value of peer feedback it's actually different from the value of expert or instructor-led feedback and I'll say I'm framing this a student and

instructor but this can also work in peer designer situations but the student or the peer has actually two roles in a peer feedback situation they're receiving feedback from peers on their project in their role as a game designer they're also providing feedback and of course there's also a third stakeholder who is the instructor right you're in the room while students are giving peer feedback and so feedback receivers when students are in this role is feedback receivers peer feedback actually helps students improve their self-assessment skills because students are much more likely to be critical of feedback that comes from someone they don't see us oh the great expert is giving me feedback it's oh yeah what do I really think about this it's easier to understand because there's less of an expertise gap the peer is more likely to be closer to their level of understanding and use a similar intellectual framework and

allowing peers to give feedback there are many peers there is only one of you and so feedback can be diverse and copious and timely as opposed to students sort of waiting in line until you're ready to get to their project as feedback providers peer feedback trained students in the norms values and practices of your discipline so it actually helps students recognize high quality work and internalize standards because they have to practice taking on this role of someone who is the evaluator or the feedback provider they're not just watching you do it they have to do it themselves and finally a peer feedback is really valuable for you as an instructor because it exposes student reasoning helps you see where students are at in that process of internalizing norms values and perspectives and it can also help a critique or feedback process scale to larger classrooms right so an

intimate studio class might be like 12 people with peer feedback you can actually have seen peer feedback work in classrooms up to 200 students which is quite a remarkable shift in scale but peer feedback also has some challenges around engagement quality and reflection so around engagement you sometimes see students check out of the peer feedback process interestingly students also get over invested in the peer feedback process so going to sleep whatever I'm just not going to do this peer feedback thing that's one sort of engagement related failure state but students can also get hung up on arguing with each other in the peer feedback scenario rather than paying attention to the thing that they're supposed to be responding to right oh I think you should make the character larger no I think you should make the character

smaller boom the students are off and the designers are not getting useful feedback quality of peer feedback is also an issue and actually this has less to do with students not being expert game designers and more with students being very reluctant to be critical so we've analyzed samples of thousands of peer feedback comments and very very very few of them are willing to directly criticize the work at hand instead what students will often do is jump directly to advice without providing any justification right here's what I think you should do why should you do it because I feel that way right and often couched in these ways of like they won't say anything negative but they'll give you some advice about how to change your project but they won't tell you why right this is actually extremely unhelpful for designers to use to iterate their games and then finally there are issues with reflection right so when you get a pile of feedback what do you do with it how do you use it to iterate your game well

one of the things you need to do is actually think and talk and understand the feedback that you're getting and there are some problems in this and this say this is actually not unique to peer feedback we see this with expert feedback to students will over value positive comments and undervalue the few critical comments that they get they will sometimes use feedback to justify exactly what it is they wanted to do magic how often feedback lines up with exactly the thing that they were already working on or they'll treat feedback as a checklist they won't evaluate it critically they'll just say oh well okay I guess you know 10 people said we needed to do X so let's do X right so there are a number of failure states around how student teams make use of feedback and integrate it into the design process and so what one of the things that we work on in my lab our methods for improving peer feedback and when I say methods I actually do mean sort of replicable methods that you can

teach or systems that we embody with technology because these students have to do it you can't do it they have to do it right so this has to be a designed approach where you're training students or supporting or scaffolding them in some way because it's them not you and like Martin uh we frame peer feedback as an end-to-end process so if you think about these formats for critique right so we're doing let's say a live in class play test and providing peer feedback or we're doing student presentations with a plenary critique well there's actually things that happen in the class before peer feedback happens right so there's this preparing for feedback and there's also things that happen after feedback like as I've alluded to reflecting on that feedback and integrated to the into the design process so you can really think about these issues of engagement quality and

reflection as they play out across these three phases of the peer feedback process so what I want to talk about today is to sort of contrasting cases to different methods that we've developed and have deployed in real classrooms and have collected data on to understand how they help students become better peer feedback providers and receivers and how it can help instructors in the game design education process the first of these is something I called Yoda method and this is really meant for live in class play tests so you can see over here on this diagram we've got like some faculty and some designers they're standing off to the side the designers are silent they are not allowed to talk they must listen and take notes and then you have some play testers who are peers right other students in the class and some observers and before this thing this thing this play test live play test

ever happens everybody right so feedback designers feedback receivers but also the feedback providers get something that I call norm setting around how P P R feedback goes one of the ways that I do this is with an exercise that I like to call five spoons in five spoons I divide the whole class into groups every group gets five spoons and I give them 15 minutes to make a game these games are not very good and that is intentional because making a game that there is no reasonable expectation could ever have been a successful game takes their ego out of the process all of these games are absolutely ridiculous right and the spoons are meant to also help with this silly principal will balance them on their nose is or you know if you them to flick things across the room and it's meant to help them say all right I made a game there's a ton to criticize about this game and I use this then to demonstrate high

quality feedback low quality feedback and ask them to participate in a peer feedback process in a situation where their ego is out of play and this is really successful at norming that getting peer feedback on something in progress is okay it's okay to show work that's half-finished and kind of crappy and that if you are getting critical comments that help you improve your game that is done out of love and to help you learn how to be a better game designer right so five spoons during peer feedback right so during these live in class play tests on games to which these students actually do have some attachment they'll have been working on them for several weeks we use Yoda which is the method the piece of this that the method is named for and that stands for experiences observations theories and advice so basically we break down what kinds of feedback who is

allowed to give and when first the players are allowed to give feedback about their experiences and only their experiences what did I feel what did I experience what did I do next players and peers are allowed to talk about their observations and observations must be concrete right when this person got the water card I saw them smile right then and only when we've gotten a whole bunch of observations our students allowed to create theories why do these things happen and what they can then do is link experiences and observations and provide explanations finally the final phase is advice so both playtesters and peers can say okay well based on these theories here's what I think you should do slowing it down and breaking it down like this addresses a number of problems in the peer feedback process it diversifies participation and perspectives because

people can't just jump to here's what I think you should do slowing it down everyone has eyes and that gives everybody a chance to say something or to offer observations especially in the first parts of the process having people think through sort of some data sources like experiences and observations increases the provision of current feedback that is justified and critical and it means the teams aren't dependent on that was some great advice we got even if the advice they get is not terrific they can actually use these lower-level experiences observations and theories to make good decisions about their design finally after feedback students are required to create a process document designers write people who've gotten feedback have to create a process document which can be worth up to a third of their brain and that's because when you design a game the artifact itself right it's like you are here but

it doesn't show the complex path that you took to get there and especially it doesn't necessarily illuminate the kinds of critical decisions you made your ability to to decide what the right thing to do is blind alleys that you may have gone down and invested a lot of work into that didn't pan out a process document is meant to capture all of these things and part of that includes selecting what feedback you want to include in the document and reflecting on how that feedback was instantiated in different versions of your game right so writing it just making students write it down and explain it helps students reflect and integrate feedback I want to show you a second system because there are many different ways to try to solve these problems with instructional design and pedagogical techniques so this second approach is actually a digital system it's a web-based and you could

get it on your phone your tablet your browser and it's led by my PhD student Aimee cook and we do we've been working on this in collaboration with Stephen Gow over at UCSD and Pierre presents is designed to be used in a plenary format so designers are on stage presenting peers and faculty are in the audience and they're providing feedback on the presentation and peer presents actually lets them go on on their phone or their tablet and provide their feedback live and in real time before the feedback before the presentation before any feedback happens the feedback receivers have to go in and write questions so the students the designers would actually author the questions that they most want to get feedback on this gives them a sense of ownership of feedback you're not just telling me some random things you're addressing the problems that I personally identified as things that I care about and I need help with with some training students can learn to

write questions that elicit higher-quality feedback from their peers and by making them have a team conversation they all have to agree on what questions they want to put into the system it makes them reflect on the kind of feedback that they want to receive during the feedback process feedback providers can comment and they can vote so they can type in that text box up there and you can see they can agree or they can mark comments that they want to come back to later and during feedback we've actually seen when we've implemented this in classrooms these multiple levels of engagement actually keep students engaged in the feedback process for more of the feedback period right so we would go in and we'd see active engagement during like a 20 minute feedback period we'd see active engagement from students about through about the first five to eight minutes and using this system we would see students continued and engage in peer feedback all the way out to the 20 minute mark which is quite a remarkable change instead of checking Facebook or sleeping what the digital

system also does is it captures all the data from the peer feedback and makes it visible to the instructor which lets you as an instructor reflect on what's going on in your classroom who's not participating in feedback for example we worked with one instructor who was shocked to realize that in this instructors previous peer feedback system only about 2/3 of the students were participating which was about the same about the same 2/3 as participated with the digital system but in order to find it out using the paper-based peer feedback system we need to go through and count a whole lot of index cards and the system just presented it to them immediately finally after the feedback process there is a receiver reflection process that's actually built into the system and it's very simple it's a set of filtering and tagging tasks that students walk through in order to start to make sense of this giant pile of feedback they've just had dropped on

them and so again we ask teams to do this collaboratively they actually have to agree on for example which comments are helpful in which comments are unhelpful it helps them evaluate high and low quality feedback and structuring the reflection process tends to reduce the incidence of some of the sort of failure states that we've seen with open-ended reflection processes like for example ignoring all negative comments or deciding that the feedback means what you want it to mean so we think that there are a lot more techniques to be designed in this space right more non-digital pedagogical methods more digital systems that can help with improving the peer feedback process for student stakeholders both in their role as feedback providers and receivers and also for instructors so I just want to offer this to you as a way of thinking about the challenges that you face with

peer feedback in your own classroom are your biggest challenges around student engagement and the peer feedback process well students just not do it is it feedback quality our students incredibly vague or always really chipper and positive even when it's not appropriate or are your problems with how students are actually using the feedback that they get then you can think about where you have resources to address it so the systems that I've described are all end to end but you can think about let's say your peer feedback doesn't start happening until the second half of the course you might have a lot of time to prepare students before peer feedback before they ever start this maybe that's where you want to design your intervention or maybe you want to think about end-to-end processes in like the examples that I've shown here so between us Martin and I have talked about these four type of feedback and critique peer critique plenary critique supervision and grading I just want to show you we made a couple

of slides with some references don't read them now we'll put up our PDF and you can get them later but we have them and what we're really interested in doing is having a conversation about expert and peer critique with all of you and I want to say we're going to open for questions in a minute but I want to especially point out that if you're interested in peer presents my student Amy Cooke is actively recruiting classrooms to work with right now so I'm putting her email address up there to prototype to prototype to to reach out and there's also I a couple of sessions on critique on Wednesday and Thursday led by Jeff Hess err can you just raise your hand and wave it around okay let's look over there and I think that's going to be an opportunity to also continue these conversations I'm gonna leave our email addresses up there you're welcome to contact both of us but as of now we are open for questions thank you very much [Applause]

hi I'm not in education but I try to participate in a lot of like local bayar because I live here local Bay Area sort of design groups and I have a lot of friends who were working on board games and video games and things that they want to prototype and take to Kickstarter and things and not to toot my own home but I am Tooting my own horn but I have been trying very hard over the past several years to learn as much about game design as possible and so sometimes I kind of feel like I know a little bit more and it can be a little awkward not being in a classroom setting and having like different social situations around the peer critique

process and things have gotten pretty heated sometimes with some friends over certain projects where I see them making certain mistakes and caring more about well is the game balanced right now and really what they need to be focusing on is is the game how hard is the game to learn right now and what do you folks think about those kinds of things it just in general yeah so let me a good I will repeat the question so the question is how do you deal with critique processes outside of a classroom situation where you don't necessarily have the same kind of buy-in to the instructor is an expert that I will say not all students think of instructors as experts and where there may be other kinds of relationships at play so how do you do that I will say that taking some of these peer feedback techniques out of the classroom can be

really really fruitful so we we have looked at a little bit about how they can be deployed in situations with working game designers whether in industry or indie and one of the things there is that peer feedback at sort of is deliberately designed to flatten some of these power distance hierarchies that it is up to the designers to decide whose feedback is important and one of the things that we're actually experimenting with as researchers is whether putting faculty feedback into the same peer feedback system as the student peer feedback helps designers be more critical of faculty feedback so I would say that when everyone involved is voluntarily there and it's a pure lead design situation then then even if someone has more expertise at game design then but socially your peers and I would treat

that as a peer feedback situation rather than an expert critique situation that's how I would tackle this and additionally I think what I said before making them realize their problems themselves is always better than telling them so if you just ask the right questions you might just get a little bit further than by telling them what you think and feel free to track us down if you want to dig in because it sounds interesting thank you both awesome talk a question for Martin so you had a great slide on if someone's easily offended you separate the product in the process etc I thought that was a great point I've tried to use a technique in the past I can tend to be I'm looking at my employee former employees in front of me very bad at giving that feedback so my question is when you build a culture of accountability you want someone to own the product to process everything together but then when you want to give critique you want to separate so there

isn't that easily offended nature so as furious if you could talk about some techniques of how you deliver that communication to be able to separate the product and the process and the review that's going on oh great question thank you so the question is how to how to impractical the design process criticizing the design process criticizing the product of design and criticizing designing criticizing the designer I think a part of the solution is actually what Jessica was presenting that he can standardize some processes and that can help in like you presented the the separating that talking about the experience from talking about the observations from building the theory and so on and I think the key is somewhere in there so by regulating what you're talking about at what point you can be more conscious about what you're criticizing and I I have a hunch that the solution to that problem is in that space if anything but

of course for all of this you have to be extremely reflected about how what you're saying arrives at the person yes and that is just I mean psychology is just a very difficult topic and rhetoric is difficult that is just very very very complex maybe you have experience yeah so I actually although I'm talking about peer feedback today as a researcher as an instructor I do have to grapple with exactly the situation that you're describing one key technique for me is actually reframing what the designer is attaching their sense of identity or their sense of value to so instead of saying you're a designer who is good at design right to shift it to say one of the things I really value about you is how committed you are to improving and getting better and therefore I'm on your side with your goals as a human being

so now let's critique this project together right so find a shared value that puts the critique as something that builds them up rather than tearing them down thank you enjoyed your talk thank you very much I mean these are issues that we all grapple with and one way or another here the question I have concerns the extent to which the peer critique aspect of a course is given is focused on in the course so you could have a course that deals specifically with this and how is that related to grading in terms of I mean so what pedagogical role would this have in the core what do you want the students to learn they could learn the critique or they could I mean or is it producing a good product so what what what role does this have in different courses and how is how does it contribute to a grade for example yeah so I will say that I think

that there's no single answer to that but I can give you sort of the way that many of my courses work which is for every project that you put together there's actually three components to your grade which are each worth approximately a third so one of them is basically like do you appear to have learned any of the course content right so I often will call this theory it's like are you are you using the readings are using game terminology of game designers right so are you kind of are you showing up a third of it is process to the process document that I talked about which includes peer critique it includes playtesting it includes your own critical judgment and your ability to decide what it is you're going to invest your time in it includes things like how you manage conflicts within the team and then third is a product and I which is just like what did you make how good is it and the reason why I grade like this is because like Martin I

really want to push students to be ambitious and experimental while they're in school and can do that and so I want the person who tries something really safe to get the same grade as the person who tries something really really ambitious crashes and burns but learns an enormous amount and both of those students I think are about as successful in my classes and in the process of becoming a game designer and the student who's gonna really succeed in hey you know who's gonna get like the top grade in the class is the person who is both developing sustainable game design process where they're trying things and really operating in that zone of proximal development and they also managed to make it work I trust of an anecdote about that in the first years I was talking a class called game design I was teaching a class called game design I had a cool teacher who was very much into turning everything into a game a game designer working in the games industry and just an external lecturer and he actually turned the grading into

a game so he he introduced a system of peer grading that was not really hundred percent of the grade but about 50% of the grade was actually decided by peers and in the end quality is something that is defining if you trust the great was purely based on the product in that case and actually the games that got graded highly or regarded highly by peers were exactly the games that actually deserved it except for some exceptions so we kind of fixed it a little bit in the end we could override the peer the grades that the peers gave gave but in the end favored because of us just given the grades that the students gave each other and it'll have been fine in a way so it was a very nice exercise actually an indie bus like packaged in a very nice game thank you hey so like with these really structured peer review systems

there's like a lot of fluency the students have to build to sort of get good at them and use them effectively how have you guys worked with like other departments or other like instructors and your programs to like help that fluency carry on so it's not just like I invested so heavily in this technique and walked off and could you talk a little bit about what that looks like and how you got buy-in from maybe people that were a little bit reluctant to do it this way so I'm just gonna repeat the question like I think I forgot to do last time so how do you get buy-in from instructors are these techniques especially for peer feedback is it high overhead so we've no mic team has now worked with people across multiple departments at multiple schools actually so we've spent a lot of time thinking about how do you onboard people to these processes and generally speaking that a faculty already bought into the idea of peer feedback having value we can generally get them up and running with our processes within one to two hours of

their time so it's actually compared to what they're already doing faculty will actually generally say these things are less work it saves them time now of course convincing them that it will save them time before they do it is its own tricky scenario but now we have enough data for instructors saying this is about how long it will take you we typically actually have someone from the research team just come in and do like a 20 minute training for peer presents for the Yoda method it's a like a one-on-one meeting with the instructor just to walk them through it and show how they should teach the students so it's a modeling process they can do actually live modeling with one of the games in the class right so these are meant to be things that are pretty lightweight and you can try if you're not already using peer feedback in your class or if it's mostly peer evaluations so thinking about peers giving grades then it's a much much bigger cell and to be honest we have not worked with instructors who

are not involved with peer feedback in some way the research shows that peer feedback is great for students so one of the things we're hoping to do is to create systems that are so easy to use that you won't mind changing your syllabus but we're not there yet actually we have a unit at the University called learning support unit which is a small department of just five people or so that research and provide training in tools like you're producing and that is how my university solves that problem that you can just book them to give a workshop to yeah so but finally actually my part of this class was developed in the teacher development program of my University so where I I and the reason why I even had the idea of talking about critique was because we have it in so many different disciplines you have it in in you have critique sessions they're called code reviews in

in programming as well as in design so I think sometimes is also just translating your tools to their language that generates their interest thank you hi there so I'm currently co-instructing a course with over here over at pretty universe and one thing we struggled with a lot is this pure feedback and just a valuation critique it feels like since we are part of a program called CGT which is computer graphics we're trying to grant cram in like the art side and the programming side and the design side all within two courses and so whenever we're spending time on these feedback sessions or like sharing presentations the students sort of feel like they're losing that time you know and not just the students who say you know want to get out of them critique but also the ones who are really fired up to keep working on their game they feel like they just lose a whole day of development time that they're together

cooperating and so we've tried a couple different systems and I do think norm setting is going to help like I honestly just want to go back home and implement that as soon as I can but in the meantime I was curious about the idea of sort of like remote critique so for example you have everyone sit down at a computer with someone else's game you play it it has in-game analytics that are keeping track and say like logging that to a file and then you step over the next computer when you're done you know it sounds reasonable from a logistical standpoint but I'm concerned that it wouldn't have like the ethical and like empathic response that the stuff you described would have do you think a remote method would be in danger of losing the value because you're not getting that like one-on-one or or that communication great question so I'm gonna attempt to

summarize it a lot of context in your question so the question is if you're in a situation where there's a lot of demands on in class time are there ways that you can do peer feedback that sort of minimize the amount of class time devoted to it but still get the benefits of peer feedback so one thing to keep in mind so I talked about in class peer feedback because there are systems that actually will let you do asynchronous remote peer feedback on other students project and it turns out that there are basically students find them very burdensome and they won't do them and so that's part of the reason why we're looking at in class but you can make it more efficient and one of the things that we've looked at is breaking students into groups so we you can actually do the eodum method in parallel

what I would recommend is having the students go around in groups of three so that they are actually hearing each other's feedback in each other's comments that's really important but one thing that I'll do is have a designer stationed at each station and then have students go around in groups of three so now you're dividing the right because everything's going on in parallel I think that could really help you right up of how I've done that which I'm happy to send you so I have no experience with remote feedback I can say anything useful to this question so well so you're characterizing it as a remote I wouldn't call it remote I'd call it parallel feedback yeah I'll also say that one area I'm really interested in exploring is actually live-streaming for peer feedback so if you're interested in prototyping this in your class let's talk and we've got some systems that can support you with it so that might help you take it out of class

time but still create that sense of peer accountability and commitment to each other as human beings as well as designers so we'll talk awesome I know another instructor whose research is focused on live stream teaching - so I would love to okay great well make the connection and we'll thank you GDC this is awesome yeah thank you so much okay and I think this is gonna have to be our last question sweet so my question is for Morten about the expert level feedback and I'll give you some context is I'm a previous software engineer and we do agile and sprints and at the academy that I teach at we do two-week sprints in the last four months okay and every two weeks we get together and then we do back with the students and as an instructor the problem to be solved is to make sure that the students get the most out of it that they can create you know critical analysis and provide

useful feedback but as a software engineer the product needs to be good towards the end so my question would be do you find yourself sticking to your guns throughout this entire process or do you find yourself evolving and making a decision and run time is when you need to kind of step into a more professional type of expert roles so the question is if I'm sticking to my guns of actually it's a very good question the question is if I kind of overruled my recommendations of forcing them to reflect on the process in favor of actually just giving them expert advice because I am the expert in the room wow this is like it is a great last question that we could talk about for half an hour I think actually whenever you design a class or a curriculum or

whatever you always have to make this TFT slider where you define if your class is supporting students learning about the process or if it is about having an optimal end result as a product and I usually think that it's a it's so complicated because students get motivated by working on let's say a great game if they're in a game development class so of course you want to support them making a great game but at the same time you also want to support them even more to make to fail spectacularly as you said before and a great game is easiest made if you make a very safe bet and you don't want them to bed safely also you want to you want them to have communication problems so in their team because they have to learn working with different opinions with different paces of working with different cultures the different backgrounds and so on so what but one of the initiatives we have in

Denmark is called patio and it's a collaboration of 12 universities where students make games in teams of sixteen people which is bigger than supercell has for example with the idea that the teams are so big that they mostly have overhead and hardly any productive work for any productive work it's done which is an awesome learning experience but the games they make are let's say questionable so I and I didn't understand why they are doing that for many many years and nowadays I actually support that decision because I think it actually leads to the better learning outcome so I try to stick to Meghan's actually and I try to actually support students not having an optimal product but actually learning how to make the optimal product for the rest of their lives or making at least and a passable product under harsh circumstances I

think there is actually more learning in that I think so I agree with you about the slider and for me the slider is set different places in different classes but even when things are more focused on the product my goal is if it's like you know teaching them giving them a fish versus teaching them to fish right so if I just tell them the right answer it's not helpful that's not learning so what I will typically do is I will say to them you are struggling with this problem I will now show you three ways speaking as an expert that you can address this problem these are three examples they are not the only three ways but I want you to use these exemplars to figure out your own solution and I give then I give them three directions using all of my expertise and I do three in order to reduce something called sort of functional fixedness where they like if they just hear one idea they'll sort of go oh well I should do that and it's gonna be hard for them to get away from it but when it's more than one idea they

can actually compare and contrast and that helps them generate new ideas of their own so that's when I'm more focused on outcome product that's how I still sort of thread the needle of if I tell them how to do it and then they make it that's like and then I put my stamp on it saying these students can do this themselves I don't know that that's true and so I need to make sure that whatever the product is I can actually be confident that those students could do it again and that's that one of that those techniques is to say here are these three options reduce the power distance so that they feel free to choose something else and then if they succeed they succeed and if they fail they fail I'm sometimes even more nestea I actually tell them one solution and then I tell them they're not allowed to use that one listen thank you very much again our email addresses are up here thank you so much [Applause]