Feedback & Performance Evaluation | ft. Ben Carcich & Aaron Smith from Building Better Games

in my entire career I have never been at a company where we did employee feedback or performance evaluation very well maybe we did it sort of okay but not super well however this function is absolutely critical if you want your team or your company to improve today we talked to Riot veterans and now game industry experts and Consultants on production and game development Ben Carson and Aaron Smith from valeran Consulting who you may also know from the very excellent podcast building better games we go practical and deep on both of these topics these are highly controversial emotional and sensitive but important topics if you want to have an impact stay tuned we are coming at you with real practical and usable information advice coming up right now today's episode is sponsored by datadot AI to access estimates rankings downloads Revenue usage or engagement

for millions of apps on the App Store and Google Play sign up for data.ai and before we kind of dive into the main topic I thought if you guys had a you know a minute or two to also talk about yourselves and also about uh Valerian or building better games the podcast I I think it'd be great for the audience to hear a little bit more about you guys yeah um I can uh Ben if you want I can go forward start kick off and talk a little bit about valarin too um so I I have a background uh I grew up in Central Coast California um so I'm a I'm a born and raised California boy um went to college for film did that for like a year right after the the GFC in 2009 hated it didn't really feel like I fit in um video games were always my thing and so um I you know when I was at Rock Bottom I finally decided to do this crazy thing and apply to this video game company no one had ever heard of I found on Craigslist called Riot games and uh I

went there and then League launched and the rest is history um and uh spent a little over 10 years there and uh kind of Grew From a baby intern getting sandwiches for Mark Merrill all the way up to a development director when I left um so it was it was a pretty wild journey and and after that I met Ben there and we did a lot of work together and really found that we had a passion for the craft of production and leadership in addition to making games which we're also very excited about and so here we are having started valarin and uh valarin is really about actually helping companies um you know really get access to that secret sauce the things we found consistently that worked many of them counter-intuitive from what a lot of game companies are focusing on right now so we're out there trying to spread the good word and add value yep yep for me my my background was actually military so I think I got really interested in leadership all the way back perhaps even in the in the Boy

Scouts but when I joined the military was an officer uh trained in combat arms and ended up doing Logistics which is very different space um and then went into video game development another very different space and getting to see the contrast and Leadership between Riot and game development generally and the military but what but also kind of identifying and thinking about what stayed true what was true in all in all the cases that I encountered what were the leadership principles that sort of undergirded success um yeah that's stuff I love to think about and I'd love to help other people think about and so I went into discipline development and yeah now we do the building better games podcast and newsletter and all that stuff and that's all about trying to help leaders be better leaders in game development awesome all right well I think we could jump right in and I wanted to start this topic off by just an observation that from my personal experience as an old guy working in Tech and media companies and gaming companies for most of my life I have to say that I have yet to work at

a company that is great at feedback and kind of doing performance evaluation and we can talk about both of these topics separately but I wanted to start by asking you both why do you think this practice around feedback and performance evaluation and in my experience especially the bigger the company it's often the worse it is but why do you think it is so bad or if you guys disagree with me please disagree but um you know I'm I'm wondering have you guys worked at a company where you feel like this practice has been executed well am I wrong is this being done well somewhere and I just have never experienced it in my career so I have worked in an environment I don't know when I was at parts of Riot I think this actually operated very well um but I agree with you that broadly I don't see it operating very well and there are a couple reasons why and I

kind of go into some archetypes of companies that we've seen and one that's really interesting that popped up in the last couple years is we started Consulting with a lot of startups and uh mid-sized companies and things like that we found that a people struggled to give hard feedback because they thought it wasn't nice to do that like there was sort of a I almost call it like this toxic kindness or this like politeness that um was so ingrained in the culture that to give someone critical feedback almost felt like a violation of of how everybody should behave and when I look at that from the outside like that that's really bad um but when you're in it and everybody's just operating and they're all being kind and encouraging and supportive lots of good things by the way nothing wrong with any of that but then when you think about okay I need to give someone harsh feedback or critical feedback um it it doesn't land well the other archetype is a company where feedback and this this reminds me more of the larger Publishers

um that and again we we've worked a little bit with some of those um and I also hearken back to my military experience the feedback's just not desired it's not wanted there right look just do your job let those people do their jobs and if they're messing up well do your job well don't get fired and uh I think behind both of those there's this idea that if you give feedback you would you're attacking the person the individual is under assault and this is the fundamental misunderstanding that if you could get a company or group and this is what Aaron and I actually have managed to create and been a part of I think at times uh in in the teams and organizations we worked with um if you can get a group to really buy into the idea that feedback is about a moment in time it's about a behavior it's about a capability it's about a particular project it doesn't Define the person who's receiving the feedback uh in any way I think that is one of the

biggest sort of mental hurdles you can get people over but when it is viewed that way is I give you feedback therefore you're bad as a person I think that is why so many people struggle with it they struggle to give it well they struggle to receive it well because it hits at your Center if someone's like you did a bad job on that project planning right like I'm a producer right like you did a bad job on Project planning um if I interpret that as I'm bad at project planning but I put those first two words I'm bad then I really try to avoid feedback because it it messes with my own ability to to operate and to feel confident about who I am and the work I'm doing and all these different things versus a frame where you say hey this project plan didn't go well here's why and and I go oh got it I can do better next time yep nope some of those things are valid some of them aren't that's okay let me see what I can learn from this and do better next time and that comes down to a concept called the growth versus fixed mindset Carol dweck we've talked about that a bit on our

stuff but um yeah it's just this idea of like are you open to the growth and the feedback that can lead to the growth yeah one one thing I'll add to that is that I think the mentality towards feedback within organizations is fundamentally broken I think most people consider that feedback that they consider that the way that they should interface with feedback is in a reactive way so so but you'll you'll most people think of that in terms of well I need to give feedback when something bad happened and I think the good the sort of enlightened thinkers out there in the world will say well hang on a sec we should also react by giving feedback when something could happen and they're the sort of ones on the bleeding edge to me the idea that feedback is a reactive state is the problem is inherently flawed as a concept if we as organizations viewed feedback from a proactive State it's something we're

constantly doing with each other all the time the good the bad the ugly we're we're constantly opening the door to a conversation about how we're all performing how the team is performing how you are performing did you do great on these things and not so great on these things and are we able to talk about those things that are uncomfortable and do we push ourselves every single day to to have those uncomfortable conversations therefore to build resilience into the organization to having more and more Awkward conversations about performance and about values and about outcomes and about effort and all these things I I I feel like most leaders default stance is that on a good day if everything went perfectly I shouldn't have to say anything like I shouldn't have to give any feedback at all because I shouldn't have to have any uncomfortable conversations and I actually think that we need to flip that script and start changing the culture to being on a good day that is just free-flowing and happening all the

time and that that is a fundamental shift as well that we rarely see in organizations I think because of the political nature of organizations it is like let's you know minimize that like we're doing well if we've minimized that yeah so when you think about how you flip the script on this to use your words I I think there's something I I kind of think about it at two levels right so there's there are the people who might be very reactive very defensive who would take things very negatively and then there's also sort of the general environment in terms of like the company whether it's the culture or the way that things are established if if you are a manager or a leader at a company how should you think about both of these things like how do you start to begin to think about flipping the script in an organization or on a team where we we have this I and I would call it

very typical traditional 90 plus of companies have this situation where you give the negative feedback someone is going to take it negatively and there's really like almost no benefit for me to give a co-worker or a direct report really direct feedback because the only thing that would likely happen is a negative effect somebody then retaliates or gives me a poor performance review because I gave them a performance review things like that but again just to kind of recap organizationally on our personal basis how do we begin to flip the script it's it's difficult to flip the script because some of the most powerful tools you'll have in Your Arsenal if you want to flip the script are going to be things that are at the foundation of your organization so like one of them is the the the proverbially speaking the thicker the skin is of the people within your organization the easy easier it will be to drive a culture

where those conversations are free-flowing now having thick skin is something that comes with experience it's a very related to personality characteristics and attitude and behaviors and these things and Ben and I often say that when you're talking about attitude personality characteristics these things they're harder to train in to an organization that doesn't already have them so what do we often say well how what's the best way for me to get those things Aaron well hire people that already have them you know and so there's a little bit of a catch-22 there that we recognize that being said if you're not already there you can train it into the organization but it's going to be one of those cultural things that requires constant High touch accountability from your senior leadership and also the senior leadership demonstrating that they do that within their group because if they don't do that like I can't tell you how many times Ben and I have gone into an organization and had the executives reflect to us that they would like to see people talking more or having harder conversations and then just knowing that

there's like awkward three or four awkward elephants in the room just between the executive team themselves that like all you have to do is hang around for a couple hours to start to pick up on and but so it's like well I mean you know you're setting the pace in all of the subconscious ways and the conscious ways right so you're never gonna see a longer term change if you don't change your behavior at the top um and that's that's tough to do right so when it comes to again flipping the script one of the things is you know building resilience Mark Merrill used to talk about this a lot at Riot and I always appreciated it he was like we need to keep feedback free-flowing because it's a it's a helpful tool in building the resilience of the recipients of the feedback like the more you receive feedback the better you get at it and the more used to it you get and then it becomes a table stake to just always be receiving feedback and you you start to you your skin just naturally thickens up so that's one piece and another one I think is training people to give feedback

effectively early and often and then the third thing I think is creating an accountability structure in the culture that says back to what we were talking about earlier if we've gone several weeks and nobody's giving any feedback that's a bad state and we're gonna we're gonna pump the brakes and we're gonna step back and we're gonna say why aren't we doing this how do we start doing this immediately like there needs to actually be corporate level for lack of a better term accountability around actually executing the feedback on a regular basis it needs to be something that leaders talk about and ask their teams about like hey are you all doing this hey are we doing this hey we missed an opportunity to do this like constant accountability because it is a really really big fundamental shift in Behavior the the thing um a few things flow in my head when I think about this topic one is the idea of psychological safety and importance and also I think how often it's misunderstood um uh and I think psychological safety is often like well we need to create an

environment where everything's safe and that means we can't threaten everybody anybody ever and if feedback is threatening we can't talk about feedback I'm not saying everybody says this but I've certainly seen that as an anti-pattern and that leads to that that toxic positivity right the toxic kindness um I love what Aaron's talking about where he talks about the thickening of the skin and I think it's actually psychological safety is a in is present in a place where people are willing to be vulnerable with each other and as a leader in a space lead with vulnerability to encourage others to follow not everybody will but that's one of your best ways to drive some of the change and what that means is ask people for hard feedback and kind of you know even early like be a little bit intense about if they're not giving you real stuff they're like Oh no you're doing great no no like where how could I be doing better right ask your ask the people on your team if you're leading a team those you lead those you serve right ask them how can you do better and

take what they say seriously and model that it's a positive thing that you're thankful that they gave you the feedback and so um there's there that's one way that you can start leading by example is requested and then respond well to it another is by giving it and giving it positively and negative ways Aaron said proactive stance here we had uh someone Francis Frey came to our organization and talked a lot about feedback culture because Riot had almost in certain parts of the environment gone too far on the feedback side where feedback just meant like it was a good thing to give feedback and what that meant is you constantly walked around and just critiqued people and walked away you're just like well that's wrong and dumb because blah and like oh you clearly weren't thinking about this then you walked out of the room and it was like well look how good I am no that's not good feedback right right that's just you you know being somebody who is capable of spotting problems dropping problems on people and then you know exiting the room in the environment and acting like

you did something positive they don't know what to do with that and it does start leading to a place where people feel less safe it does actually break down people's ability to be vulnerable so instead think about the positive feedback you can give as well give a lot of positive feedback to your team right if you're a leader or you're just a team member right this is game makers that go out of game devs out there give when you see somebody else do something that is good even a little bit call that out encourage that one of the things what human beings are so responsive to encouragement and so if you see something good encourage it you you'll probably see more of it and if everybody's doing that with the good stuff now we're increasing the good stuff and it also means that now I'm trusting the people around me because I see that they're seeing the positive things I do they're encouraging me that if they give me negative feedback I know it's not because they think I'm bad because they're trying to tear me down I know it's because they care about me and they've spotted my good things they want me to do better they're probably calling out something I'm doing wrong because they want me to do better too so

that those are some simple things yes give uh like develop the thicker stick in also develop an environment where there is vulnerability that's possible like that's the true nature of um of psychological safety I think is when vulnerability can be expressed and heard and everybody's open to that and to the learning that comes from that um but also like I was reading collaboration equation by Jim Benson recently and he got into a big argument with his manager and after the argument they were like yelling at each other in someone's office you know way back in the day and um I love this example because after someone walked over and was like hey do you want to file a complaint and he was like what and he's like yeah I mean that's a grievance you know that's a senior director just talking to you a rank and file he's not allowed to talk to you that way do you want to file he's like no I don't want to file a complaint he's like well he can't do that it makes you unsafe he's like it is totally safe because I was able to have that conversation the unsafe thing would be if I couldn't have that argument with him we were talking about the project not each other right

and I'm going to go work it out with him and he's just sort of told the guy to like you know get out of here I don't want to file a complaint that was an example of and I'm not saying that's the place for everybody to be at where you're all yelling at each other or whatnot but there was psychological safety there to have a heated conversation and that came from them their willingness to give each other feedback to process that to go back to think about it and they they eventually came to a really good conclusion and worked together I you know I haven't finished the book yet but um yeah it was a really cool example of like collaboration sometimes looks messy it looks heated it looks like conflict because it is um and so you have to create an environment where that can live right so one thing that I'm thinking about as you guys are talking about this kind of feedback and and Aaron to your point there's got to be like accountability and you know hardening the skin requires reps and I think some of the ways that you guys have described seeing more on the individual level where like you are asking for feedback you are kind of like

directing your own path at a company to get feedback but these largely seem like more informal systems and so one of the things um I would be curious about it whether at Riot or elsewhere what kinds of formal systems do you think should be in place and one thing I also think about just just so that you guys so just from my own experience one thing that I changed was that to kind of um basically disassociate performance and feedback where so now what I do is we I kind of check performance evaluation out the at the window I don't know if that was good or bad and instead I'm doing like monthly start stop continue where it's not formally attracted this this is basically here's my feedback to you you take it and basically this is more informal so that it's not like this yes okay here are the the fat things you're doing and it's

going into your performance evaluation and it's like being tracked so I I guess what I'm wondering just to get back to the point is from a formal kind of when we're talking about ceremonies around feedback more formal systems what do you guys think about the kinds of systems that would work or would be more effective when it comes to feedback yeah there's I I like the idea of um Regular check-ins and and the specific thing you mentioned there that's intriguing to me is that you deliberately separated them from the form from any formal performance evaluation so it's like going to the gym and being like well I'm gonna kick it up to 50 pound dumbbells today but then you couldn't quite get enough reps in back to your point about reps and so you you know you drop the weight you feel a little silly but that's about the extent of the consequences versus going to a weightlifting your uh competition in your local town now you lost the competition because you weren't able to so it gets you in the mentality of

always practicing and trying to push yourself to the next level without the severe consequences of like well you got this promotion or you didn't get this promotion I think that's yeah that is actually a really clever management tool and also by the time the competition if you if you follow the analogy does come around you've already been practicing so much as part an inherent part of the system that by the time the competition comes around you're very comfortable and used to practicing all these different um you know weightlifting techniques or whatever um as far as formal tools um you know it's it's interesting to note that a lot of the companies that are best at this don't have those formal tools and one of the reasons why is because again it's such a culturally foundational thing like that like a lot of times I think at Riot it really did come out just from the people who were hired with so we knew when we were doing interviews we were looking for the people that were more inherently comfortable with that kind of communication and we would hire those people and so actually I think it was a

a Miss on Riot to not have more formal Tools around this one formal tool I did see that we used on our teams was actually to get the team together every six weeks and sit down in a room and actually go round robin with everyone on that specific leadership team where one person was in the hot seat and the other five people would give them feedback openly in front of the group and it was extremely awkward and uncomfortable both ways it was awkward and uncomfortable for the person in the hot seat because they were not allowed to talk they could only listen and take notes that was the rule and for the for the givers of the feedback it was awkward because now you're having to sort of declare what you think about this person's performance what's working what's not their attitude all these things in front of five other people so it's just awkward all around but it it was a very specific process that we would execute every six weeks where we'd get into a conference room a private conference room and do this and and actually I would say about 80 or 90 of the point of

this was to actually build that resiliency and that comfort so that when we were outside of the context of that room just going up to the other person and saying hey um I think you could have done better with X would be a much easier conversation and that actually came out of um the live Services Group at Riot they developed that process and used it within their group and you know some of us just stole it and requisitioned it because it was such a valuable way to sort of build again do the Reps as you said we're just constantly doing feedback reps all the time I'll also say that as somebody who's part of that while I think the first time you do it it's pretty awkward both ways yeah as you get used to it it doesn't it becomes less awkward um quickly and you because you realize the other people in this room again they're not here just to gain points by taking me down a notch right gain relative status by by you know demoting me in everybody's mind let's say um they're here to help me and I'm here to help them and if we all are

succeeding which is what we want then man imagine how much better we could be if we actually paid attention to each other and we gave this feedback well um and so you know another another tool that was a a formal tool and I like these um they're 360 reviews uh you can do those and I think one of the big things is the person that's having the 360 review that's at the center of the 360 review right that's saying like Okay these people are all going to give feedback about me if you're the their manager and you're kind of running that process what you said is so important separate it from performance evaluation and that might mean that if you do this let's say you do both every six months have them separated uh you know have them be every three months you get one of those things and then it sort of rotates so that's three months apart from the last time uh from a from a Time perspective because if they happen I think this was a mistake when uh when Riot first did this they had them both

happen kind of at the same time it was like right we did a bunch of performance uh like 360 reviews and stuff and then immediately after we had performance evaluations and you could just see how the those two things got like twisted up in people's minds and because of some consequences to certain people that after that you just had people very deliberately selecting people on their 360 review boards that were only going to say good things about them because if they said bad things and I wouldn't get a promotion or I wouldn't get a raise or I might get fired or whatever it was like you didn't you don't want that Association but that comes back the idea of if this is a culturally present thing that's when you're winning because now people are always doing it all the time you don't go a week without getting feedback so when it's performance review time yeah I got a bunch of feedback in the last month but I always get a bunch of feedback in the last month we all do it's just part of how this rolls and one of the most important things that I think the leaders at Riot did and I'll say there are limits to this

but I'll I'll kind of describe the technique when you heard somebody talking about someone else in a negative fashion no matter how polite or impolite that was you pushed that person the thing that someone would say it was it was like a a meme almost at Riot right it was this it was such a cultural tenet what did they say when you told them and if they were like ah shoot you know especially once you've been there a bit you knew like ah crap I was talking about somebody and I haven't talked to them yet um I need to go talk to them first you know before I go and complain to anybody else and you go talk to them first because one of the things that happens when you go talk to that person you are forced also as the feedback Giver to understand the complexity of their reality because maybe they're going to talk to you and be like well this is why I was doing that you're going to be like oh shoot I thought you were just a jerk in fact there's a bunch of good reasons for why you haven't responded to this or you haven't done that or whatever um and and then by the way you then continue that conversation you go like okay well

what could work right and now you start working on a solution rather than just talking about this person as if they are a problem um so that was so embedded in riots culture for such a long period of time and I think that meant that people were constantly giving and receiving feedback not always great at it but it was it was happening it was happening a lot and I think when that started to fall apart in certain parts of the company that was when Aaron and I started getting really concerned actually because it it's hard to replicate with broad formal processes like 360 reviews and all this stuff it has to be part of how people walk around um because the other thing is feedback is like finding bugs the faster you find a bug in game development the cheaper it is to fix the more rapidly you give feedback to to the event happening that requires feedback the more effective that feedback will be and so if you only do like a 360 review every six months let's say you're giving somebody feedback about something happened four months ago your

memory is clouded and and suspect their memories clouded in the suspect nobody really remembers and you're just like I don't know I just didn't like how you talked to somebody or rather they I they didn't seem to resp and it's like does any I don't remember that ah shoot you know feedback should happen quickly and that's what you're trying to encourage and that means it is almost definitionally if it's operating well and informal rather than a formal process because formal processes tend to happen at cadences right um and you need you need the faster response like you need to you know if you're a leader and you've got a manager and You observe their meeting or something and you have a bunch of notes if you can as soon as that meeting is over give them feedback again assuming there's there's safety there and whatever um uh you know give them feedback immediately after and they're going to be like okay I can remember that I can remember that meeting I can remember what went well what didn't wait two weeks and it's like I'm sorry I've been in like 50 meetings I don't remember what was going on um so anyway that that would be yeah

there I think this is something cultural it is something that should be informal and yes if you have formal processes that assist great but the default of feedback needs to be uh informal actually it needs to be Behavior right and I I do think that um I I feel like the informal systems are often much more effective just because if it's not formally tracked then you're not holding back as much right where you can be more honest because you know it's just for that person and that person is going to take it as okay uh this the reason why I'm receiving this feedback is for for my benefit or hopefully the interpret it in that way um but maybe my next question is is is around feedback is really around the delivery of feedback because this is something that I've kind of struggled with throughout my career in that I tend to be very direct in clearly not everyone is the same there are some

people who just want it straight up just tell me what I'm doing wrong I want to get better just be very direct don't you know don't sugarcoat anything and then there's other people where you know they you you got to get the [ __ ] sandwich out right so so the nice and then the the real feedback and then the nice thing um and so when you think about delivering feedback to different people how have you thought about it I've been more of like a more more the Rambo over my career and I'm slowly starting to get a little bit a bit better at like nuance and adapting feedback to different people but if you guys were to give recommendations or suggestions on how to think about delivering delivering this you know what is essentially this the same message but to different people in a more optimized way how would you guys think about that yeah one of the challenges with some of the conventional

corporate methods were trained to use when it comes to feedback like the poop sandwich for example is uh the the I don't think that I can't imagine someone who actually feels better when they get a poop sandwich because you know that there's poop in it you know what I mean like you know the the the lovely bread tomato and lettuce that you added on there is merely a delivery mechanism for the poop so like it's hard not to be cynical if you know you can feel in the air what's happening you know yeah and so I I I tend to feel like we're still running into that same problem culturally where we're just struggling with the fact that we have to have a tough conversation now that being said um there is I one of the things I really want feedback givers to understand is that if you're already really pissed off by the time you're giving feedback or you're already really frustrated your

chances of delivering it in an effective way are very low and this is actually one of the big criticisms I have about most of the corporate model is like if I were working for like a big sort of draconian for lack of a better term company like big established Corporation where people didn't give a lot of feedback and I had some senior director and I was just like a you know mid-level producer or whatever and I had some senior director walking in the room and say I have some feedback for you I'd be pooping my pants you know it's probably not a good situation that I'm about to go in and that's because like the only reason anyone would ever give feedback is if things have gotten so bad that they had to they felt obligated too and so again there's a pro activity that proactive stance we need to take on this to get ahead of that as far as the delivery the impact feedback model from yes book crucial conversations still to me I use it in my personal life all the time it disassociates in a very healthy way the nature of the feedback

from the other person's emotion and their character and I think that's the powerful aspect of it so the the impact feedback model is to actually uh provide factual observance observant observations basically so to say hey um Sally I've noticed that you have um been to work late the last like two weeks consistently and um your team has mentioned to me that you haven't been showing up to the stand-ups and they haven't really been able to get a line on like what your work is or what you're doing and as a result they're they're worried that they're gonna fail the Sprint the Sprint um and they feel like you when they try to bring it up to you they can't talk to you about it um so you know as a result I'm concerned and then this is again this is the key part saying I'm concerned that perhaps you're not taking this very seriously or I'm getting the impression that maybe this isn't very important to you and

then what's what's key about that part of it is that you're actually now internalizing responsibility for the narrative that you've constructed right instead of saying well obviously you don't care or obviously you don't give a crap that's where it becomes accusationary and that's where or accusatory that's where it becomes sort of it feels can feel attacky when instead you say hey I'm telling myself this story as a result of these things that we both agree have happened and the impact so that's the impact that that's having on me and then follow that up with like hey a recommended change of behavior so if you could come in on time the next couple weeks that would mean a lot to us or like we really need you to do X um this is how you sort of like the three sort of phases of factual observation impact in this sense of like this is the impact this is having on me this is the story I'm telling myself and then three is recommended a behavioral shift or or specific follow-up actions is actually a really really good model and it's also very direct and honest to

there's no fluff in there necessarily I think it's really actually just again about psychologically separating the emotional parts from the factual Parts yeah the the thing I would add to that too is when you when you're giving feedback a mistake is to believe that you only communicate one way in a feedback conversation now sometimes time doesn't allow for more and if so be incredibly careful about giving feedback in general one of the things that I do in that because I love the impact model I love crucial conversations it's a book worth checking out if you're struggling with feedback or trying to bring in this idea of hey this is what I'm seeing this is what I am experiencing this it seems to me that blah and then you ask what do you think of that like what are you what are you hearing and you you oh you let

them respond and you know some people respond to that super well some people don't respond to it super well but actually it can be really helpful uh for them just to know this is how this is impacting me right that's the idea of the impact model this is how that what you've what you have done has impacted me um this is this is how you know in that case that that would be like to me there's a there's a problem there in the one that Aaron described in that like wow some of your team was talking to Aaron about you and and Aaron's now reflecting to you that like hey they didn't feel comfortable talking to you um and they expressed that to me it seems to me that's that like what do you what do you think about that and and let there be a back and forth there to build the understanding so uh impact model asking questions then making the recommendation um for and and again if you if time if time only allows like the the one that Aaron said I think like facts impact recommendation that's sort of the core of it um I love I love it if I can provide

time and space to for there to be dialogue uh because sometimes that helps me realize oh there's a lot I didn't see as I mentioned earlier and that's something that's always like it is so it happens it is so rare to find somebody who is actually just a malicious agent inside of an organization it is uncommon um and so so often when you are able to open up that space and say Here's what what I saw here's how that impacted me questions right like right what do you think of that um that also provides then it's less of an attack and it can feel less of like a drive by all of that that whole thing feels less of a drive-by because if I contrast me saying this is what I saw blah blah blah blah versus like hey you're really messing up blah or like do you even care about and or whatever and you hear feedback often when it's especially when there's a lot of um

a powerful negative emotion involved you hear feedback often comes out that way as just like it's just an attack it is an attack right and it's very easy it's puts the other person in a defensive State and they feel like they need to protect themselves and now they're worried especially if it's like the senior director talking to the mid-level leader right like it's they feel they suddenly feel like oh no well how do I get out of this um and how do I know that wait that's not true oh no they didn't understand you know and but they don't if you never get a chance to respond it doesn't increase your idea of vulnerability it just is like oh shoot how do I make sure I never have a conversation like that again um and so you're what you'll do is you know keep your head down and potentially um even hide your mistakes and both of those are negative pavers yeah and I I wanted to shift over and talk about performance evaluation but before we do there's actually a couple of a couple of other follow-up topics I wanted to kind of try to close the loop on regarding feedback in the the first is like this notion of

like how do we even like prevent us from getting to the point where feedback needs to be given and I feel like the number one reason when I think back on my own experience and like mistakes I've seen other people make is basically this lack of communication around what are the mutual expectations of a manager in a direct report and because it's like a lot of these managers are basically asking basically expecting their direct reports to read their mind you're not doing things the way I want you to do but wait have you actually talked about okay what is your understanding of good performance and now let me tell you what my expectations are and then to align those so that you never even have to get to that point where like you are disappointing me you're not meeting um expectations and so you don't even have to give the feedback but I want to also just talk to you talk to you guys about whether you think this is as

common of a phenomenon is is it's it's been in in my career and from what I've observed and then what are other ways in which we can even like before it gets to that point yeah be able to communicate or set an environment in which we could even get get to the point where we can minimize the number not not that we I mean you should always still get feedback but we can minimize the number of situations where performance or other things are kind of slipping where you have to get feedback yeah I I mean I think it's beyond clear that the issue you're describing about lack of clear expectations and the negotiation around expectations is an epidemic like I I'm comfortable using that term it's an epidemic in the western corporate world um I that's I don't have a ton of familiarity outside of uh the Western corporate world but certainly in Europe and the US it has become tragically bad

um to the point where I think the vast majority of people have little to no clear expectations between them and their boss or them in the parties they're accountable to and I agree with you I think one of the logical outcomes of that is that when feedback and performance evaluations happen they become punitive and they can very become easily become punitive in nature because it's the only time where you can deal with the stuff that upsets you it's the only time you can deal with the things that bug you and even worse is that by the time you finally get to that point and everyone can sort of get it all out all the stuff they've been holding on to for the last eight to 12 months now the person on the receiving end is being surprised by 75 of that and this is just it's just bad outcomes across the board and it's bad management it's uh I think it's a failure in the discipline of management people management particularly most of the good people managers I've encountered in my career

fortuitously stumbled upon the skills that allowed them to be good managers there's not a lot of formal training around this stuff there's not a lot of formal mechanisms around this um you know there's a you know accountability as a concept is Central to this conversation and it's interesting how many organizations talk about accountability like it's the only piece that they'll discuss out of this topic you just brought up we need more accountability like we don't have enough accountability our people aren't accountable we hear this stuff all the time when we go work with executive teams and it's and again it's interesting if you listen closely to the flavor of the way that's being expressed that I'm frustrated and I need a way to just like grab people by the shoulders and shake them up and be like you're not doing what I want you to do and I need also a a maybe a subtle or not so subtle bat in the back that you see that's ready to smack you over the head with to make you do what you're supposed to be doing and the thing that somehow no one

remembers to do is to sit down and have a conversation about here's what I expect you to do up front right here's what good looks like here's what great delivery of this project would look like and so it's interesting you know we had a conversation with an executive team we were working with recently where this that whole stage was set like we need more accountability people aren't taking this seriously blah blah and we asked the question to the executive team I said hey how when was the last time you all sat down with your immediate direct reports and really had a conversation about setting clear expectations for the year and they all just kind of had this confused look and awkwardly looked around the room and you know the CTO turned around and said you know I I don't think we've ever actually done that and and Ben and I said this is the first step to getting to where you want to go with accountability and so there's there's a whole setting expectations and also negotiating around

that too you know it's like I've seen people you know have a long vacation or a maternity leave or a paternity leave that like shocked everyone and actually materially affected their performance that year and it could have easily it was a conversation that could have easily been taken a note on a piece of paper somewhere six months ago so it wasn't a surprise to anybody um so these things are are it's really all about like sitting down and and having that conversation up front and I I think a lot of that burden does fall on the managers but I will say to direct reports as well if your manager doesn't know how to do that don't just roll with it right you know sit down and initiate that conversation conversation because you're the one who's gonna suffer in the end if that doesn't happen right so I wanna I wanna dive into the second part of what you were talking about with the feedback like um how would you reduce it and it was interesting because at first I was like okay hang on there's a framing thing here right is feedback a problem no but I also get what you're saying where

feedback is overhead you know from some perspective you can view feedback as overhead it's a it's a growth learning continuous Improvement based overhead but it's overhead so yeah so okay how would we reduce it and I thought through my career of like how do people get feedback when do they get feedback when have they not had to give feedback and what's interesting is encouraging curiosity and humility can reduce feedback because people will ask more questions and are more likely to have the conversation and assume they know what's going on up front so you were saying that there's a case where I'm managing somebody and they go and they do something I didn't I didn't expect well one we acknowledge that this is just a reality of knowledge work like we are in the space of creative Endeavor there are the we describe things in terms of problem experience engagement this sort of thing and there are multiple solutions to every single one of those problems and so if we have a specific solution in mind that we need to tell the other person that but if we don't even know we have a specific solution in mind we may not Discover it because we assume it's the one that

they're going to take and then they take a different one so one Clarity of expectation setting to what Aaron was saying and then the second one is the and this is something that actually I think I have seen more in the highly effective senior game developers across all disciplines that I've interacted with when they are asked to do something or solve a problem or whatever it is they will ask questions they will default to curiosity and they will start asking questions because we're not we're not on a factory line we're not laying bricks right like this is the wall it's defined lay bricks until the wall is this tall right that's it you know leave the space for this window this window okay cool we're done we are in a space of incredible creative endeavor um and so I think one is you can never get away from all the feedback but if it's happening more frequently you will have less of it because it's easier to course create course it's easier to course correct if you do it you know every two days for five minutes instead of having it all

like Aaron was saying build up and then it's this giant explosion nobody knows what to do what to do with and the second is if you don't want to get a lot of feedback one you should understand your own motivations for that but two be really curious be really humble ask a lot of questions clarify what people want from you um and so that would be a behavior that I could see at an organization that would reduce the amount of feedback that needs to be given because we're already we're proactively engaging in the conversation to understand what good looks like in any given situation got it great and to just close the loop on feedback I think the the last kind of kind of major issue I see with feedback is the dude or the the girl whoever who just cannot give feedback and so Something's Happened and they're seating right they're just so angry and upset and it's like well have you talked to Harry or

have you talked to Sally about this issue they're like I'm gonna do it I'm gonna do it and it's just the anger just continues to boil over and they haven't done it they haven't done it and then you know to to the points that you guys have made before it often does turn very punitive very angry and in my opinion that's I think both parties are equally to blame yeah whoever is at fault and the person if you're not giving the feedback you are at least 50 to blame if you cannot give that feedback but what kind of advice would you give for some of these people in you know they're they're actually quite a lot of them in my experience who just can't I the I don't know what it is like some people can just have those conversations and some people I I feel like they have all this I don't know tension and nervousness or whatever they just can't do it yeah it's everyone is everyone is different um yeah you know it's

um I'm I'm actually gonna avoid uh to too specific of an example here but um there's there's someone I know whose family uh has a um a distinct history of communicating passive aggressively and so the way that they sort of uh get out their negative thoughts or emotions or critical thoughts is through like subtle Jabs at each other and sort of like throwing a kind of a a mild but stinging comment into the room like kind of rolling a grenade into the room but but to to directly have a conversation is the most terrifying thing that any of them can imagine and they'll actually jab at each other be quite nasty a lot of the time when they have not nice things to say or they're frustrated um to just to avoid the direct conversation the reason I bring that up is because some people were literally raised either culturally or family culture or the place they went to school

or wherever in a place where like the idea of sitting down with some somebody and having a direct conversation is one of the most threatening things that could possibly happen right and so that's just one of many explanations I think for why a lot of people struggle with this it is absolutely a cultural thing and so if the cultures that you've been a part of in any system you've ever been a part of in the past have taught you that this sort of conversation is not a good thing or or is to be avoided or whatever um that's going to be with you you're going to carry that with you and you have to work with yourself to address that um that being said it I I'll be honest if I were starting a company tomorrow I would screen heavily for this I think Ben would screen heavily for this one of the things we this is one of the reasons why we tell startup Founders that like it's great to have a good game idea it's great to think about how big you want your team to be it's great to get

funding like you need to do all these things of course but like you that we often hear startup Founders having this conversation and saying well culture you know like whatever like I've got a pretty nice group of guys like we'll figure the details out later and I'm like this is one of those details you need to figure out now you can get the best creative director in the industry but if this person can't give or receive feedback worth a damn you are literally like walking along with a limp from day one like because some of the most critical conversations that will lay the foundation of your very business and product are going to be challenging they're going to be threatening and they're going to happen today so really think carefully about who you hire and because there are some people that it's not that they won't get there eventually but they may not be there today and the work required for your team to get them there today may not be an investment that you have the time or energy to make so it's like you I I we really try to encourage people to be

honest about this stuff it's like again we we lean so far into the skills like this person has the best skills like well they but no one can talk to them that's a huge problem so they're a little black box we just hope it works and uh that's that's not a direction I would go especially when I had a team of five people so the thing too if I if I were to have someone let's say I was managing someone discovered that was a trait that was part of that again cultural background I'd love to Aaron called that out very natural to end up in this space um it's train it's learned behavior the thing I would probably do if I realized that they're struggling to give feedback is again get them to start giving feedback to me let's start practicing okay you're gonna give me I'm I'm your manager I want you to give me critical feedback also I want you to start giving other people positive feedback all right let's let's hold on negative like neutral or positive express your

opinion about something a different way that you might do something oh that that might work here's another thing that might work what do you think right just get them in the idea of putting their opinion out into the world in a non-confrontational way um so positive things neutral things like differences not not critiques but just like oh here's a different way you might do that suggestions basically like here's a suggestion start giving people suggestions start giving people positive encouragement um and and start giving me critique right let's start building up your ability to do and realize that I'm not I'm not going to hate you for this because actually that's that's at the root of this is that if I have the direct conversation there there may have been a point in my past in my family or my upbringing or a previous job where I went to someone who had a direct conversation and from that point on they were mad at me it put this hole in our relationship and there's a lot that goes into psychology of relationships that like you know can be dived into here but if

you've had that happen it teaches you don't be direct don't be you know Joseph you you're able to do that you're able to go forward and be direct and probably there's people in your past that are like I don't like that Joseph guy um he was you know uh and and Aaron and I both have that too he made me have bad feelings yeah exactly right and and this is it is it's all like it's emotional level and you can view like oh what's the quality of the feedback and all this but what you're really trying to create is a place where that person realizes it's okay to do that and that they should be doing that and also once they realize they can do that for you okay here's here's critique to you and it's going to be like the smallest most useless stuff at first um Fine Again encourage encourage it it's in the right direction encourage the micro Behavior that's in the right direction you'll get more of it and eventually it's going to get to that point where you they're realizing I'm giving you feedback and you're appreciating it and then there might eventually going to be like okay cool you're gonna go give that critical feedback to that person you're struggling with and you're gonna we're

gonna you know I'm gonna coach you through the model and how you're gonna do that and then afterwards you're going to talk to me and by the way if they blow up in your face I want you to know I have your back um like I am going to I'm not going to be a manager you're just going to be like what you annoyed blah oh well you're clearly dumb like yep sorry nothing I no no I have your back I encourage this I train for this you you might even do more things to set that up now as Aaron said this is all a Time investment this is all a developmental investment you're making and so be aware this is time consuming and that's where I I would double down what Aaron said if I'm a startup and I think feedback is important I would screen for this heavily because I may not have time sorry I in some sense that would be such a priority thing I would have to make time it's a time and risk thing the risk is exponential at that stage as well if somebody's poor at giving and receiving feedback because you know if you all of a sudden things go well your prototype's great your VCS are like

let's do a you know a Serial whatever scale the team up to 100 and now you need to put seven people underneath that creative director who can't give or receive feedback it's like well now you just you just created a team-wide issue instead of just in one that one person struggles with right so yeah all right so let's shift over to performance evaluation and the objective is often tied to skills assessment and compensation so oftentimes a pretty sensitive topic and the ways in which performance evaluation that I've seen in my career have varied a bit and have not necessarily been super great and so I'm wondering if you guys could talk about in your experience that you know companies that you've worked at or where you've seen a good example um maybe you could talk about some of the issues and challenges and whether you have seen a good example of this done somewhere yeah can I just talk about the uh the what I consider in my

own mind there's a paradox here or not there's like there's a contradiction and it's that when you actually look at and I'll get to like how I think you should do this okay um in the studies that they've done on performance evaluation stuff almost always it seems like it's a net negative for the organization and it's really that's really sad and again you can see like well they're just not doing it right and yeah that's that's true probably in some cases in others it's like this is a really hard space and simultaneously it's really important because if you have no way to evaluate performance except informally um there the likelihood of you basically causing cultural microbiomes to start existing in your organization as different senior leaders or managers uh have different expectations and never level set and never sort of come together to talk about what good looks like is so bad so

it's I I view this as like that obviously the organization wants to be able to answer the question of how well are my people doing who are my best people like you know where do we need to invest work and all this different stuff and and I think but but the reality is that most companies even with their performance evaluation systems have no idea um they don't they don't know who their best people are they have no they they struggle to understand that um and and the systems that they put in place actually seem to cause more harm than good but the alternative is to just be operating in the chaos of not knowing or any individual leaders based assumptions like you're the CEO right so like whoever you like well obviously they're good and whoever you don't like well obviously they're bad that's honestly what a lot of this can boil down to in the absence of any of these systems and and I hope we can all have the humility to acknowledge like that's probably not a good system actually especially as organizations scale you you you shouldn't trust your judgment that much because we all do

bias towards the people that are like us and away from the people that are different than us and that's distinct from their ability to be productive um so anyway that that to me is like there's a there's a real challenge here there's like these tend not to work they tend to be really bad they tend to cause more harm than good and simultaneously I absolutely understand why organizations very much want to have them because they're trying to figure out who to promote and who to give raises to and what type of person we want to hire and all this different stuff it's a real it's a real conundrum uh yeah I don't think I've ever seen a performance evaluation system that was um that felt really good and very satisfying overall and I'm left to kind of take a step back and go ask two questions um one what is the purpose of the performance review right because um I I get really I get really um skeptical when I see stuff where it's like we have this massive job family and

it's like well how do we know if somebody's a producer G4 and when they're going to move up to a producer G6 and so we need the performer and I'm just like when we start talking about that stuff it almost feels like it's about um like the the pro the process it's like the cart before the horse the process is leading the organization um the the second thing I ask is like what are we signaling that we value through the performance evaluation and yeah depending on the way you execute the performance evaluation you're probably signaling some things deliberately and you're probably signaling some things non-deliberately and I know this sounds abstract but it's actually really important um a classic example of this is something Ben referenced earlier where it's like well we do the 360 review like if you do 360 reviews are very popular right now we have everyone go through this massive exhausting Gauntlet of feedback right before we then decide who gets paid more and less so whether or

not you know like when you go ask you're like well why don't we separate these well you know it makes sense and we have to do it this time and it's a lot of work so it's like you know we've got the financial people over here doing these things and it's like they have a lot of like reasonable Acts explanations for why they glued the two together here's the problem though you're a manager and you're giving you're writing feedback for one of the the people that you work with one of the other leaders you work with you recognize while you're writing this feedback that over the last six months you've had quite a bit of friction with this person and you've been too busy to talk to them about it so you're now in a pickle your rule for yourself is that you always want to have a conversation with somebody face to face before you start writing something on the permanent record but you it's you know three days from Christmas Eve the performance reviews are due in 72 hours and promotions are coming up right after that so what do you do well the obvious answer is you

just don't write that part down you keep that part to yourself and so now the entire system is breaking down right because of the incentives that are present there and again I I want to apologize for how abstract this is but like this is the kind of stuff that I think I would like to see leaders at companies thinking more about because you just you now you have a crappy 360 review that didn't capture some critical feedback um you had some probably follow-ups that if it were just about feedback only and not about promotions and stuff that they probably would have just been like well I need to schedule something with this person it's no big deal we're just going to do the 360s again in three months it it eases that that that friction um and then also yeah you have like an incredible recency bias as well so now you have 360 reviews where like a bunch of people are like oh crap I'm gonna be as nice as I can also because I don't want to screw up this person's promotion

I know the first thing their manager is going to do when they're talking about compensation increases pull this well this great information I have like available they're going to pull that up and so now you have the pressure of like well I don't want to be the reason that this person doesn't get a promotion or I don't want to like hand their boss ammunition to not give them a raise and so now all of a sudden again the whole system is now warping around the reality of what you the situation you've put people in and so I don't think any specific process solves that because now this is these are the processes that I think are actually the more well-intended ones from the from a cultural perspective the other ones which are literally just about checking boxes like the traditional corporate hierarchical structures um are it's a it's almost like it doesn't even really matter that much I feel like what the content is it's more about the system and more about the process um yeah I think again the question is the core question is like what is actually being signaled what are what are the real behaviors that are happening and what are the real outcomes

and are you actually getting the information you need like I that always bugged me after we did 360 reviews and I saw managers just walking around waving them around like look my person did really well they got great feedback on the 360 and I'm like did you not have any other information about their performance over the last year than that one document like did you do you not have us back to your point Joseph about like how to keep that Evergreen and what's the process like do you not have like a series of 90 Day plans where you can illustrate that they have or have not met their objectives set over the last year like where was your diligence before the performance reviewer the 360 review and and that always bugged me like when I would go like you know we'd have our meetings or we'd go in with the senior leaders into a room and we'd talk about like we'd share information on like who we thought was up for a promotion who we thought deserved a compensation increase we'd look at the ranges and everything like this and we'd sort of collaborate as a group of leaders and I'd often see leaders come in and say well look at all

the nice things everybody said about Bob on his 360. I'd be like yeah but Bob hasn't met an objective in like the last six months like just factually speaking like I'm I I have a bunch of people that work on the same projects that he does and they can get a single thing out of Bob I don't give a [ __ ] that like seven people thought he was a really nice guy and they'd be like oh really I didn't realize that and I'm like well okay so now we have a systemic breakdown here right so and again I'm I'm perhaps a little bit more shrewd than most managers right when it comes to this but it's like that's the those are the kinds of things I think about when I think about performance evaluation right is it's actually I think about 80 of it should be the work that the managers are doing every day to like set clear expectations um I really think if you that's one benefit of doing that if you're actually on top of that and proactive about that as a manager by the time performance evaluations come around it's easy yeah

easy so so I wanna I wanna try to answer your question because I said I was going to around like okay well how would you set up a system right because again it can be an organizational imperative one I think you don't do it if you're really small if you're if you're a startup or even moving up like you know I don't know what the number is probably somewhere between 50 and 500 somewhere in there this is going to start becoming an issue and I think one of the reasons is the performance evaluation is to Aaron's point it can be a level setting tool it it's attempting to some level to ensure fairness um in inside of the system so that one person who's you know politically influential isn't giving all their people quarter of a million dollar a year salaries um just because they happen to be really good friends with the CEO and they're good at raving about other people regardless of performance out of that space right um so you you have performance evaluations you sort of have you set expectations you you know you make the job family or whatever boxes when you're

very small don't worry about it um second thing we talked about the idea of if you have a formal feedback system instead of doing formal feedback systems and performance evaluation do it the other way do it right after do the feedback after we've decided who gets promoted or whatever right like if you're going to do them at the same time do the performance evaluation first and that is a clear signal also the managers that hey you don't get to wait till as Aaron was saying you've gotten a bunch of feedback from the 360 so that that basically you get to shortcut your responsibility to actually understand how your people are doing that is not okay you need to know that in an ongoing way so what we're going to do is we're going to line that up after and if the only thing you have to say in our performance evaluation conversations about like who's going to get a raise or not is like well five months ago let's say these people said nice things about them that's a you're failing as a manager um so then how do you actually do the performance evaluation bit like what do

you evaluate people on and here I I like to I've seen a bunch I've seen like potential skills technical capability all these different things suggested my two favorites are are you culturally aligned with the organization so do you manifest the behaviors we like to see in developers on a day-to-day basis that's one of your axes and the other one is something like performance or value delivered and I I that is not the same as skill set right you might be an amazing coder but if you're spending all your time coding things that aren't valuable that aren't solving the problem I don't care about that so I care about like your ability to contribute to Value delivery and your cultural alignment if you're off on cultural alignment over time you become you become someone who creates anti-pattern behavior and I don't want to promote someone who's creating anti-pattern Behavior right because if I promote you when you're culturally misaligned now I'm showing that to the company that you can get promoted while

being culturally misaligned and I encourage cultural misalignment through them performance regardless again of technical capability and cultural alignment if you're failing to deliver like Aaron was saying you might be the nicest guy and you might really be great at culture but your outcomes are terrible you are not able to perform you are not able to deliver value um in that world I take that person and I say like hey cool we're excited about your cultural alignment we have to work on these outcomes like how are we going to help you be more effective because if you imagine a company that has massive cultural everybody's giving each other feedback and we're all you know behaving in healthy ways and we're non-toxic team members but all the value is zero we're gonna go out of business so we need performance we need value delivered we need outcomes and we need cultural alignment and so those are the two axes I pick I don't care what your skill sets are I don't care and there's one that there's always funny people I've seen this done with like we'll we'll evaluate them based on their potential and I was

like that is it's it's I don't know I don't know if you follow any sports I watch some sports it's always funny when they have you know oh there's this phenome you know Prodigy 19 year old football star in Germany or something like that and they're like oh he's going to be the next you know Ronaldo or something you know and there's all this and they're all attempting to predict someone's potential and they are wrong so much more often than they are right um breaking someone's potential tries to predict the future in a way that we're just not well well set up to do but we think we are so I actually don't like it as a metric because we really think we can tell who's going to be great and who's not going to be great um but it's actually way harder than we think especially once that person starts getting training and starts giving getting growth opportunities and all the stuff so it's so yeah so don't pick potential I don't like potential as an axis I don't like uh technical capability as an axis I like performance slash value slash outcome cultural uh Behavior and just to maybe

uh go one level deeper in terms of like thinking about or maybe maybe we're actually to pull out a little bit and think about systems more generically so it does seem like the kind of General kinds of ways in which people might do performance evaluation systems could be as you stated then like um based upon skills or uh characteristics and then you can do a 360 and have different people evaluate you based upon those skills for the role the second would be setting up you know oftentimes quarterly targets so performance like I will do these things and you know hit these specific things um there are some companies who actually also use okrs as performance either even though I don't think they were designed for that but some people do and then there's also the very controversial uh methodology around Stack ranking which I think people got mad at a blizzard for

when you think about those different systems uh could you just maybe maybe you guys could just kind of highlight a few caveats or you know gotchas around those systems sure I actually really like the idea of having more quarterly updates or smaller term tactical plans roll up into the performance review because then um you know because of those two axes been mentioned the um results and culture I think you tend to get more of the culture stuff naturally through a 360. type environment and it helps cover the results side more effectively which I think frames the overall results as far as things like stack ranking and things like that there are some weird things out there yeah let me talk about okrs for a second um you're right they weren't designed for that and I I don't I I immediately ask myself like why are you trying to use that other than that okrs are really popular right now um like everyone loves okrs it's like one of those things that everyone just talks about all the time but I'm like

what is the specific value that okrs bring to performance evaluation perhaps I'm missing it I've I don't um I mean again I I like the idea of setting targets like managers setting targets with their people but like sit down and have the conversation and then set the targets and then write it down on a spreadsheet like write four bullets on a spreadshe doesn't have to be any more complicated than that as long as the people that are relevant understand what it means and it's clearly dotted lines back up into the organization's higher level goals you're good and like to to a large degree I tend to be like well there's 80 of that is human judgment no matter what system you build so if you don't trust the human Judgment of the humans in your system you have bigger problems than whether you're using okrs or not so I would just say like for me that's literally what I do I have like a little spreadsheet that I use it's like the 90-day plan and I sit down and I'm like okay what are the four things we're gonna knock out like four goals we're gonna hit and if they hit those

and I agree and they agree and their team agrees like we're winning like I don't need a specific goals framework for that I I don't think um as far as stack ranking there's a lot of this kind of stuff like it's it's weird so here's the problem with stack ranking unless your whole company is culturally exactly on the same page about what kinds of things go into the stack rank Ben and I have a phrase which is there's there's explicit culture which is the stuff that everyone talks about you know like you you can have a conversation written on the wall written on the wall and there's implicit culture which are all the behaviors that people actually do every day that no one really talks about the things that people [ __ ] about around the water cooler as sort of being the frustrating parts of working at that company um but but they're not written on the wall it's like you know like feedback is often indirect and highly political like that one's not going to be written on the wall right but like we all know that's how it actually is to work here

the point I'm making is like all of that stuff all of the explicit and the implicit is going to go into your stack ranking like if the truth about the way your company works is that if everyone really really likes Sally because she's super fun and she has a bar near her uh area of work and she you know and she's amazing and throws the best parties and and so you know but she like she hasn't delivered anything in a year like and she gets up high on the stack like she might be higher on the stack rank just because of that if that's something that you value but you don't talk about a lot so it's there there's or you know or if or if Jim like you know uh Ben said earlier he has a great backdoor relationship with the CEO and goes over to his house and has pool parties and watches his kids on the weekend or whatever like these things it sounds ridiculous but these things will find their way into your stack ranks if you don't have like clear cultural boundaries around this stuff so like I I

don't also the weird that's a weird kind of comparative system too that I'm like I'm not sure makes sense I I understand stand the intentions of it right it's like it helps you anchor with your high performers and things like that but again you better know and be confident in the criteria on which you're measuring high performance otherwise your whole system breaks before you even you also have discipline to discipline differences so when I think of yeah um stack ranking right okay so I'm going to stack rank do I stack ring across a team do I stack rank discipline most people will stack great discipline because they're like I'm comparing like to like yep okay cool what if I well one I might have game designers right and I've got some game designers that are working on a live service game and I've got some game designers that are working in r d okay cool what how easy is it for me to compare those two things because the the while they technically all touch inside of the design discipline the

connection between them hierarchically structurally is like you almost have to go up at the top of the org often before you reach it and so how aligned are these managers how much are they talking all these things and even if they are what if I did a really great job hiring designers and I've got like eight designers on my team and they're all killing it and their teams love them well now I stack rank them and one of them is in the bottom 10 percent right and there have been places like there have been places that have said if you're in the bottom 10 we have to fire you right um that's the rule that's our rule we stack rank we say bottom 10 middle 70 top 20 or whatever like you you know they usually do some sort of division around that someone's going to be at the bottom what if they're actually doing amazingly well it's just that everybody else is doing even better or to Aaron's Point has a manager who's better able to advocate for them yeah um because because the differences get really subtle simultaneously what if I have a discipline where everybody's really struggling um and again you hopefully don't have this but maybe if you were a startup and you weren't so worried about a particular discipline you hired in some

people their lower skill they were Associates like maybe straight out of college you just got a few people well now you have a set of people and someone's going to be like the best and so like okay well our rules say you know again what are you trying to do with a stack rank our rules say we need to promote you or we need to give you a raise or whatever the be careful about the structure replacing human judgment when it comes to these decisions and that's where the stack ranking can be can be troubled one of the things Ben and I will do a lot by the way that will introduce is a concept to organizations is the difference between role and title so like when we when we tend to think about titles we'll think about your discipline and sort of the core skill set that you bring to the table and that's mainly so that when other people look at your job title they can make a set of us generally true assumptions about what kind of skills you have and that's about it but like another great example of where this can break down that I've actually seen in many different types of performance evaluation systems is let's say you have a producer and producers are generally

like the processing people right they make the spreadsheets they do the jira projects that you're at your company they're they're the organizers The Operators what if you have a producer that had a creative director that got really really sick on their team and then they stepped into the the product role the creative Direction role they had a little bit of design chops or maybe not in the past and they did an amazing job but as a result they had to let drop a bunch of their like traditional sort of task management or operator responsibilities because they thought that the the creative role was actually the more important role for the team at that time and now all of a sudden they're dropping balls and what is traditionally seen as their core areas of responsibility and now they're being stack ranked lower even though they're actually technically adding more value to the organization and perhaps from another lens might even be more valuable as a producer because they're more flexible in more roles they can take on these are a lot of like the nuances and subtleties that like a lot of these very official systems tend to not take into account and that's where

again the human judgment thing is so important like I love the idea of an engineer stepping into a product role because that's what their team needs or like a producer that can you know can can do some creative direction as well or whatever it is I think that that flexibility in an organization and allowing teams to sort of dynamically collapse around certain roles and and being open about what value looks like and not like strictly boxing people into their specific like you know you're a producer so where's your spreadsheet like or whatever it is I think is actually a really good thing right there's um I'm sorry we're going on this one I'm not looking for a quote right now tell me how you'll measure I'll tell you all your I behave was that was that gold Rat or yeah gold Rat yeah okay okay cool so so Lei who goldthrad um wrote a book called the goal worth reading even now uh for sure one of the things he was quoted as saying is tell me how you measurement I'll tell you how I'll behave and that um that comes into all of this if people think if people

are stack ranked then they're going to try to be high in the stack rank so you have to be really careful how you stack rank that's how I would say that I'm not saying you could never do a stack Rank and have it work if you do it you better be really clear what's important because if what's important is to know the right people and all your people are going to try to figure out how to know the right people or they're going to get knocked out of your stack rank regardless of their effectiveness if it's technical capability you kind of asked earlier what if I have someone and I evaluate them based on their technical skill well if you do that then you're going to have everybody chasing technical skill and one of the challenges in video game development is technical skill does not equal good product necessarily I'm not saying you don't need some technical skill people need to be good like if I had nobody who knew how to develop games and I tried to make a game it wouldn't probably wouldn't go too well but I want people who are great game developers and great at adding value to the product and if their portfolio or their CV if they're an artist or something doesn't look quite as shiny because they're not as expert in a classically like art space I'm okay with that if

they're phenomenal at creating the concepts I need to get my game done so when you when you evaluate everybody on their technical aptitude it seems like it's really easy you're comparing like to like but the reality is that there's so much more to results than your ability to write code you know find bugs um create art all that different stuff and actually Joseph to to that point I think a stack rank one area I could see a stock crank being super valuable is if a bunch of senior leaders or managers got into a room to talk about the their overall discipline team and they wanted to sort of like stimulate a good conversation let's stack rank all of our people and then talk about why review like like to use that almost as a way to surface what we're valuing oh it's like okay um Tom thinks his guy is number one on the stack rank well why why does he think he talk his guys above these other four oh it turns out that he is focusing more on technical aptitude whereas I'm focused more on culture and like it could that those

kinds of conversations if you didn't use it outside of that room could actually be a really good way to get the juices flowing and expose some of those assumptions so you're saying it could be useful depending on the way it's utilized essentially yeah because ultimately it's a tool yep yeah and and the am like again that I pick on Amazon here I think they did away with this but when they had the like you know top 20 or 30 get a raise or a promotion or whatever and you know middle or 70 or whatever and then bottom 10 get fired a little seven percent like you know maybe a small bump like keep up with inflation bottom 10 you get fired when you create those systems it seems It's what what you are literally doing is removing human judgment from the evaluation of human behavior and you're letting the system do that now look maybe at scale they're like well that pays off for us in the long run I mean the military solved this whole problem by just saying everybody makes the same thing based on Rank and time and service period

so whether you're the best Captain the Army's ever seen or the worst if you've both been in for six years you make the same amount of money we don't care right adjusted for your location um based on cost of living that's it they they simplified the systems that there's no there's no need for any of these that they still do performance evaluations and reviews and they're quite uh inbred in the way they function like if you're not if anybody if anybody isn't top Block it's a huge red flag that something's wrong which is hilarious because top block I think is like top 10 it's like everybody knows that's a lie um so anyway so they've like their system becomes contrived around this um and I think a very negative way so yeah what are you using it for now there was a middle system that you mentioned you said technical evaluation something and then you said um yeah so basically evaluation around skills or characteristics one second would be run performance targets right so like yes yeah go ahead so okay so so

here's the thing I love like Aaron was saying setting expectations if you want to use a no care model find they were designed for organizational units like teams or groups not for individuals and I think there are some things that struggle when they get down to the individual level because if you think about the team versus the individual there's actually a much more variance in an individual's life and work than there is in a teams um so there's just a lot more like up and down and if someone has some things outside of work go wrong or even some things inside of work not go the way they expected it's easy for them to look like they're a dud because they didn't hit any of their goals that's also where again be careful of these systems replacing human judgment um because what people will do is they'll set easy goals and they'll find a way not only will they set easy goals but they'll find a way to convince you that their easy goals are hard if again tell me how you measure me I'll tell you I'll behave well it seems like at this organization people that set four goals and hit them all get raises and

promotions okay what are four goals I could set that I could swing past my manager that I could that I'm almost positive I could succeed at that will give me a raise and a promotion and by the way you can see that it's like oh that's manipulative in the system it's like no that's exact in some sense that's exactly what they should be doing for their own job security um because they also know that if I set four really hard goals and I hit none of them and no one cares that they were really hard goals uh if I'm punished for that you better believe I'm going to set easier goals next time and easier goals next time until I get to the point where I'm just knocking them all out and then you're gonna have an entire organization of people knocking out easy goals and nobody taking on the hard challenges and so this is where the conversation should be I want you I'm your manager I'm evaluating you I want you to set hard goals I want you to take on big challenges I want you to really try to move the company forward in whatever your area of expertise is and I also understand that you're not always going to get there just know that I'm going to help you and when we get to

the end we're going to talk about what happened and why you did or didn't get there and when you're in that world and you're encouraging that and they're not punished because of circumstances that were outside of their control and that sort of thing um you know that's when you can start really having uh again the human judgment yeah I think I think that that approach is critical like I I'm shocked at how few companies do that Joseph the um the sort of more regular evaluations because actually when you do them more often it reduces the weight of any one individual evaluation and so if you do take on a couple crushingly difficult challenges and you don't nail any of them then the next time you might be able to actually be like okay well let's do too hard and two reasonable and and so you can find a balance there between delivery and risk taking whereas like if it's all one big bang at the end of the year it's it's like it has to be good right I I also want to call you like you said at the beginning you're doing start stop continue monthly yeah and one kudos

to you because that's a Time investment and I know it is I know that that's like probably taking putting a lot of meetings on the calendar for you and the other managers in your organization and also I think it's probably a really good thing long term for your company but I'm just curious that's my opinion like how is it working for you I think you know it's it's it's funny because if I'm late they'll ask for it and so so I I do think because it isn't tracked and because you know it is in really solely intended for the benefit of the person receiving the feedback and it's not tied to Performance of value evaluation or compensation or anything like that I do feel like it's been better received and better acted upon at our company and and to your point like you know we are a small company so you know I've I do need to figure out performance evaluation at some point but um I would

say at least from a feedback perspective I think that as a startup we want a system that's not overly formalized or too heavy and it felt like this was a good way of just being focused on the person we just need you to get better try to get better and I don't know I I so far it's been I think more effective than anything I've tried or experienced in the in in my past but um you know I so I at least for us it seems to be working that's awesome that's awesome yeah way to go like I said it's it I know that stuff is time consuming and I really appreciate that you're putting that time and it's it's really cool that they that your people seem to be appreciating it too yeah well I I think it's not as time consuming if you make it a habit to like I I actually keep a a like a you know kind of like a notebook and like anytime I notice something I'll I'll write it down then I'll try to talk to them there and then it'll re-reappear and the start

stop continues so and I also think the other thing that is an advantage is like I've realized that some people take feedback in different ways and so if something happens and you talk to them verbal and then it shows up in the start stop continue written and so by having it kind of reinforced both ways when you get it both times I think it it that helps as well that's pretty cool I like that so if basically you're using discipline your own personal discipline in like a habit Loop like a pattern of behavior um to to make this few it's just it just doesn't feel as costly because it's like hey we know this is how we operate and I remember to jot my notes down that's that's really cool I love that great um so I thought I could ask one last question and this is a question around uh uniformity and so you know whatever system you have people are going to be behind that system so whether it's you know the evaluation of skills for me to anyone or

whether it's stack ranking or anything and so when it comes to like and and I actually have heard this might be a problem at Riot where you have like the hard dude and the easy dude and depending on who's giving the or who's executing the performance evaluation yeah you know it can have a pretty dramatic impact on whether you get that raise or whether you get the promotion and so like Aaron and I were hard managers yeah well so yeah it's it's it's really interesting what what you say and actually there were a lot of like the older school I think at Riot were harder in general um like there was a particular time where I'm like well the first thought I ever had which was where I was like I I've overheard water cool conversations of people saying things like well I might want to try to switch over to this person because people like this person really does promotions

like that was probably that didn't start happening until probably about halfway through my time at Riot and by the way I think that you know you can't avoid this sort of thing at any company as you especially when you scale this aggressively um and and and to be honest there's a lot of weird incentives there too because for that manager can look really good without and then not even in a malicious way but like if all their people are just constantly getting promoted all the time it looks like they're running an amazing team right so um and and um I had people give me advice at one point when I was switching projects saying hey you know you're switching managers you're switching parts of the organization like this isn't going to be as um beneficial for your career trajectory even though this problem is harder and the company's saying they really need this this isn't going to be this Pro this is a less percentage chance of turning out well for you as it would if you just stayed here with us and so um I

I think you do need uniformity but like when I think about uniformity I don't again think of like the job architecture that you have at like EA or something like that or even like honestly I think Riot has a pretty robust job architecture and I don't I don't nowadays and I don't know the details behind it so I can't really like assess it I've just been disconnected for too long but um you know certainly there is an over defining you can do with this stuff um what I would say though is the the uniformity I would want is like what does it mean to be like a senior producer or what does it mean to be a senior technical artist what does it mean to be a mid-level technical artist and specifically in terms of capability and ability to demonstrate certain kinds of results like I think there actually needs to be uniformity around that and interestingly I don't feel like Riot ever had that um and I'm I don't believe I've ever really seen a company that had that in a

way that was satisfying to me and again I think a lot of those jar back job architectures are supposed to solve that but they almost go they almost solve it by doing it in like a checklist way where it's like you did this you do that your team's this big and I'm like those are not the things that matter actually so interestingly enough one of the things Ben and I would do for example just one example one of the key differences between a mid-level producer and a senior level producer this is like our own brand that we created was when you become senior you need to understand the concepts that you apply on your teams deeply enough to be able to actively teach them to other teams and other producers that are more Junior than you secondly you need to be able to actually create Frameworks now within which teams can succeed it's not enough for you to go in and manage directly the tasks on your team to make sure the team's

staying efficient you need to be able to put in place a structure in a set of habits and a set of behaviors on that team that they can be more operationally efficient such that if you left that team they would still keep doing that that system would sustain itself Beyond you being there every day and they would know how to solve that problem and so there's a again there's a level of abstraction there which is like you're not just a doer anymore you're now a teacher and you're a systems Builder and a systems thinker and um and you can repeatably do on many different kinds of teams what you've proven you can do on a couple teams that you worked on personally so there's an elevation there and that's just one of many examples we had but like I think that those roles need to be defined in that way when it comes to specific outputs and capabilities that I that I don't um often see a lot of companies focus a lot on and and again there even with that there's still going to be a degree of nuance that requires human judgment

so Joseph can you restate the question real quick I just want to make sure I captured it because I I want to figure out there's anything I have relevant there sure yeah basically like the concern is that there could be different managers who even with whatever system you're using have a different basis of judgment or evaluation whether it's characteristics or a set of performance Target to manage your a might be like oh that's incredibly difficult and says manager B may say oh that's super easy it's depending on the specific manager you have that's executing a performance evaluation the you know your mileage oh I see so you mean like that how a manager would assess certain outcomes being easy to do or hard to do or or but but overall like but that has an outcome in terms of like you know your performance evaluation will be different based upon the person yes yep so so this is where um what

Aaron was saying earlier um I love that he brought this up there was a he and I were both involved in conversations where you get together with a lot of other of the managers and hiring managers um around performance evaluation times or or and you basically try to level set um and this is one of the best ways to do what you're talking about um I think this person is high performing right and again clearly defined axes uh will be helpful here right so let's say you pick the ones I like you pick culture and then outcomes delivered right like soccer did you are you getting outcomes not good to you at aligning to the culture now someone is walking through why they think someone is or is not good in that space um like hey these these are the things that matter to us great you think this person's you know there's different systems you can use nine box 16 bucks totally like create several axes and have everybody jot down a number or whatever like there's different stuff you can do um

getting in that room and having that conversation will surface the disagreements especially by the managers that are near each other with different perspectives uh because I I was in those rooms sometimes and I would see somebody and they'd be like this person's amazing and I was like wait tell me why and you know the most common reason is the one that we've talked about a lot which is well everybody likes them but sometimes it was actually more subtle it was well they seem really good at doing this process stuff and I was like okay how has that led to outcomes and sometimes the answer was like crickets and I was like like what is it that they've actually done who have they helped you know we're as leaders in our case we were producers right so we're talking about production production our producers are generally Force multipliers not direct ads to the product talk to me about how they force multiplied their organization well you know they like refined the process

needed yet but did tell me why that helped tell me why that helped everybody and sometimes it was we don't actually know but it seemed like they were really good at jira or Excel or fabric I like that you brought that up then because that meeting will help surface that so now that that person might have just ran and gave that person a promotion but now because there was accountability amongst their peers at the same level it was like well hey wait a second you're about to promote this person but this person has done the same amount of process work and delivered the results or still or whatever you know or it's like they haven't done any process work but look at the outcomes from exactly leading look at the problems that they identified that were going to trip this whole giant you know 400 person project we were going to be a disaster and they spotted these problems that's somebody who I want to elevate and by the way when you talk about that too in the same way of like the impact feedback um in that conversation it's not you're dumb for thinking that person's great no it's hey what do we want to see more of

in our organization that's what should be the outcome of those sort of like the easy and the hard manager need to align on what do we want to see more of really really great spreadsheets and processes and favro boards and Trello boards and is that is that what we want more of because you know if we're if you find yours that a couple people want to promote and again this is a very producery example but or or do you want um do you want more of the producer that goes and actually like like you know air drops into teams and really helps them clean up their overall systems and focus on what's valuable and deliver faster and like all the like very tangible outcomes right right and I think I've seen this in art sometimes manifest as wow they did such high quality Concepts and did we need high quality Concepts we were ideating this other person produced like 74 rough Concepts that got us to an answer right they got us to somewhere we were going while this person was very

carefully tinkering and very slowly and not iterating well Ben and I reality was so beautiful yeah we talk a lot about this actually but that and this is another this can be another challenge again with stock ranking and it's funny when I when I have an allergic reaction to stock ranking as a tool that is used broadly and organizationally and it's not but it is funny it's not because of stock ranking itself it's because of the way that I see most leaders in game development thinking it's like I I can't tell you how many times I've heard somebody be like this person is the best artist or this person is the best engineer like you will never find an engineer like this guy I mean he like the the like everything all the code he touches turns to gold and it's like therefore they are an ad to the company and it's like I I have seen many situations not even just a few where the person with the most technical aptitude was not the highest outcome producer on

their team and like that that is not an uncommon scenario in game development so I always worry when I hear people like there's there's an almost obsession with technical aptitude and game development like if I I think there's a lot of leaders out there who really believe if I have the best artists the best Engineers the best VFX people the best you know like that I will create the best product and I'm like that is not a one-to-one in my mind at all like not even close and in fact I've seen many situations where I worked with the team of like lower technical aptitude people um perhaps more Junior and they were because of their willingness to collaborate and generally be more agreeable with each other and like actually like lean into like working as a team and being open about how to solve problems and not being stuck in their ways or whatever they were able to run circles around the quote-unquote senior team and so it's it's uh it's something to be careful about I think yeah for

leaders nothing nothing wrong with being a senior just like there's nothing wrong with being Junior it's just it's just be be aware that there's actually a collaboration is more important than technical aptitude your ability to collaborate well and deliver actual outcomes um and it takes a lot of humility uh to I think do that well so yeah I think that's the answer that I would give to your question of how do you kind of try to level set and it is we'll bring those people together in a room and have them talk about it yeah one of the one of the challenges with all these systems is that we are humans we're in human systems and human systems have politics yep um and they have influence games and they have like somebody is friends with a CEO and somebody else's and all this different stuff and you know that you can't fix that through any process um you can attempt to mitigate it and if you if you focus hard and carefully on culture and behavior you can minimize

the impacts um but it's always going to be there to some extent and I think that's one of the other things is like aim aim for a a fair just Equitable system right and try to really really really try to get there but also recognize that you're probably never going to and it's not that's because we're humans it's not it's not like it's not like you failed you know because someone who should have been promoted didn't get promoted and someone who shouldn't have been promoted to get the promotion or whatever um you know if you if that becomes obvious fix it try to fix it um but also like man we we make we all make so many mistakes every day um and and even even when when politics in the influence game is largely benevolent and positive for an organization which I absolutely think it can be um it's it's still there and it's still it's still contorting just a little bit still warping just a little bit some of the the perfectly just system that

you're attempting to create in performance evaluations and feedback and all these things great well guys I think that would be the last question for me so maybe we could just wrap with like if you have any last message for our audience and how can they get in touch with you guys yeah I mean uh hit us up on social media I think is the the quickest way to get a hold of us um as far as like parting advice um it may not be comfortable advice but it is the truest advice for me and my career and and that consistently comes up with everybody I work with in our industry um human judgment good human judgment is the thing that I can most directly attribute to great games and high quality outcomes and um you need to build an organization that produces as much great human judgment as possible and uh stop trying to build processes that create that because they usually have the opposite

effect and if you don't know how to build it then just hire people who already have it and have that be the number one thing you screen for human judgment ask them questions about the decisions that they've made in their career and if you hear about those stories and go wow this person consistent he makes great decisions they might be a good fit for your team if you want to follow us uh we have a podcast and newsletter both are available via buildingbettergames.gg there's a building better games.gg newsletter if you want to sign up for the newsletter and the podcast is just building better games it's on Spotify and apple and all the major um podcast platforms awesome and I'll provide links in the show notes gentlemen I really want to thank you for this conversation you guys are incredibly deep thoughtful considerate guys and I I just really appreciate how much you you guys really care about your profession and I I can tell just how deeply you have thought about many of these issues so thank you very much and

for our audience we will catch you next time bye thank you bye-bye thank you [Music] foreign