Game Development Best Practices

hello everyone and welcome today we are going to be talking about game development methodologies and potentially best practices with how to structure and execute development we'll cover topics such as team structure typical game dev phases key engineering processes and finally best practices and tools and with us today to speak on these topics i feel very fortunate to have some industry heavyweights with us here today including first chris cobb co-founder and cto of pragma platform who also formerly worked as an engineering lead at riot games and held software development roles at both pop cap and microsoft next we have nomi relu i think that's how you pronounce your last name who is an associate producer at square enix montreal but who also worked in production and ubisoft on games like rainbow six siege for honor and assassin's creed origins and finally we have sean chen who is cto at century games the beijing-based games developer and

publisher and who also formerly founded mobile game studio out act and was formerly a technical director at roblox and worked in software engineering at tencent ea and ubisoft welcome everybody thank you thank you hi so so kind of just jumping right in i thought we could first talk about basically development teams in terms of structure roles and responsibilities and i know for example no i mean you used to work more on the console side and now kind of shifting into mobile but in terms of like the different kinds of team structures and rules that you guys are seeing can you speak to like the different things that you're seeing and maybe what can be more appropriate in certain environments relative to others and maybe if you do want to start know me sure um first of all i was quite surprised actually when i when i started on mobile uh at square enix that there

is a lot of stuff that's actually the same uh it's very similar right for team structure we're still making games so there's a lot of things like on the day-to-day that's um the same i feel what for me what changes the most like a team structure is the type of games you're making um so for example uh if you're making like a tripoli traditional game or i would assume if you're making like a a mobile uh uh premium game as well uh your team structure is gonna be fairly the same right it's like you work on a product no one sees it and then you relaunch but if you work on a gas game i think that's where you have a lot of difference in your structure because you're not gonna need the same expertise uh you're not necessarily gonna have to focus on the same team so i feel like that's where you you see a lot of the difference more on the type of games you make right and maybe even within mobile i would say like we're seeing kind of a couple of different structures

right like so for example uh for some high production games like some of the bigger studios like ea we see like these huge teams well huge in a relative sense where you might see teams that are between let's say 30 to 100 versus a smaller more compact more supercell style team which might be you know 10 or less or you know like between 10 to 15 for example and so uh maybe chris do you want to speak in term can you speak in terms of like some of the different kinds of like maybe team sizes that you've seen and maybe i don't know i know specific kinds of uh role differences that you've seen as well yeah i think when you start breaking down like you know who you need on a team it's going to definitely depend on the genre you're working in like naomi said i think that oftentimes when you talk about the team scale um there's two main factors that i would say you're going to have to think about which is one how you manage a team it definitely has to change based on your team size

yeah because when you're small and light you can just kind of you know work around a whiteboard and keep a to-do list whereas if you're managing team of 20 to you know 100 suddenly you need formal processes and check-ins and you have teams that depend on one another um you know in terms of the the kind of people you need i think that one factor beyond the genre that changes how many people you need is the difference between content development versus like systems so if you're trying to work with a smaller team and you're building a kind of recurring scheduled event-based systems you can oftentimes invest a smaller number of people in making something that is gonna you know go on a cycle right like a seasonal or battle pass kind of thing if you're going for high volume content where you have to produce tons and tons of new art new animations new game play elements that's when your team size starts to really expand um definitely on the like production and art side um but it will also increase you know the size of the engineering team you'd need right and shang i i know that in china what we've been kind of seeing are some of these like

mega teams like you know with companies like whether it's 10 cent timmy or i even heard wild rift had what for a mobile game but can you speak to some of those issues and like like even for some of these teams that we're seeing in china like these huge teams you know what maybe you could speak to some of your experience there yeah yeah it's i i don't think that's a china only uh scenario it's only basically if you want to make triple a games you need a huge team um for example in china example is uh gaming impact uh they as far as i know they have over 600 developers on this single game before they launched again so they probably have more technically i don't know um but i i i used to work for uh aaa games when i was a ea at ubisoft um i remember the largest team i worked with was madden and nfl you made

a serious game it i i think the team size uh exclude the central pack team it's already over a hundred i can't remember for sure but something like that at tencent we used to make a mmorpg um that's also a triple a quality game so that's the team size it's close to 100 and probably 80. but yeah but when you start to move to a smaller mobile based game you see the team size drastically decline and particularly at different phases and when you are at early stage you don't want a large team to work on the concept you know you're gonna you're gonna actually hurt your development you can't be i i we can talk about that later i'm sure you're gonna be on the topic but uh so usually you see a relatively smaller team work on concept work on iterations and

then when you prove the concept right when you plan to go massive development you you grow the team you hire a contractor you hire full-time employees the team size could could go larger i i definitely saw a team size over 50 for mobile game development even it's not a very complex mobile game like social game or type that type of game slg type of game will have large i mean as such as a chinese terminologies 4x game let's have a larger team um it could be close to 100 but but you rarely see a mobile team larger than a 100 because mobile is relatively uh lightweight um unless there are some type of mobile game like like the kings of uh arena i can remember owner of arena whatever and some game honor of kings

owner of kings that's right it it definitely has a much larger team because they need that they have a very high demand content development they have a higher standard of that so so it definitely varies of game types and platforms yes it sounds like if you have a heavy content cadence or maybe the other issue is i would assume that maybe the reason why ryan's wild rift team might be so big is they're trying to they're trying to speed things up they're trying to get something out fast that's right it makes sense right the size of the game is definitely um it requires that i mean because they're very sure about what they are going to do they basically just work on existing ip and make sure it works on the uh the platform it's definitely going to ship that they don't worry about any cost or anything like that they have the triple a experience that's why they can have a much much larger team but for other smaller teams they really can't handle that kind of size i thought that maybe the other aspect

of the team structure that we could talk about is with respect to like specific roles from my perspective in terms of on the more of the product side it feels like there's usually like this big difference with respect to like console or formerly console studios relative to mobile studios and even on the mobile side there's a difference when we talk about a product manager versus a producer and like what those roles mean or they entail and then it'd be great naomi if you could speak to maybe some of your experience at ubisoft and square square enix in terms of what does it mean to be a producer versus a pm and then from a technology perspective it'd be great to hear you know when it comes to like the development sprint like how do you maybe you could speak in terms of what a typical dev team structure looks like and then what would be the scope of responsibility for like you

know an engineering lead and and that that kind of thing but no i mean do you want to start in your experience what have you seen with respect to like producers versus pms and things like that yeah so like the pm job for me was definitely like a new thing when i joined square enix okay because i don't know if other aaa have this role but um at uh when i was at ubisoft it's not the role we had like the this world is failed but by many different people if you will yeah even if you have many decisions and everything so and like triple a is less uh maybe kpi driven than mobile right there's a big difference there so as a that that took some adjustment uh for me like to really well understand the role as well and how it interact with producers and also with the creative team right uh because um i think i think um

like the product manager is yeah it has some things that go with the producers as well like that overlaps a little bit or that is similar but i think there's a big creative side as well to it so i see i see this rule is really between in between like in between like a director and a producer a little bit again i don't know if it's like that in all companies but for us that's where it's okay it sits yeah and maybe from my own experience kind of like the the sort of patterns that i've seen is for example to your point it does seem that a lot of hd like pc console companies this role of pm is relatively new you generally have a producer and like within mobile teams it does seem like the big difference is really around in that you know some mobile game studios the role of a producer and a pm are different it does seem that the

companies that have more of a console legacy structure they're usually producer driven meaning the producers make a lot of the calls with respect to like features backlog product vision things like that whereas the more mobile-centric companies it's more pm and then what we're also seeing is like like the interface between a pm and producer to engineering is often different at different gaming companies as well where whereas when like at a company like zynga a pm may never directly interface with an engineer and you have a producer instead as that interface whereas at other gaming companies that are especially the ones that are smaller and more full stack you have the pms who may be directly interfacing with engineering overseeing development sprints running the scrum things like things of that nature but maybe sean or chris can you talk about your experience and maybe if you guys also have any thoughts in terms of

what you've seen maybe some thoughts in terms of your preferences or what you think are there's a better structure for whether it's you know on the console side or the mobile side maybe starting with you chris yeah so i mean one of the difficulties with this conversation i think is that every studio has its own names and so you'll yeah it's really tough to standardize what it means but like if you zoom all the way out i think that you've got some kind of vision holder and that's like the person more on the creative side you know trying to bring trying to imagine like what the end you know experience is going to be for players and then you've got to have somebody who's helping run teams and so whether that's the same person or whether you know and and i think that this gets into the the different flavors or styles so yes at riot we had a formal product manager on a team but they were purely from the product lens of prioritization and then we had a separate dev manager or probably what most people think of as a producer doing the like running the team scrum but for heavily

engine like engineer oriented teams then the tech lead was very involved in that process and prioritization too because there's so many technical decisions that it you know dictate what order you have to build things in so you know i think that when you get down to it there's like a couple key questions that if you can answer well i think that it creates a lot of clarity for the team and if you get it muddled then people can get confused and part of that is ultimately who is making the decision of what the team is focused on right um but second that sometimes doesn't get account for is like how much resources you're going to invest in this effort right so on the one hand you say well we have to have this thing it has to exist but the question is well is that a two-week effort or two-month effort and a lot of that's going to come down to some of the details of how you're going to execute which means that the engineers need to be involved at least in that conversation um so in general i just say like when when i've seen it go wrong is when um it's unclear who gets to make that final decision on what the team works on but but a little bit more subtle is um

if it gets uh kind of unclear um how much investment the the studio is ready to make on a particular feature and so i think when you're smaller or it's earlier and you're just trying to iterate quickly you really just have to decide what's the most important thing and how long are we going to spend experimenting once you get into that full production cycle then it has to become a little bit more mechanical and predictable but at that point you want to solve some of your big you know risks or tech issue areas so that you can actually build a schedule i think teams that want to try to build a schedule before they've answered enough questions oftentimes are just gonna you know suffer from picking a date and finding out that there's way too much iteration happening for that to be really reliable so that gets back into those development phases um so and sean what are your thoughts and maybe if you could like describe for a typical let's say mobile game engineering team what would that team kind of look like in terms of roles the number of heads or something like that right sounds good yeah i was about to say

chris pretty much summarize what i wanted to say um i actually can't speak to aspect which is a little bit new um one is the the culture difference i think the western uh team and eastern team they are still quite different um which i wanted to mention a little bit the other is we can definitely talk about the tech team structure after that so the first thing is like the the the eastern team mostly originated from the japanese video game development is uh um basically they are producer driven um so you you basically see a producer in the team who's a is the biggest guy over there and make all the decisions and the producer is like uh doing the western world executive producer but in my opinion it's it's more than exact producer it's basically just the owner of this entire product entire product line entire team right like the morale driven and everything so like this guy makes all the decisions

his vision is important and uh or her he or her vision is important and uh and and that that's that's it and this is this is basically the uh the spirit of this entire team uh you you mostly see like when sometimes they release the game and uh you see a lot of fans uh they want the the the copy from the city from the producer this is culture kind of thing it's very similar in china and the producer producers number one guy in this entire team you know for example in our company the producer owns pnl of the game and makes all the decisions and we do kind of the review with the producer directly even the tech review the producer participating that most of the time that person needs to know everything this is quite different from the western game development and western people usually own what they they're responsible for like the tech director on the tech

and art director on the art and producer mostly do the production things like that but in eastern well it's always like the top down structure and produce owns everything and then the art director and the tech lead those things like that so so this is the number one thing i wanted to mention the second is we definitely talk about the the tax team um so i think it's structurally pretty similar i mean i mean there is not much culture difference because i used to work with those cross-competent developments and think the easiest thing that we can connect both sides is the tech so people speak the same language and it's much easier and the structure is easier always like a a tech lead making decisions and doing the infrastructure designs um tech designs and then like senior people doing kind of senior work and junior people work on other like uh other stuff

so it's very straightforward for the tech team so so basically one small difference is which only when i started working in china i know this is like most chinese game development development studio uh they tend to uh separate the client team and the server team and there is always a server team lead and client team lead in the team so it's not quite the case in at least in us when i was working there like we have gameplay design uh gameplay programmers and they they work on they could be full stack or they could just work on gameplay code um there are some like when i was at ea was in uh the online teams online team that would not only work on the server stuff but also the client uh stuff that related to the online game play so uh so that's kind of a little bit

different because in sims in my opinion it seems in in the eastern culture like people tend to make the responsibility climb the boundary much clearer so people will focus on their stuff which could be a good stuff but i personally wanted to in my opinion like i think a team that they collaborate more usually be more efficient so i'm personally encouraging team to to cross the boundary as much as possible not just to like focus on your stuff um so so that's my opinion all right and maybe kind of shifting to the next topic which is really around different phases of game development i think that at least on the mobile side there's generally some notion of pre-production alpha beta soft launch hard launch but what the objectives are and how whether it's the team size or the

specific objectives of each of those different development phases are different based upon the company and so i thought we could just kind of talk about how it may differ for each of you and nomi do you want to start in terms of like what you're seeing whether it's at square square enix or ubisoft or just more generally speaking uh yeah um i think like what you describe is is pretty much also standard for um for aaa at least my experience and and triple a right we have conception before pre-prod as well like when your team size is really small uh and it's a very waterfall what i've found so far at least how we're doing it at square enix right now in montreal we have some phases that overlap for example on our side we've uh we've decided to go live very early um so it means that we're doing production while also being in beta phase in life so we're really doing like our

learning our live ups while also continuing to make to build the game itself um so that's something that um i think slowly um like big triple a company or at least like a pc company are doing more and more uh but i find it like much easier on mobile maybe because our teams are smaller as well uh and there's like a proper infrastructure as well like on google where you can be on beta track and stuff like that um so for me that was a a big change and it's also there's big gains of going live really fast because like you can have real-time uh feedback and you can really see like is our infrastructure holding up really well or like do we have to change our matchmaking or network uh structure or anything uh but it also brings like a complexity for the team and i think we're gonna talk about that later but like how do we structure the work

and how do we make it clear to people what are the priorities and everything because of your because we're doing two phases at the same time right and so to some degree if i'm hearing you correctly no i mean by trying to go live as quickly as possible what you may be doing is just trying to like you're trying to get as much market feedback as quickly as possible so to some degree the way that you structure your pre-production relative to let's say your your alpha and beta might be a little bit different because you're just because if you do value that market feedback you may be kind of speeding through those phases is that is that fair to say or um i think it's not necessarily speeding through them but more doing them in parallel okay um so because for example for us how i would like visualize it is that basically our beta phase is feeding our production phase okay so we still have production goal but because we have constant feedback because we're released fairly often as

well to the public so that means that we can really iterate as soon um as we want on things while also working on the more long-term things which if you think about it is not really far from what a gas game do for example what we did on on rainbow how we worked is a little bit like that right because you always work on stuff for the future while also maintaining everything that's live and doing your operations and your events and everything so it's a little bit like that but to a smaller scale got it chris what about you how do you think about those different development phases yeah well i definitely would say that um what shang mentioned earlier on about keeping that team size small during that exploratory phase is essential um you know the ability for a small group of people to experiment and iterate quickly is really important we had one r d team that um you know ended up kind of i guess they shut down one experiment and moved

people onto new projects but they were doing a retrospective on like what that looked like and what that meant and one of the key insights they had was they had grown um from you know six people up to i think 20 or 30. and they said that once they got to that size it was so hard in that earliest exploratory phase to get any consensus to get any buy-in because everyone's going to have their own ideas so i definitely think that there's that key aspect of you know starting small early on and then i think that it's it is a hard transition though as you go into like pre-prod because the goal with pre-prod i would say is about trying to get the engine of content development or the process of content development built and that is engineering work to build systems and tools for content developers but you're still trying to iterate encore systems and feedback and so if you try to create a pipeline or a system too early then it makes it harder to continue iterating and adapting however if you don't build those tools for yourself especially if you're pursuing like a games as a service title you know that gets really painful later because maybe

you've you know kind of managed to produce enough content for your launch day but if those systems aren't already online by the time you go live then suddenly you know that next content release becomes a real struggle so the balancing act i think as noemia was describing it's it's both working on those long-term things and making it easier for the team to work and still trying to produce value and that is i feel like it's a nice edge you have to constantly try to balance um you know if you over invest in short-term content then you might find yourself in a really bad spot where you cannot keep your content pipeline going on the other hand if you spend all your time trying to bake these really robust systems you might find that that system is shaped in a way that doesn't let you iterate on your gameplay features so i i do think that that's the the fundamental trade and uh it's definitely not not an easy one but it's kind of um it's a fun journey too sean what do you think so yeah i mean echo on that um i i think one thing i was trying to emphasize with my teams are

like process and it's very interesting that we're talking about that right now is even the team is small you should still follow some kind of guideline you you don't just like be ad hoc and you you you could be kind of agile but you you you don't want to be making a mess for the future development so um so a few things we want to do is first you you need to have a i think we are going to talk about that later but i can give a kind of an early sneak peek inside you you wanted to have some kind of process defined for your future development and you try to stick with that process basically for example if you want to run a daily build you need to have that like ci cd set up so you you wanted to make sure like in

the future when you grow the team uh people will know where to get the build and they need to follow certain process so they don't really break the build they don't don't cause troubles to others and like certain guidelines need to be followed and also like the uh the the asset creation part is similar like code coding standard and the arch asset creation and we have all the standard like setup that you need to follow that it's not taking a lot of your time and it's probably just like you need some time to read that and you just when you work on your stuff make sure like you you stick with that that's it um but if you don't do that in the early stage uh you're gonna find it very very hard in the future to uh to kind of uh kind of optimize that and uh that's like always the number one thing i talk to the team even the team size is a small and very early stage because if you don't keep you emphasize these

kind of things people tend to go wild they always think like i need to get this done really quick i have so many ideas i want to read on that and they because they don't they don't care much about the future so they actually people tend to focus on what they are seeing like even when you play a game right and when we do game design we want to always provide the the shortest feedback to the player so they they immediately get excited and they keep playing okay instead right you you but you you need to but it's it's for professional development and you need to emphasize that like you need to think a little bit ahead you don't always focus on what you are seeing right now so uh so yeah that's that's what i wanted so i think the most important thing for uh for process okay yeah and maybe just for the audience i can kind of give my own views in terms of game development phases and the way we do things at our game studio which is that for us like we view pre-production

as really about the elimination of key risks so if we have a game design and you know we have a game that we want to make what are all the key risks so that you know the top three four or five things that can kill this game you know is it you know is it the control scheme is it a monetization mechanic is it a specific kind of feature and then we try to build prototypes in pre-production against those key risks and then for us after that alpha for us is more about the core loop and the key features like building those out and seeing if those work or not beta is around like developing you know at least a initial layer of of content soft launch then is about filling out all the content and about proving unitary economics so on a per player basis can we make money and then a hard launch is scaling the project and making sure that we can you know uh fill you know fulfill the economics on a project basis so that's that's kind of how

we view it but know me i thought it's interesting how you guys are trying to go live as soon as possible and and having the dual tracks in terms of development one additional thing that we are trying to do right now is to you know we're still pretty early in development but we are trying to set up like a very tiny additional prototype team so that not only do we have pre-production and trying to uh trying to uh trying to trying to gain a better characterization or understanding of key risks in pre-production but then also having like a monthly prototype team which might be just like two or three people where we continuously start thinking think about risks for the game and then every month build like a small prototype we're we're trying to uh we're trying to build that out right now as well it's really interesting um and i think like the like um when you describe all your faces i think also like the goal when you go live super early is quite different right we're not

gonna look at uh monetization kpi for example and stuff like that um but you're still like it's more gameplay focused and like fun focus and retention and seeing things like that but um yeah and it's also it's a great place to prototype as well because um it's lower risk if you send something that yes after all it didn't work as well as we intended maybe or like we tried something i know on rainbow we had the test server which is remind me a little bit of this in the philosophy behind it yeah when we would send early early features right i i do think it's very interesting though because the ubisoft approach versus the approach my studio takes like the if you think about it strategically like the way that you that those two approaches are developed you actually have to have a different set of expertise right so like because the way that you guys are going you're probably going to want

to be more data oriented having facilities to do multivariate testing being able to get that feedback from your players and so you need that infrastructure and you've got more of a data orientation whereas with us we're a little bit more internally driven right more intuition focused we believe that here's a risk we and and for us because we're not getting data from you know from from the public we're building prototypes that we are intuitively through our own intuition are trying to make judgment calls on for for example and now now there's some things we're going to run tests on but by and large it's going to be more intuition driven so i i thought that's kind of interesting these these few different approaches yeah that does um i was just going to jump in to say that that that reminds me of um when you talk about collecting feedback from players and how you're trying to iterate as a team there's always that challenge of you know do you let every point of feedback you know redirect the team's priorities and change everything or do you have to kind of stick to your intuition or or you know you

kind of the direction you've set and just like the rest of this activity it is very much um one of balance and trying to find that right uh thing but there's a couple key like you know i guess bits of advice i've received over the years that have been really helpful and one of those is if you know the problems your team is facing you know when you talk about addressing these key risks if you already have a clear picture of well this thing doesn't yet look the way we we think it needs to or you know it has these essential characteristics it's okay to wait on feedback at least for some time it's it's kind of like you know don't have somebody um revise an essay you're writing if you haven't done your own revision pass and taking care of the easy stuff that you already know um so you know just as with everything you're trying to strike that balance but it is okay to kind of um another one is like if you're doing player labs or you're doing internal testing if if three testers have already tripped over one bug or one problem or one friction point you don't need the other five testers or ten to go through that same experience

so you can say hey we have already got this and then help them along so there's a lot of theory around how you run you know tests and labs and generally you want to stand back and let the player experience what you've created without interfering but at the same time once you've gotten the feedback you don't need to kind of continue receiving it so i think that as you talk about collecting input from players it's okay to say well we've gotten that feedback we understand this or hey we already know about this problem so we don't want our players to have to suffer through these you know friction points that we already know about so that's definitely an area that i think it is good to trust the team and and you know yeah kind of embrace what you know i think you make a strong point right in terms of how do you use the data right that you do have to have a balance both intuition as well as taking what the data is telling you so that's that's definitely something that is both sort of art and science and kind of the hard part about making games yeah and and there's a i think that's a whole other probably uh recording but there's a big team management component to it as well i

think as soon as you're a live game or have life part because the devs are gonna read the forum and how do you manage that right like you said chris like so not every comment is uh treated as like something you need to act upon right now but that's a whole other discussion okay and maybe moving to the next topic i know we're kind of getting short on time here but definitely for me like these are all super interesting topics but the next topic to think about i wanted to discuss key engineering processes that a development team should have now whether it's you know code reviews sean you mentioned like a daily build process i don't know if you're using jenkins or cloud build or however you're doing it but like in terms of more on the engineering side actually so on the engineering side whether it's ci code review things like that what are important engineering practices and know me i don't know if you have any thoughts on that from an engineering

perspective but uh maybe we start with you no probably not as much because i am not a tech person uh but uh but i used to be a closer which is the person who arrived at the end to make sure everything is wrapped up and we can ship it and code reviews for me are bust okay always chris i guess i'll just quickly say two parts that are i think really important that the first is you have to be able to run your game and i know that that sounds obvious um but when teams go too deep on like when they have you know an 18 different branches and different copies and clones and you've got all at the end of the day you're shipping one video game which means all the code all the art all the assets have to be combined and i think getting the team to practice building their product and running it every day or whatever that cadence is but but as frequently as you possibly can is essential and um you know i'll confess that

early on with pragma even though we're only doing you know the back end part of video game development we kind of skipped a couple steps on getting some of those ci systems and those build systems set up because we were working closely with a game partner so we just kind of leaned on that but quickly we ran into all the friction that is like hey but if we can't take it end to end on our side it's hard for us to know whether it works or not and even though it sounds obvious um it's really important and so i would just say that really and it gets into the ci and cd part but it also gets into how you manage the code base and if you let people you know different parts of the team work for months at a time separate and try to glue it all back together you're going to find that you're going to end up paying more cost in integration than if you help the team practice integrating frequently and again if you're only shipping a single game product then you know you only get value when all the code and all of it's packaged together and you can play your game so all right what about you sean so it sounds like ci seems to be something that that both of you guys agree is relatively right right yeah so so yeah

right because right now i'm in a relatively large uh company because we have many teams uh so one advantage we have is uh we have the uh uh the luxury to have a central attention and they can focus on certain stuff we have for example central back-end central platform essential client essential graphic so they will be able to deliver certain packages and for example the cicd system we are building that with our central banking team so they it's not just before the daily integrations also for game publishing so they actually the entire thing is uh integrated into that so we're still working on that it's not really there but uh but the the blueprint is pretty big beautiful so i wanted to brag a little bit about that so uh because ultimately when you are developing you wanted to simulate as

much as possible with the live environment um things like that because you don't want to build something that you tested fine locally when you ship that to the live environment there's all kinds of different weird issues so since we are for example we are working we're using a docker and kubernetes to uh to do the entire cice and the publishing and it works very well with the local testing because and because the local test environment is going to be very similar to the live environment so our central qa team will be able to intervene testers found their own servers and do the tests by themselves without even asking the game team to set up server for them or the ops team to to set up a live environment for them because they're going to be able to spawn this similar environment and doing the test so i would say a smaller team you really don't have the opportunity to do that because it's really time consuming because you you

could spend like several months to just to get one bit of stuff done um so so smart team i would encourage them to really just focus on using leverage those uh open source uh solutions like jenkins or team city and help them to build the pipeline and follow the best practices some people other people already show them right because the easiest thing is to copy right it's like you you just need to copy that and tailor a little bit to your team because different team always have their own situations you you can't do a perfect copy because otherwise there's we're not here to discuss about process right it's always uh some some kind of specific uh issues for example your team could be too large a team could too small um and you have like missing key persons things like that and you just want to tailor the process and the the

tools for your team and i think that that should work in general and uh but but yeah ci is something i think is pretty important uh i i'm not a big fan of code review i think large team or the games already on uh is live ops in live ups large gaming live ops uh they probably want to do code review when i was at ea working on madden uh it's a console game you know oh old days that when you ship when you you ship the the disc you you won't be able to to do any patch right on like today like i was always kidding like the programmer got lazy lazy and lazy here because they they always can fix their stuff certain ways i mean old days there's no way when you when you ship or call that gold right we celebrate that i mean people take one day off uh off to gold because after that you really don't have anything to do you literally just

can't work on anything so people tend to be very uh like strict about the uh the code quality and so that's why it's very important to do code review in those kind of circumstance and similar similarly for a large game uh like it's already live ups you want to make sure your your your update doesn't break the game and it's very important i would encourage code reviewing for that type of thing but for the early game development prototyping even like early development even massive development for a new game i don't think it's important you it's up to you i mean certain team there's few like quality is the key go ahead with that but i'm just not that type of person i think uh efficiency is more important but would you also think that it also has to do with situational context as well right so if you have like a small team of super veteran programmers coder view yeah a little

less important but if you have a mixed team with some junior people who need to also learn then maybe you also do code review and then you could also just think about you know what parts like there's this critical part of code you know your netcode or your whatever right like maybe you want to code review that part your ui your other code yeah it's just like whatever that's a very good point yeah i i was i was very generic uh it's definitely gonna be case by case uh for example if you're a new hire join the company for just one month you want to make sure and at least you use code review to to review this person right you think this person is culturally fit with your engineering team or not i mean this is the easiest way to check that uh second you don't want that person to build up you break the build um and uh yeah so that's definitely important and certain certain core uh system uh you you do want to make sure it's not uh like some certain bug gonna be tight over there

um so yeah i agree with you it's always a case by case yeah so um yeah just to talk a little bit like um you know we have an interesting group in terms of uh having a lot of experience on the team uh but you know you you'll find it with a lot of companies you'll hear things like well let's make sure that we don't have um you know single points of failure right so you want some you know more than one person to know an area but i'd say the most natural outcome is that you're going to end up having one expert that owns one area of your system so you know our group has gone pretty deep down the kind of extreme programming like very formal kind of agile processes where we practice test driven development and um and pair programming i would say that that's not a common approach but the areas that i've seen that be really successful are it naturally addresses some of those questions about single points of failure it naturally builds the code review into the process of having people work on it um there's definitely the production tension of like hey are we working fast

enough or not but one of those areas that i guess we've leaned on in particular and and some of this emphasis is on back end development when you're running a 3d physics engine and doing simulation doing very controlled automated tests can be a lot more difficult so there's different tools for different jobs but um one area we've seen though is that by reducing defects and shipping fewer bugs um we definitely have seen an uplift in long-term efficiency so you know we had different product teams throughout riot for example and some of those teams if if they had issues with quality uh in a few circumstances we had teams saying things like well we're spending 50 percent of our time on live support and dealing with problems you know and so you say well half of the team's effort is now spent on solving the problems of the things that you know were were built before um whereas for a couple of um teams that i worked with you know we were able to maintain that quality bar such that in general it was you know less than a 10 drag it was something like hey we might have a couple two or three incidents a year this is in the context of a live service game so there is something i guess there is a case to be made for quality which says

if the team is not having to constantly go back to previously shipped features and fix and repair things then you can kind of have that steady progression of like hey you know we can always build and ship new features um which is great um but it is going to be dependent upon which area the code base you're in and there's definitely areas for which some of these patterns are a lot more difficult um like in the middle of a game engine and chris when you think about whether it's like a lead driven code review process versus peer base versus i guess what you're doing now with pair programming can you speak to like when you think one is more appropriate versus another or that kind of thing yeah i think they'll all have benefits i mean just getting feedback on things like names and structure is going to just improve the quality of the code because it makes it easier for the next person to come in and read it so even peer reviews can be really valuable um i definitely agree that you know having reviewing new folks or less experienced folks can be a benefit just because it gives them you know teaching feedback loop which i think has a lot of value one area that we're actively trying to explore and discuss is about those coding standards which is like

you know sometimes you'll get arguments about formatting and that's not something i care a huge amount about but there's like a deeper level to that which is like how do we how do we structure and organize things how do we name things and even though that sounds small um it it does make a big difference for the team especially when you get to a larger team size it's like if we have a set of there's the folder structures here how things get named here how things you know work together then you learn how one area of the system works and then you can translate that that helps you move your team around engineers can work on other groups stuff um so i do think that there's a case to be made and that should probably be lead driven because i've yet to see a consensus-driven approach kind of that outcome got it yeah i wanted to echo on that that's exactly what i was talking about earlier like it's a it's something something small you need to do well in order to scale and uh typically that's lead driven because it's just like a it's because it's there's no best or bad like no good or bad it's just like

once that when standard everyone follows that's it in the future like everyone on the same page it's much easier for communication there's no gap so that's it naomi did you want to say something yeah i my answer was way too uh it generalized too much i'm sorry because i'm like i spent the last five years on live games so for sure that that paints my lens of what we need but also do you feel like the bigger the team the more review they should because um it's fairly new that i'm on smaller team um back at ubisoft teams are well known to be really big so i i wonder if it's colored our processes and our best practices right for code reviews because you have a lot more dependencies if you have i don't know like hundreds of people working on the same branch like three four five hundred people working on the same branch every day yeah i mean these guys are the ctos but

the bigger the team the more mix of junior versus senior and that kind of thing i would assume that you'd have a higher requirement for for code review and things like that but in summary sounds like the three sort of engineering processes we're talking about that are probably important to look at ci code review and then having some type of coding standard is what it sounds like we're saying here right okay so maybe we could move to the final topic which is essentially what should the development development methodology whether you're agile or waterfall in terms of the if you're sprint based whether it's like a a two-week sprint versus one week three weeks you know things like that like maybe we could talk about that and you know it's kind of funny though because i believe there was a study it was called the game outcomes study that kind of showed that there was actually no difference between teams that you know employed a waterfall

methodology versus some form of agile methodology in terms of outcomes for games but wondering what you guys think or what your experiences are in terms of what you believe are best practices when it comes to agile waterfall you know your your sprint structure and how you've actually kind of do and then we can kind of dive in into the sprint itself i and i know sean i've talked to you about this before as well in terms of how you structure it and all that kind of stuff but um no maybe we could start with you again and your thoughts in terms of like just this development practice methodology yeah i have so many opinions on this but i'll try to to be concise uh i think in a nutshell what's the best practice is whatever is going to work for your team at any given moment right um i think when we come up with our best practice for our team it depends on people's style on the

teams you have like is it a five-piece people team is it a 20 30 400 people team um but i think what is key is to adapt it as well right i think um proper agile works really well when you're in conception or pre-production where you want to do when you want to go fast you want a hittery quickly fail quickly if you have to as well um but once you get in a bigger team and your features take longer than one sprint well maybe that's when you need to move on to a more um hybrid model of like some things are going to be more waterfall and some other or maybe more agile i am not an agile evangelist as you can see but um uh but yeah i think i think adapting your style to where you're at in the project to the team type of team you have um i feel also if you have a really big team it's very valuable to have

different uh layer of methodology so you give your team leads as well in your indiv or your future team if you have cross-functional teams you give them liberty to organize their work in a way that you can follow as a producer but also in a way that works for them on a day-to-day basis right um and uh for spring plans specifically that that's a tough one because i think again earlier you can have really short sprint and it works but uh once you get in production uh it gets more difficult to have work sometimes that is uh that you can break in small chunks uh and then jira becomes a mess if your work can't fit into uh into like the exact length of your sprint uh we're still exploring with that at the moment actually on our project so we've just moved to four-week sprints to see if that would work a little bit

better give a bit more time to the team because we're uh entering a phase where we're building more the systems and the big structure and really focusing on stuff that we can't rush so yeah but we need to adapt i think that's uh i think that's something maybe it's just on the management side but i think it's something we need to do more and to ask yourself why do we work this way especially um for example if you're trying to follow agile and give development often you're gonna have a lot of pain points with you trying to follow it to the letter and well maybe it's not because how we make games doesn't work but maybe it's just that those points of agile are just don't fit with how your team work and maybe there's another way to to to go about it that's going to be as efficient like you said if the study is funded there's no difference

but yeah chris what do you think um it's it's a really interesting topic there's a lot of books written on on the subject and uh you know it's funny because the the origin of the agile methodology and it was named such because what was it came from an engineering group and what they said was well we want to learn how to move fast and change and so the whole structure originally had been built to say start with a certain set of practices and then every week or every iteration you should be evolving the process but as soon as something gets a name then it has a strict set of rules and then people have this rigidity and i think that's a very natural um so i definitely subscribe to the lowercase agile idea that you have which is really what no noemy said which is you need a process that works for your team so if people are doing things that feel painful to them that feel like they're wasting their time in meetings that they feel like they cannot plan or execute in a way that feels good then then that's broken um so you know um

i think that the goal is at the end you want to ship a product and if it's a service based game you want to ship a product you know once every couple weeks or once every month and so help i guess it kind of goes back to some of the technical practices but really getting the team practice that we need to be able to build and run our game and test it and play it and and that testing component really matters so i i guess i don't find myself very prescriptive on you have to use this formal practice and frankly i think any decent prac you know any decent process that's right consistent um will be fine as long as you're paying attention to the pain points for the team um and you know one of the things i always try to measure is does the team feel like our meeting time is a waste i mean every engineer i think or artist probably like wants to be at their desk working um but i will say that when my teams have been doing the best it felt like oh we're getting in a room to like plan together and everyone was like involved and then most of the time the you know the when it's going badly is people are on their phone they're not paying attention

so i do think you want to tailor your process to the team and if not you should change it john so yeah i think uh one thing important for the process is the uh expectation management um so well why i mentioned that is like there's a few good examples with myself uh what i have experienced when i was uh running my own startup out act um initially it was our investor wanted to heavily involved in the game development so basically we ended up having like a two-week review so we had to tailor our entire process to be a two-week agile process that we are keeping making demos for for our shareholder so uh that's quite disruptive and after a sinkback you know i'm doing post-molten i think because we used to blend the the other

side we thought that they kind of want to macro management that they want macro management and it's it's always like in my opinion when we were discussing about that was like one side issue but uh later on when i review this entire thing it's two sides it's not just one side the thing is like the the the expectation was not met um it's not properly managed so the thing is like you need to set a very good expectation uh out the environment because it's different right it's different cases because it could be a external partner it could be a publisher it could be an investor it could be or just your team different team they all have always have expectations and different people have the expectation first of the thing most important you need to align that expectation put everyone on the same page and this is in my opinion the most important thing

about all the process you just like make sure like people knows what others are doing and what your your team goes and this is what i in my opinions like very important thing for for for scrum or for any agile process right like they are doing those scrum meetings they want to set a goal or something like that and at the end of the scrum meeting and they review that then they break that into small chunks so it's much easier for you to set goals because if the goal is too far away people just fall apart so so this is important if you can properly manage the expectation of all shareholders and it could be the manager it could be other employees it could be your peers um it could be senior executives and then your entire process will be much much easier and this is the biggest lesson i've learned from all my previous experience

and i i've been always trying to tell our teams like uh don't be hesitate to communicate what you are trying to do with all the uh the your your like uh stakeholders and uh because sometimes people try to present the best part or the prepared part to them so particularly when they are making a demo they they try to wait until the last minute i call that like artist effect they always they don't want to show a half-baked uh work to any other people they wanted to finish that before showing but that's actually going to be a big issue in actually in real life environment because if people don't see something they're going to have all kind of [Music] kind of guesses or wrong impressions they they're trying to think are you you didn't work well or you you

are trying to uh uh you kind of was driving away from our original goal things like that so you keep communicating all these things to your shareholders to your team so everyone on the same page now any process could be much easier so so yeah in my opinion i don't worry about what kind of process you use you just pick the one that you you you feel uh will work best for your team and and and and stick with the uh expectation management that and that's it yeah the way that i kind of think about this um and just kind of taking a step back with with respect to whether it's agile or waterfall no no me you mentioned that you're you know to your point you know it kind of depends on your your team in situational context but i kind of view it from the perspective of actually complexity domain right so like if you know your target if you know what you're going to be developing and you you don't expect a lot of iteration then

then for in my opinion the the obvious solution would be a waterfall methodology because you you know your target you just need to execute against the target and then i would say for for agile and to the point that you guys that you're making shong in terms of like expectations and when it comes to like uh you know i mean you mentioned like a four week sprint i i think the the length of the sprint two weeks three weeks four weeks has to do with to the point you're making shang about well who the stakeholders are and what is a reviewable piece of code or you know what in agile what's called pspi potentially shippable product increment so what are you able to show in terms of a product increment and so in games i've typically seen something like more often two weeks or three weeks in the web world it's usually one week because you're able to quickly show for a website or whatever you know you can you can see things in a one week increment versus you know in games you generally need two weeks three weeks

if you're in some live ops teams or four weeks for example because that's how much time it takes to have something that is reviewable to stakeholders and things of that nature but yeah that's kind of how i think about it and then in terms of like maybe just kind of going at the weeds a little bit more do you guys have any specific thoughts when it comes to for example uh the structure of the sprint like having like that initial planning meeting how do you guys come up with the tasks who does the task breakdowns in terms of costing stuff and this is something that's usually religious you know is it is it based on t-shirt sizes complexity time and you know i've gone through all that stuff but i don't know if any of you guys have any specific thoughts on the the sprint structure itself anyone want to jump in on that all right one thing i can mention is uh you you keep seeing uh particularly

about engineer uh delaying and it's it's kind of a norm in game development and it people would think that all right after several iterations you know the estimation you better estimate that right you learn the lessons you know you need to spend and you keep seeing delay and people keep adjusting their estimation and you keep delaying that so and people working very diligent and people like work day and night and they still delay so uh the the reason we keep discussing about that why like there are such kind of delaying it's not about like people not learning and people are learning and they try to like give certain kind of uh offers to any of the estimations and they still get that is i think this one thing is like the biggest thing is like when you still going back to that expectation management is like when you set expectation

and you you try to do that uh like within your control so how should i describe that it's like um say if there is a task assigned to engineer and that engineer uh would say that okay i can do that in two days that might that's my expect that's my uh estimation but let me give it another day as a buffer because i know there can be a lot of iterations things like that so the engineer gave that three days to the pm or producer and it's recorded in in jira for example and start working on that then the engineers think okay i have three days to work on this thing so the engineers basically working on this task with three days in mind not today in mind that's the biggest problem like basically they think they can

they have control over three days so they basically just scale this entire thing for example if it's a two-day process if it's if it's a two-day task then something you probably will skip you probably just like get a shortcut to certain thing but you think with that three day in your mind you think oh i'm gonna do this in a perfect way i'm gonna restructure this code i'm gonna get there and you end up spending five days on that so that's always the big issue i keep talking to our engineers like when you when you estimate that for two days you work on two days you leave that one day buffer just as a buffer you still think you should finish that in two days not three days so so that's actually one big lesson i've learned and i try to communicate to the teams but unfortunately it's not always working and i i want to keep mentioning that i think that is that

going to be uh always a challenge to uh to to to overcome yeah yeah yeah i'll just um i think that that effect is probably universal i think everyone's experienced it there's a couple really good books on the topic and one is by a guy named donald reinertsen about product development flow and he basically studies like the toyota production system and that kind of kanban ecosystem but he really does a great job of saying hey this model is applicable to product development however all the tuning parameters all the variables are actually like almost the opposite in manufacturing you're trying to literally produce an identical piece with the least amount of change the the most reliability in product development you know software is you know you write a piece of code and then you can run it on a million computers so every feature is by nature unique and new and interesting that's why we're adding features to our games to our products and so what he describes is that inherently iteration and experimentation has to be part of that

journey because you're trying to find something new you're not trying to replicate the feature you wrote last week because that one's already written and so what he tries to describe is that when you think in terms of estimation it is not about a commitment to i will get my work done on exactly this scale in exactly two days it's just too hard to predict the future and nobody is very good at it so he basically tries to describe like an estimate should be treated as a like a bell curve and it's like the middle point so if you say two days you should think of that in terms of well sometimes you actually are pleasantly surprised usually it doesn't happen but sometimes you get your work done in half a day more often you go long but importantly if you treat it as not a commitment of i will get this done in two days and instead it's a bell curve or it's a distribution of of median expected completion time that makes it safe for people to give honest estimates and it makes it so that they don't feel that they have to pad the estimates because i think that the effect uh shang is describing is very common for people to want to add extra buffer time and then they expand the ambition of

their work and then of course they run over so i think that that's a key insight into estimation that if you can make people feel safe that it's not a commitment but it's an expected average completion time then it makes and if you plan around that you have to build buffers into your schedule not into your estimates because as soon as you start doing that you stack a buffer into the estimate buffer into the schedule and now everything this is how things go you know two times over four times over right and i think it's a good point that you know software tasks are inherently you know it's not a perfect science so estimation is always a little bit off and so to that point like how do you guys recommend to do estimations and like chris you're saying it should just be an in potential expectation but not a commitment uh noemi in the console world i kind of feel like oftentimes it's it's considered a commitment

if if i'm if i'm correct and so what do you guys recommend in terms of like if you have a whoever's running the scrum if people are running behind what should the reaction be or what what should the attitude be is like oh we're slipping again ah oh well let's let's do better next time guys or should it be like guys like like maybe you could speak to what the reaction of a you know producer or a product manager should be with respect to estimations and maybe what your experience has been you know i mean um yeah sure sorry i allergies uh yeah i think i think by having like two level of uh of planning you also help with the slippage for example um it was the same when i was in tripoli and it's the same on mobile i plan a milestone in three months and then in that milestone we're gonna plan

all the sprints um but when i plan the milestone i do work with estimates which i add the buffer in time because also right if someone tells you it's gonna take me three days it's still gonna take five days because they're not going to be able to work on it full time for three days usually like especially us we're a small team so people work on many things at the same time and we're live so they can be called on doing life support as well which usually takes precedence over what we're working on um so when we do planned a milestone we planned uh we plant buffer for live but we also plan some work that can slip because it's not mandatory so i think that that has worked really well so far for us because it means that people can work on other stuff that like no one is gonna no one is gonna come in and uh like

and be disappointed if it doesn't ship because it was like a stretch goal or an initiative that we wanted to add like something that's really nice and we do want to have it but it's less time sensitive um so that's been working pretty well and then into this print we try to do the same thing as well so uh people are uh because we booked uh life support in every sprint as well that can also act as a buffer if we don't need it so that like ultimately it it it balance itself out pretty well over the long run so of course we don't want to slip in sprint but ultimately if we slip and sprint but the milestone is fine we're still in a good position right like in the grand scheme of thing we're still gonna arrive at our gate or at our next like team goal that we want to hit

um all right guys uh any other thoughts on the estimation aspect and do you guys also have any specific thoughts on like you know uh time versus complexity versus whatever i would say the the formal models that i've looked at would suggest that every estimate by itself is uh wrong um and if you have a consistent uh measure a a consistent form of estimation it becomes predictive in the long term which means that in aggregate they'll actually become reliable even though literally every single individual one is wrong and what's really valuable is to keep a consistent uh form of uh of estimation so to to shang's point if if the team keeps adjusting how they're estimating every time every cycle to try to account for it then the model will never kind of like

stabilize um so okay well you know what i think um we're kind of running out of time here i actually have a bunch of additional things i wanted to talk about uh you know this this kind of topic is very interesting to me but in the interest of time maybe we could just wrap it up here and maybe as a final message do you guys have any message for our audience maybe starting with you know emmy uh sure well um we're hiring for sure so if we have a lot of position open if people are interested they can look at square enix montreal and uh yeah for me like to wrap this up is really like to keep your production strategies uh adaptable and flexible okay thank you chris um yeah no um i would just say that uh you know this fascinating topic that uh i had fun talking about and um definitely encourage folks to um yeah dig into literature around the subject and uh at pragma where um

our team is growing and we're focused on building you know back ends to power online games so i'd love to chat with folks all right sean yeah i think ultimately making game is a fun thing to do and it's quite different from other stuff so so what i would recommend is always follow your heart create a fun environment uh work with others uh laughing holiday and uh and that that's actually a pretty a positive attitude and they'll actually help you to improve your efficiency and uh that's very important in my opinion more importantly in that process in my opinion maintaining maintain a very healthy environment okay got it all right well thank you very much for your time everybody and i think that's it thanks everybody bye thank you thank you bye bye

[Music] hey [Music] you