Making Release Day Just Another Monday

[Music] did the other thing is on the other side of that wall is my lead designer running a talk right now at the same time as me I think that's why he has everybody shouting so can you all do the same for me and please shout and we'll get this started fantastic thank you very much it's always great to start something with people cheering for us rather than coming at the end so I'm Connor I'm the senior producer on runescape and I have had the benefit of while doing all of the other things that you do in production also spending three years focusing on a single products pipeline and moving it from things that may have caused issues on to what we have today which allows us to every single Monday at about 11 o'clock put out a new piece of content and we used to have a problem where we would slip from Monday to choose their Wednesday Thursday and you start encroaching on future releases as well so what we're going to go through

is how our rapid release cycle has allowed us to improve that over time so what is runescape quick question I was like starting with this one who's heard of runescape hey who's played runescape whose still plays runescape we still have people so that'll do it was the first free-to-play MMO it's been alive for 15 years continuously and over those 15 years we've continued to release content week in and week out right now we have two live versions we have the main game which is the one who's been running continuously and a number of years ago we forked the game from a 2007 build and that has had its own progression and now has 300,000 members we we tend to have 500,000 people every single day and we have over 2 million people every month so it's a big game and there's lots of people there we've a player with an insatiable growing player base we're looking for more and more content and as

a game we currently hold the world record for the most prolific updated game and those that updates aren't just little bug fixes or slight changes they are big meaningful pieces of content so we do quests which are two to four hours of rather than fetch quests you're talking about custom narrative bespoke encounters new environments the event systems are small time limited pieces of content that players have to jump in and get involved with with brand new skills being added into the game rarely but we had one at the start to the year and then we're always releasing new bosses new monsters new pieces of content that way so the important thing is that every week you come back to runescape you see something new and substantial to get involved with and that means that we can hold our players into the game and this is something that I'm hoping that by the time you leave here you'll understand what you can do to match that for yourselves and I see this is a bit of a challenge a release challenge and there's three components to it the first part of your updates what is an update what does it mean to you we have we have

large updates that come out each week but for yourselves it's about finding that balance between as a development team what can you do and as a player base what you actually want it can be a difficult negotiation to have and we'll go through that in a moment and then you have to consider your pace and the cadence that you're going through how do you release how often and how do you make that as simple and as smooth as possible as we update weekly that means that we can refine and adapt it very very quickly if you're looking at more of a monthly or annual update then it's a bigger day and you've less chances to make those tweaks and improvements so hopefully you learn from us going crazy and you can take them into the bigger update that you're trying to do and then once you've gone live all that content that you've just put out you need to make sure that you can maintain it because no one will ever give you any credit for fixing bugs it's just kind of what you're supposed to do but it still takes time you still have to give your thought and your attention to it and so how do you keep making sure that while you're maintaining your game and your service you're also releasing new and big updates the whole players back into

the game so let's start updates and like I was saying it's a negotiation you need to listen to your players with runescape we look at our social channels we have that conversation Facebook Twitter Instagram and we listen to what they're saying and we try and take that on board we also like speak to them on forums and then in-game conversations you'll have a certain type of player will engage with you and so should you have a different type of player who'll only speak to you and you're in citing your game and in-game people are really happy when you have a dev just kind of turn up and have a chat I mean it's a nice thing to do but we also have to look at the qualitative or the quantitative types of feedback so you're looking at analytics data what our players actually playing they may say they like one thing but they're actually playing something completely different make sure you're getting that kind of balanced up and you understand what does an update need to be to really get good feedback from your player base then the conversation moves to your team it doesn't matter what you promise to your players if your team can't actually deliver it and the people who best

understand what a team can deliver is your own team we like to game for being comfortably stretched so we feel like we're kind of pushing ourselves every time but we know we can do it and every time we do it we push ourselves a little bit further to see how much bigger we can go we it comes down to us understanding what kind of rate can we deliver at what's our velocity and getting through content and then with all of that we let the team actually decide the releases we speak to our players and get a pool of ideas that they're really looking forward to but we let the devs decide what that would actually be as a smaller team rather than top-down decisions of what the design would be we let the devs power that and feed it back up and so the first thing that we did when we were having problems with trying to get our releases out on time as we thought what we need is we need some strict checks and process so we'll put it in controlling the steps that people are going through we'll put in our sign offs we'll make sure that everybody's on board we put our detail upfront we make

sure everyone has agreed as we step through it and you know it it stops waste right you don't have to build something that you won't be building or using later on and we detail our face are all of those ideas well what actually happened is we kept slipping anyway and every time we do a release we look at it and some are asked for more process and more checks to be put in place and that didn't did not work for us so I have a couple of samples of responses that we got when I asked the teams what do you think about how we work and you see it was sort of reasonable if it's practical we're always scrambling to get things out everything is merging into a single blob stakeholders and it's just the same things over and over again and we got this back from the teams so with this in mind we decided what we need to do is really look at what we've done with our teams how we used to be structured is everybody would be in there each individual disciplines so you'd look at artists be a concept modeling animation any artists we're in one group we'd have

our code design and everything another group and then you pass it on to a QA a very standard waterfall process where you step from one to another to another and it's those pass throughs that were actually causing us a lot of the problems and heartache that we were having trying to get content out the door you have problems of it getting delayed feeling like they didn't understand what they were being asked for the handover didn't exactly cover the e/m the requirements at the start so we decided to split it up and we made six autonomous teams these teams are like small little development studios and we took each of the devs and we moved them into those teams and you'll see you have a mix of every discipline those teams have a unique identity each one of them does things in its own way and they talk about things in their own way and they're like six completely different groups of people and they bring their own flavor to the game we also expose that outside of this the studio to the player base and they understand which team is responsible for which update how they'll expect stuff to come in they tend towards specialties but we didn't design the teams to be specialists just kind of happened it

goes at their own way of doing things what we did then is we created to release trains and we assign the teams to it so the content updates are those big meaty large updates those bosses the quests any of those raids style content and then we have supporting updates which are a little bit smaller smaller events putting things into the store all those kinds of stuff and what we do is we plan in chunks of a month rather than trying to think of 52 extra things to do throughout the year we're looking at 12 big blocks of content that will be made up of a number of updates and each of those teams is assigned to one of those months and they work together so we have a support team and we have a main content team looking at a month as a chunk of content and understanding how they can work together to get the best flow through the month and handing off to the next team that will be in the following month also means that each of your teams doesn't need think about the whole year you can split it out and you can start understanding I can deliver this month I don't need to think again about a release day for a

couple of months I have to think about the release day every week but that's not their problem that's my problem and then we look at the types of content the teams are doing and we can put one of those other teams and they'll have a larger step between their months for a Content team or is the support team because they're doing smaller updates their cadence can change they can only they leapfrog each month and it means that we get a better understanding of that velocity like I was saying you understand your team you understand your velocity and you build your cadence based on what they're able to do and you let them feedback to what they're able to do but of course splitting up into those autonomous unique teams is a new form of challenge for communication before we had to make sure that communication went through the disciplines and now we have to make sure that one team doesn't develop the game in one direction another one develops in the completely different direction so we put other structures in place and the first thing is disciplines so we brought those guys back but it's more of a temporary group on outside of the development teams themselves and this is

how do we work make sure that even though you have a reduced number of each of the disciplines working very closely together you ensure a community of practice across everybody through each of those teams and this is done through show and tells just demonstrations of how we work and you have the same thing for our coders make sure QA and so those guys still feel like they have a bond and they're still part of a larger team even though they're in the smaller teams to deliver the actual content the next thing we had to do is what we're making and make sure that that also works so we started to see kind of a divergence of you know is the lore going to go in the right direction and we created councils and these are groups of interested individuals across each of the teams and they're interested in various parts of the game like combat like balance like the lore I wanna make sure that we can cause a coherence through all of those teams so with all of these things in place we now have our teams who understand what they're building they're

making sure that everybody's moving the right direction even though you're a part of you're one of six autonomous groups who are really in content so we've got it all tight we're building our content we understand our velocity now we need to make sure that actually when we're putting it out the door doesn't fall apart and that's the second part of the challenge and the release and to be honest probably the media's part of this but I prefer the first section so I'll go through it anyway when you're releasing we have actually a seventh team we have the release team who are slightly different to everybody else and it's made up of our build manager our lead tech a group of dedicated testers who are specialists in merge conflicts which I'm sure everybody has had to deal with time in and time out they know what they don't need to worry about as much of the content they need to worry about how it comes together and the impact the issues that you will receive when it goes outside the door we also bring in our community management team at this session so they understand the content that we're releasing and are embedded within the development team so they know what's going to be happening in the

future and then we also have marketing in this group as well so the marketers are are understanding the content they know what's coming out they can work with a release team to make sure that the right things are in the right place and it's this kind of split of different disciplines once again and not the normal disciplines we had for development but the people who will be most impacted by what goes outside the door of the studio it's become incredibly effective for us in this way and with this team in place we had to build a bit of a process around it the other teams the six the six content and support teams they work pretty much in the way that they like we try and step back I don't wanna put process in the way of them getting good content out the door and because they're a small enough group you don't have to have a heavy hand in what they're doing it's more of a guiding touch but what we did find is there is a place that you do require process and sometimes it needs to be fairly strict and that came in the release and this is why we were able to do things on a Monday so if you consider us trying to put something out the door we've got nice three weeks here and if

you want to release something on Monday on week one I want to come in our week three I want to come in Monday morning and know that that's going to go out the door at eleven o'clock and I can start thinking about the thing that'll come out in the next couple of weeks as well what actually happens is the content team will finish their work and this doesn't mean finished everything because there's somebody still dabbing on they'll be furious it's more that team has come together and they agree that by the Thursday they can sign it off the sort support team will be coming along at the same time as well so together they feel that in isolation what they have built along with everything that has been released previously is ready for players to consume it may not have everything else is going to go out on week three but on its own they're happy with it what happens on the Friday is we kind of smash it all together and hope that we can make a release that'll go out the door and everybody will enjoy and that's where we lead with the lead tech and the build manager that's their key place where they come into play they know the game they know the structures and we just have to deal with merging

lots and lots of things together and what happens for the week - who's he we tested we verified the merge didn't cause any problems we target what key areas of the game would have been touched by the content that's going into that update on that Friday I need to have a sign-off from my team because if they don't sign it off in that Friday I'm not ready to go live it's very rare that that happens in fact I've only had one slight suggestion that we may delay something in the past 12 months because we have to make sure that the previous week on that Thursday that those content teams and support teams are happy for that to go out the door we should only be dealing with March conflicts may be very strange for us to be dealing with any other types of issues and the main reason that exists is because on that we - we also released on the Monday and the test team we're doing something on the week beforehand and then on the following week they're also testing something else so it all gets a bit convoluted so something falls out of place we have a lot of things to start juggling and fix it's important for us

to be aware of issues long before it gets into this process this needs to run smooth alright we got it out the door probably the media's part but shortest as well so it's quite nice we've released something we have a game runescape is a game as a service so it makes sure that there's always something for people to come back to but we need to make sure that we're dealing with any of the things that come up we don't want to die a death of a thousand cuts from all the little things that get released each update I'd love to say that we're great and every update we've ever released had no bugs doesn't exist nobody's ever done them what we found is that process at the end which I said is incredibly important and has allowed us to move from slipping to Tuesdays Wednesdays Thursdays and lock it into place and know that by the time I have my lunch on Monday I know the game updates gone out and I'm looking at what the players think we put that bureaucracy in place but it reduces agility imagine if I had to wait three weeks to deal with a bug issue on Monday there's also been two other updates since that originally

happened and if it's a major issue I'm going to just lose players people are going to disengage we need keep that quality going we also need to demonstrate that ability to react it builds a trend build and fosters trust in our player base but continuously listening to what they're saying and adapting to it quickly part of that is allowing us to do quick hot fixes so there is zero downtime for the player if we're doing slight tweaks it's very occasional that something would be a major issue that we actually bring the game down but the other thing that we have done is built that last team so you remember on the six slides one of those teams didn't move anywhere and that's our ninja team they have one goal and is to make everything run smoothly they're focusing on what are the things coming from the players what are the issues what are the slight bugs what's the thing that we released ten years ago that's still kind of niggling and it's now not working as well with a piece of content we just released and they have no restrictions they have no need to go through the same process they work directly with the release team well I said directly closer than all the other teams they're getting more fixes

in lining them up and getting ready to go out the door and it's because we have this team we've actually improved the health of the game overall in the past three years that's all they've been doing is just taking out those small little rough edges and making the game a better experience for everybody we've seen the players the players loved them they made that beautiful picture on the end that might lead designer told me not to put in for I put it in anyway but they they love them they are always talking to them if you look at the runescape reddit you'll find that there's a tag saying ninja fix-it they've just adopted this as something that they found incredibly useful but it also means that internally whenever the Ninja team go and get involved with a piece of content they're fixing in the backend I'm sure anyone who's had to deal with a game for an extended period of time knows lovely story of legacy code these guys are cleaning that up it might be a small fix for the player but the next time I'm trying to do an update in a particular area of the game if the ninja team have been there they've made it better and so the dev team are getting faster over

time there's some bits that we've had is they've released named 26 updates because they're so close to the game and they're so close to the players even a small little tweak from these guys is enough that gets an invested player bought back into the game and we can call it out as a named update they've delivered seven hundred and sixty little tweaks and fixes throughout the year and they've just made everything better if we do something with a ninja update we get an immediate positive return from the player base the sentiment on any update that has a ninja update is always better than if it didn't have a ninja update and this is something that we have just doubled down and we're increasing the amount of effort we put into the ninja team while also trying to make sure that all the other updates continue to go out which is a fun challenge but a different talk and with this in place and with these these teams doing that kind of work what it's meant is that we're looking back in on ourselves and so we have the game being maintained and we're looking at everything that we're doing but that doesn't mean that it stops and we're

always looking to continuously improve and we have three parts that we always do we're incredibly transparent with how we work before we were very enclosed very protective of what we've done now we're open and honest about everything that is a problem what we're developing or how we're developing we talk to everybody and get everybody involved in communicate are in conversations and when we have that shown and made clear for everybody we're inspecting what people are actually saying so we understand what their problems are and then we adapt and make it different we've gone from a place where we had process and our answer to problems was to put more bureaucracy in place and put more checks and more balances and we've stepped away from that I mean let the team lift up what they need to get fixed and we fix it and we focus on what goes out the door not what we're doing inside in the studio so some things on what we've done in the past year or so we had a hundred and twelve named updates going to the game most weeks don't only have one update there'll be a couple of things because we have a large player

base and they all have different wants and needs that was across 48 weeks because we take Christmas off and a couple of other ones across the year but actually of all of those weeks in terms of inside the studio none of them had any they are any week without a change and they're not all big changes I mean there are some things where we bring in hardening play throughs towards the end of the project just to make sure that maybe our backlog doesn't mean anything because the game we've just built we need to make sure that we make that better rather than just doing the thing we wrote a week ago are getting better involvement from our concept artists earlier in the game earlier in the development but there's also things like somebody didn't like the headings that we used on retrospective documents so we changed those and there was one they were we've decided we have weird which fruit Wednesdays because it was getting a bit boring in the middle of the week but all those things make the lives of our developers and our teams a little bit better and when you see that things are getting better over time and continuously changing and making what they're doing easier for them we get

better sentiment internally and that's reflected in our player base and they enjoy seeing what we're doing so what we're going to do next what we're trying to doing it's a little bit scary we're trying to get the team's more autonomous we're trying to give them more ownership let them pitch the release schedule let them pitch what they're trying to do our devs are incredibly close to our player base every single one of them is on Twitter constantly but they're listening to what the players are saying on what they're looking for and I can't tell them what they should do when they know it better than I do we have our design and our top-level group making sure that we're going in the right direction but the actual specifics can come from our team and they'll probably do a better job than I will anyway we're also trying to become more autonomous not autonomous more automation we have that process at the end I do have the specialist testers that are dealing with merge conflicts but trying to pick up an entire code base and move to more test-driven development and unit testing a more continuous integration is difficult but it's something that we want to do and

it's something that we'll do with our future games so that we know that we are testing it thoroughly and we can push out faster and earlier without having to lock in that three week process we also want to try and move to a place where we are continuously deploying I'd be much happier on my release day if I'd actually released it a month ago and flicked a switch for the players because I know it's been running we've done it recently we released a telemetry package into runescape the telemetry package itself and all the backend infrastructure required to run it had actually been running for about four or five months previously and we were just gathering data to make sure it worked I meant that our release day was this button switch we turned it on but actually we turned off the thing that said don't show this to anybody and suddenly everybody had access it was the smoothest release day I've ever had but we had to put things in place that would hide it from the players while we were testing it and that's difficult built from the ground up would be a little bit easier a lot of what we're trying to do is retro actively fit it to a very old game and so I hope you enjoyed this there's some takeaways I encourage you

to mix your disciplines every time I speak to people and they have problems of communication it's generally because they've siloed the discipline teams holding the cue way off to the end handing things from an artist to a developer and back again mixing them up means that you're getting better integration between those teams and they have different ideas that help each other's work consider leapfrogging that was the biggest change that we've done recently and has had a good impact teams aren't as stressed they get pretty stressed one month when they're just about to release something massive they've been working on and they're a little bit afraid of how the players react blast stress is based on them hoping that the game is good enough rather than worrying if it's gonna fall over when it goes live so it means that they are able to focus on something big and don't have to have everything else there we still have our group at the top we're making sure their work going in the right direction consider why you're doing things in process sometimes it's really useful sometimes it's essential but isn't always make sure you're doing it for the right reasons and then maintenance can be a massive distraction

I'm sure everybody here who's released a game or released any sort of service has seen the impact that you've had on your development ability when you have to deal with all the problems that you're seeing as it goes live it's going to happen you're going to have to deal with it make sure that you've put it into place and you understand how you're going to deal with that maintenance before you go so thank you very much I hope you enjoyed the talk [Applause] I tried to go fast enough we got 5 minutes if anybody would like to ask any questions so I imagine since it's a continuous development cycle you have you know schedule far enough ahead did you have to pitch out your entire schedule and just start over when you switch to this no but because of this structure that we had in place we were able to transition nicely we knew what we were building and we knew the day that we'd have to move so what happened is we just move one of the teams into it I would recommend that if you ever do something like that don't do it over Christmas like we did because the first team that was releasing on January was

crying through the Christmas holidays don't do that that's a terrible idea hi how do you prevent the dev teams from handing off tech debt to ninja and then who deals with live issues and then secondarily do you have any issues with loke with your with your tight schedule sorry what was the localization so to answer the first question ninja will only ever jump on a project after it's a number of months old they won't fix things immediately we make sure the detective our detective and our bill manager is there merging that's their kind of place to really put the final check to make sure if things okay there's also code reviews and everything during the process as you'd expect in terms of localization we have internal localization team which is really handy Jagex has been built up from the ground to do everything internally so handing out and having to come back it is definitely a pain point for us though we have to make sure that we are getting all of our content locked down early enough and with all of those things coming into the localization team at the

same time it's very difficult but our guys are great they manage to keep up on top of it and we're trying to make their lives easier right now it's actually on my list of things that I need to change thank you I'm gonna keep swapping it okay hi um so I was wondering how many producers does it take to manage all these teams or are you the only producer or how does it work one yeah there's only one person in the runescape team who's the tight to producer and ask me what we have is a product owner so if you're familiar with scrum and the concept of the product owner and scrum master we tweaked it a little bit each of the teams has its own product owner but those teams are also are those product owners were also developers it's kind of stretching the rules but worked for us and we've kept it that way each of them has two jobs but they also have support for myself I do have two project managers who tend to do things that go outside of the development team but I don't have an internally focused producer on the making okay thanks no problem thanks for a great talk what happens if you miss a Monday like I don't receive a sign-off on Friday you

postpone one day or one week so trying to cover that it's a we wouldn't it depends on the issue we have occasionally done a Tuesday it hasn't been nice because also slipping to the Tuesdays reducing your test time on the following week so maybe an option as long as the following week is in not a massive update there's something a little bit more routine we everything we do we try and employed that we make sure that the content coming into the team is a release team it's actually ready at that point and if it isn't going to be ready at that point I know weeks or months in advance and at that point we are adjusting the entire release schedule not just one or two weeks so it's the concept of making sure that you're looking holistically and everything you're changing not just that week and then the following in the following week it's a bit of a juggling act but it's do I have a quick second question if your ninja team is a release an update that's not critical they follow their release schedule the release team schedule yeah if they're doing something that is a significant term named update they'll always do a they will work with the release team to

make sure it goes through the regular process it tends to be fixes or tweaks or changes where they just kind of sideline it but to be honest we kind of do it on a case by case basis one of the things we have kind of leaned on a bit occasionally is we kind of fudged doing it on one day is we had an end we had a ninja update ready to go and found a very last-minute bug I came in right in the middle of the release team's test week we ended up having to slide in a ninja update at that time we saved it and was a bit hairy is but it is very very rare that happens it's nice having a backlog ready to go thanks thank you I have so many questions is a fascinating talk first question how big is your team how many developers I believe we're about 80 poly including into the dev code so how do you deal with conflict it's when Pete you know people disagree on process or you know what goes into a release I rely on myself and I'm the bad

guy there are cases where I have to say we're doing this and not that we tend to have those conversations fairly far in advance there are definitely pieces of conflict but quite often it comes down to myself and lead designer we'll have a chat and try and come to a good conclusion so far we have the trust over I say so far we have the trust of our dev team and they believed they were doing the right thing so when we do make those choices they've seen the benefit that it's had in the past and they trust us if we lose that trust I don't think I can keep doing that but that's a bigger problem and lastly real quick so you said the ninja team jumps on you know months after a release goes out so is the dev team that put changes into a release are they responsible for monitoring those changes live and responding to them for further absolutely if you consider the release team themselves and the testers there they actually get bolstered by whatever team is doing the release in that week because that team doesn't really have anything else to do they need to focus on what they're putting out the door so they become just kind of

a bit of an aide and that also kind of helps with a conflict in that teams tend to share amongst each other when they're in trouble we frequently like each of the POS and myself will sit down either weekly or monthly depending on how much stuff is going on and just chat about what's going on and make sure that we have a good relationship between all our teams rather than a competitive one so everybody's kind of helping everybody thank you thanks my question is kind of related so if you had a bug that went out you know through testing got out to a release how fix would be how fast would be your hotfix time would it be you would inject into the next week's release so it'll be two three weeks out you know if something goes wrong our fastest hotfix has been ten minutes we will fix anything there's a major issue within one to two days holding my hands up we released the we released the skill and there was a significant issue in it and now skill is a major piece of content trying to disaster skill is a major piece of contents thousands of hours of gameplay level design are the

balance and progression curve was something that had an issue we went from a point of trying to deal with the progress an issue to having a fix to getting it like I have a built tested and live in under three days so as a team when we have to mobilize we work fast and we've built our deployment systems in a way that we can push something out quickly so that's the big way that we try to do this and it's that release team that does that fix and pushes absolutely well the release team will test it but the development team will be the people who make sure that it's working it's it's their responsibility well mark - oh man I'm sorry yeah yeah parts of giving them ownership means that they also have the response great power comes great responsibility right so we lean on them real quick in in one slide you showed the distinction between content and support teams and had on the Friday two weeks before the the Monday released you had a merge of their two builds and I was just wondering what you were merging at that point what separate paths were being worked on since the merge was took

an entire day so to give you a structure or an idea of the structure of our code base we have a main line and each of the teams will have a branch for the release and we're pulling those branches back together we actually go round main right if the this gets a bit technical boy we build a release so we build early cycle by taking to child branches dodging main putting them together and then pulling that back down into main because trying to merge the main and then build out will actually cause us more of a delay it's something that we've come up it's one of those many many changes that we've done yeah so yeah those two are coming together into their own release build and each of them will have the previous release release so when that word that build is made on the Friday the following Monday or the other development teams update all of their streams to make sure it matches that it's a bit of chaos but we've gotten it working that's right thank you legs okay great talk I was wondering your ninja team if you have any problem with motivation since they are they are like non as creative as

other teams and if you allow people to change teams like if they're in the engine team and they're just fixing bugs that they want to do some release through some content we let people change teams quite frequently I say frequently we let them choose to change teams there occasionally times when we've had to move to different groups so the first question was moderation is know if they're always motivated to to fix bugs fix issues you have any like motivation about creativity from your ninja team they work very closely with the community team and themselves watching exactly what's happening with the players also we have a lot of analytics and stats coming back so we're watching that too and we let them kind of choose what they're building from that okay and that motivates them yeah okay Oh motivate sorry yeah definitely if you see a release come out and you have about 15 20 or even 15,000 people screaming saying that is the best Forex ever they have a goal where on a major release they want to make sure that at least three posts on reddit are saying this is the best ninja fix ever and it

jumped over the release content it works oh thanks I'm going to have to wrap up I think I'll do the two we can so if anybody else has any questions I'll be over in the wrap-up room and we can we can go there okay do you have any analysis working in each team or we have a analysts working for all teams is it analysts yeah yes so we have a game analytics team they're our shared resource rather than trying to employ six analysts also they have a more holistic view in that case okay all right thanks last question so you have multiple teams of different disciplines how is our key organized like the your official boss technically all line management follows the discipline lines there is no line management within the development groups so it is it's a it looks like a traditional line management hierarchy okay okay all right thank you very much everybody if you have any questions i've been rapidly thank you

[Applause] you