Panel: What are the Best Ways to use Tools to Bring Data into Design? PG Connects Digital #2

we're here for the talks and I think it's about time to then head into our next talk which is a panel a panel about what are the best ways to use tools to bring data into design this panel will once again be moderated by Oscar Clark will be at its to this room in a moment there it is Oscar has no less than four panelists on this one so that'll be a come on board straight away yeah so it's kind of funny actually my thinker so one of the comments made in the previous panel was about we have too much data hey guys

here we we do rely too much on data so I'll hopefully try and answer some of those thoughts in this session too so again thanks for the intro hi everybody so yeah we're going to be talking about you know data and what's the best way to bring data into design and let's start off with getting you all to introduce yourselves so I Jasmine you wanna start just describe who you are what you do and in particular in terms of your interest in design and data can you guys hear me okay yeah perfect so yeah my name is Jasmine I'm a senior UX designer at mainframe ministries we are in the very early stages of creating cloud native massively multiplayer online game and I've worked really many years on mobile live games we're working with data was part of our daily practice I mean yeah hey everyone nice to be here

my name is Mila I work as product manager at fund a factory where mobile games studio based in ojos Denmark and we basically transitioned from doing a ton of work for hire stuff for larger clients like toys and games and kids to worth doing our own as were free to play games it's been a quite a journey that I could do like a separate talk about but but my role as a product manager basically I started as a programmer then transition into a brief stint as an artist we odd then a producer and I'm a product manager and I'm lucky enough to have a team a tiny team but a great team of like data analysts and our own in-house you a department which is fun so yeah programmer off she's never normally synonymous with great stuff but I'm sure that wasn't the case with yourself no no I rely realized the error of my ways and I was definitely better here than doing actual code so this is if better for one a minute oh you a mute - you're on mute sir

check out the yes sorry sorry no now you hear me hi everyone I'm very excited to be here my name is no Chandra I'm founder and CEO of Jeeva company behind they are games maybe you know smishing for and as a founder for me was new challenge to combine creativity and nature together who works together but I think will be very interesting to talk with you guys Oh max awesome pleasure to be in me all of you guys a shame it's no person but you know do all we can here all team so I'm max I'm a level designer of an industry for about just shy of seven years now and I primarily work with with data to help improve my levels as well as the design decisions that go behind them so yeah that's me pleasure again to be here with you all cool great stuff so let's dig into it so I mean why do we

care about data and what did a did ever do for us anyway I mean one of the reasons that I think it's interesting to ask is because I was in a session earlier we were talking about outsourcing and workflow and it just came up and Chris from booms up said yeah we rely too much on data data's not as important now what do you think you know it's data important and why is it important that what's it for Jasmine what Julie would you start with that absolutely so I mean the first thing to note is that you know data and analytics are first and foremost it's a tool it's not the single it's also not the single only tool that we are using and everything that we are using is to help us to make an informed decision right and we also want to learn about that if we make and made a decision that how did it perform in a certain way like did it did it do the things that we wanted it to do so in that that is where you know data comes in as a tool basically to help help us to verify and and and to

measure you know what have we done and did we you know hit certain targets but you know it's not you can't of course rely only on that and there are just certain risks you sometimes need to also take to maybe make something new and to see but again you know to see if it works you do need to preferably use data in your own day-to-day life I would say a meal yeah I agree data is one one tool in the sort of belt of tools we have available to us as developers I think I actually listened to Chris's talk and I think he does have a point that if theta becomes fetishize to the point where it's like the end-all be-all truth of the entire game like then we've lost something along the way something important I think data is something that should trigger interesting discussions between your designers your artists your UX designers and of course in terms of me as product

manager I also need to look at so if we are we looking at data the right way but in order to answer that question data can ask the questions but you need the entire team to bring in their expertise to actually bring out a potential answer to that because it's a complex complex questions that the data often ask us yeah I think that's the key thing there isn't it's like we don't want to fetishize data for data's sake but we do want to know what questions we want to ask and I think understanding you know what we're asking and what we expect from that answer before we check the data is as important as the information itself I suspect so I mean do you agree with that or if you got a different way of looking at it then we have ideas from beginning designers developers artists and we are we are clicking in the data if this ideas are really come to to

point to to end customer to player if they understand right and I think the data data is amazing it is new news cycle behind free-to-play games then we now developing in from from concept prototype to production soft large when are very important to decide okay let's follow up or not and to to globalize and these these are behind behind whole new development cycle of the games which is now used in free-to-play free-to-play model but again it's only two and on the beginning you you need this is creativity and and human brain come with something something new something innovative I actually think it's not just about free-to-play I mean obviously I'm gonna take freedom a fan but actually what I realized is when I first kind of really saw what people could do with data in games was when I was at Sony and I saw

the guys working on Killzone 3 and the work they did with heat maps seeing where people were seeing where shots were taken and where hits were made to see kind of the timing and the movement around the space you couldn't have guessed that you actually had to see what people did and seeing heat maps on on a map I know it doesn't sound like the most complicated sophisticated thing right now but it had a profound impact on the ability to design a better play experience not just one that was about getting more money and I mean so I mean max I mean yeah as a level designer is that something you think is important I mean or are you looking for other qualities when you're looking at data well I mean you all raise like great points and me us you said it very well with like being part of like prototype stages and stuff because when we're saying data this covers like a broad spectrum of what type of data how you perceive the data and me'll you

mentioned about like it's how we look at the data right and that's all comes from when are we playing it or should we say when someone playing it from my point of view the biggest data that I've taken is from watching play tests you know sometimes we'll have like player research labs we'll get people who've you know come off the street literally but I've signed off and they've come to test that and what you need to look at is how are they playing it what is the state that they're playing it in terms of the game like what are the controls and all that starts to feed into it and it's about like how do we digest and take that because what actually might be an issue you mentioned heat maps in saying about like the I was looking over in this direction well what could be is that they were looking over in this direction because of the level design or because there's something maybe not functional with the controls at the time so there's so much to take in and like I I agree with everyone it's about like how we use it and how we consume it because we keep saying data below there's so many ways that data can be used so many way interpreted and it's about though saying

is how do we how do we get the information I think it's it's obviously be helpful if present if not people like Jasmine in the UX design are there to help digest it relay on to myself and the team as well so it's about you know there's many things go into it but for me yeah it impacts my my job on on many different stages or in terms of crazing levels but it's interesting I think because you talked about getting people all straight obviously not literally but generally we're paying those people to come and play the game yeah and the behavior difference of that compared to say the first 30 people that you get who are nothing to do with you and seeing how they play a game those are very different types of questions that you're asking and different responses that you're getting I use a method that I discovered in the food industry that was designed by Pillsbury and NASA to feed the astronauts it's called hasip hazard analysis and

critical control points it's just a fancy way of saying we look at the flow of the user journey and we use that to decide what are the points of data we can capture and what are the expected results and then we use that knowledge to test now I mean do you guys have particular processes and routines that you look to do to you know ways that inform your thinking Jasmine do you when you're looking at kind of assessing a game do you have a particular model that you work to well it first it really dependent on where we were in production is it a live game are we in early production you know are we just about to wrap up you know there are very different ways to do games user research and to validate you know depending on where you are like currently I'm very very early production so we do a lot of just you know audience research and work with data like from Quantic foundry which is more about motivation models and understanding what kind of features we are crafting and what we set out to do and and map them against that that we

are doing the right things but when it came to live games definitely you know there there was actually again it's a vast array of tools within that that you can I use I did surveys so and also I had you know players over for interviews so it's so it's like a mix of you know qualitative and quantitative data and you know just as you say we were we were of course looking at you know either the lifetime of a specific part of the users flow you know within that depending on that you know it also you know we negotiated okay what is the best way to find out about them like you know qualitative data or like you know user interviews we we were doing for example for really endgame players who played a really long time and we really wanted to understand they were extremely involved into a social game plan we really wanted to understand okay what is it that they truly care about within the most kind of social circles so we felt like okay the best is that we sit next to them and can

ask them about certain things while maybe for for early game it's it's more useful to you know look at quantitative data and like to just measure and see okay for a bigger sample of people how's the experience of this part of the game and then depending on that come up with solutions of you know either you know changing certain certain balancing changing something there or introducing you know features to you know maybe make the journey a bit easier for them it sounds like what you're saying to me is that you kind of there's a lot of thought experimenting this the early stages I mean we might have a persona that we say okay so this type of personality or we might use them like the game refine you've got a bunch of personas that you use there are other companies doing that I find that but someone particularly useful um but then as we go on through the development side of the game it sounds like what you're saying is we're refining and we're kind

of replacing our thought experiment with an actual piece of evidence is that I mean it's an always ongoing and iterative thing I mean we are learning you know every day you know it's not something that stays static it's like okay today we are gonna try this and you get a new kind of data but yeah definitely and you know that depends also on the company the kind of game that you are making and you know the culture of what kind of approaches you take and what you see is most useful to you it's you know as you know was mentioned already before it's the team finding the approach also what works for them so a meal I mean for you what is there a particular formula and does it also change for you at different stages of the development process yes I think jasmine touched on a ton of different points I think one of the starting times I think is worth looking at and something that jasmine mentioned at the end of her her space ride was culture and how important that is because you

can bring all the data in the world to the team or to the studio but it depends on who is the team who is the studio that you're bringing this data to and do they trust Quantic boundaries the datasource do they trust a youtuber playing their game more than when you pinpoint you canyou can find hundreds of data find seeing like looking at a heat a heat wave is a great example right like that's that is a very clear problem in a very clear solution more often than not it's not as easy as that but take the heat map for example you can bring out a heat map and say we might have a problem here but maybe it's not tangible or painfully tangible enough for the level designer until he sees a video of a streamer playing the game and failing to progress in the game so I think it's always starts with establishing as a studio or as a team or you know preferably both how are you gonna approach data how do you as a team wanna work with the data that's coming in and and what data sources are you gonna are you gonna work with that doesn't mean that you as an individual on a team can seek out new

knowledge in the form of discord use of surveys or do focus group testing or we haven't done that it's not great right but but I think it speaks I really like Jasmine's point about the culture like that that that is gonna be the starting point for every single data set you're gonna get in but it's also going to be very important in terms of are you gonna actually be able to act on the data that's coming in because you can get the biggest pile of data and just list after list of like we need to fix this this and that but if if the team you're working with don't understand how you got to that conclusion you need to you need to take five steps back and you need to take them by the hand and take them on a journey it's showing them this is how I came to this conclusion this is the data sources this is a methodology that we're using to take this stuff out because then they understand the the journey of

like how that data came from the game that they worked on that they never like they always always build it with the intention to get it right the first time but you never do so making sure that that you sort of taken by the hands you're then we're not we're not saying you did something wrong but it's not working the way we wanted it to let's let's work together as it seemed to fix that sounds like you need a culture of curiosity yeah and I also think like our data analyst for instance Maya she's a fantastic data analyst she found a great article on medium by Matthew Emery whose previously been a clue who talked about like this method of sitting about gripped a grid of hypothetical features and the potential impact on LCV and the potential effort for that feature and by lining it up in a grid is it's not a NASA process but it's the simple process where everyone can understand I put in this much effort when we pretend to get this much out if you can get the team in a room together and agree on what's the

effort what's the percentage of win then the next thing that you go in your team backlog becomes very evident but I again like that that process makes sense for us right now where we are because we're live product it was vastly different when we were in early access and it was again vastly different wouldn't move from early access to live no serves to early access early access to soft lunch and this from soft on sly Bob's so Jasmine's point about sort of what face are you in and what data do you need is I can't applaud that enough that's 100 percent one point yeah dig down a little bit further and that sounds to me the further we are in the earlier we are the more qualitative we want information sometimes generally the more quantitative we need sometimes yeah yeah it depends right it depends on like how big of a game are you're building everything again they can can kick out the door in a week and test it on a wider audience do it like don't hold back okay and snow and if it should always feel bad like when you're sending something out into the world it should

feel a little bit bad when you're sending it out with this an indicator that you're just pushing the envelope enough right you shouldn't feel comfortable Ike this is gonna work perfectly for us then it never does but getting to that conclusion earlier it's gonna be a benefit but it depends on who can you reach first if you don't have a developer license yeah pulling people from the street if you do have a developer license and can ship your game on Google Play tomorrow do it don't don't hold back you can always relaunch the game with new addresses or under another name that's fine yeah I mean it's not people talking about real data really makes a huge difference again the real data early and fast I think matters I mean that's I mean do you agree with kind of the phrases or anything you think this data like 100% this is the winner this can be used using in our game in future but otherwise this is combination and what I

really like is this combination when you look to data and also while you are checking with feedback and combining together and also animals very right you you have to understand which we talk to in in your team because this is combination of organize and put put things together and I feel like I think I think not Mark Pincus said this once when he was asked about Zynga and they are so late or they even and here he answers someone like something like that the data are good to to rice next better better question and never give you give you straight straight and I really like that because it's basically like the deep-diving more understanding what you are doing if you are following your your vision this is this is much more about data there is no like a be test it's it's only only

few times when you can decide based on this and what I really like and I believe especially especially in PvP games and world is a sport Chandra and you mentioned it Oscar is this data which are used also also to players back then players can improve their strategies and to look to result of the matches and and statistic of the gyros etc this is this is other part of the data and here I'm in India I am comparing what we are doing with in in PvP games with sport and look to sport how they develop their strategies based on statistics like Premier League in soccer it's it's amazing and I think this this the constant and slightly a slightly it's visible already in eSport yeah I think that's kind of interesting particularly this idea that and the data

the questions we ask in data are to ask more questions and being undefined that balance between from a questioning point of view whether or asking someone's opinion or whether they were asking to see what their behavior when I'm trying to help people with live off as I'm looking at audience sentiment not just in terms of you know what they say would also be able to measure that against what they actually do in the game I'm I'm interested in seeing what the the market you know outside of this game what that's the general sentiment is there or around games jobs onra and also then to being to compare the data performance of the particular game I'm helping all of the nearest ago I think it's always for me about finding there was its balancing point between and you know emotion and and factual behavior but somehow managed to find a way to get that back to the original vision of what design the game was a

I think if you lose track of the original vision you generally break the game I mean max I mean I mean does that make sense to you does that fit with the way you think yeah I mean it goes back to your it's a great point mate and you know everyone basically took all the great answer so thank you guys you just left me with nothing appreciate it no so but like what you mentioned there Oscar Mayer goes back to the original question that you first asked on this one this panel starts it's like how much do we trust data how much do we give to data how much do we give into our design to data and that's that's the balancing act because like we mentioned there that we're looking at these systems we're looking at this feedback why is it towners is it raising more questions and then sometimes it might raise questions and like I don't ask them anymore because I've got I've got some kind of indication of where we need to go and that's why we need to press forward but it's it's always funny especially in games because we all start virtually I kind of paper design or in our heads or on some way before it touches a computer and what it always finds out I'm sure

we've all experienced it it's the fact that what you think is going to work doesn't always work necessarily or translate well and that's the toughest part but because of that beginning player feedback testing through all of this you eventually find out that okay it might not work like this but it'll raise the questions and also the inspiration to get out okay well actually let's try it this way and I can still reflect on the term of design so it's that balancing act like we've mentioned at the start it's always about trying to make sure that he answers the questions and ask the right questions for you to me it comes back to the quote there's always attributed to Henry Ford I think apocryphally that if I asked people what they wanted they'd have said a faster horse yeah we do no not say that doesn't matter the point is that and knowing what we can what we can realistically ask people and what what the data actually means I think often something we we don't talk about I think that amenities point and we talk about

a/b testing all the time but maybe testing is not the whole of data it is a really important part missed by lots of people like Jasmin's born earlier but it's also part of a journey of you know using our the senses available to us as a game developer which is what do people say what do people do that's it and there's that other famous quote which is like you tell people what they want you know like it's it's all sat here the end of the day to believe in why we've chosen to make the games that we've worked on you know follow the ideas and like we believe that there's something is there like we might have to refine it smooth out the edges but you know we've got a march on with with the visions that we that we all choose on here today so another one of my theories is that not only we have to pay attention to the lifecycle of the game so where we are in terms of production pre-production early access I mean depending with your PC or mobile when you're doing soft launches or life and

and ongoing live stages I also think we need to take into account the lifecycle of the player and to me that's part of cohort analysis understanding the difference between someone who's just discovered the game when they're going through the like the learning stage whether they how the game fits into their daily routine or not and and then the people are engaged maybe even super engaged even those people are about to lap so that how you knew you know re-engaged a sigh let's call it or if they returned I think are actually asking people different questions depending on where they are I think it's more informative and at least that's my way of doing it do you have a way of looking at audiences how do you split up the types of people who are asking information from Justin together thought on that yeah at my last company when I worked on life games we worked with audience segmentation for example and again I can just say that culture is really important at this point because it's so many different disciplines that need to work on that because what we are talking about is all the primary and

secondary you know game systems or the game mechanics the rules you know and and and refining these kind of you know the economy and everything that is going on because that is part of the flow so you know game design analytics product manager and UX need to really closely work together and you know one of the things that we did that I thought was really useful was audience segmentation depending on specific parts of the game which wanted to try to figure out how and this is exactly the kind of life cycle cycle thing how do players transition within you know two different either you know play play modes or you know become interested in other features and therefore mature within the game so we did a really we basically looked at who that with the data analyst and they did you know I have to say they did do all the work I always feel like ok I am there and I helped to drive and I get informed about

the process but they they are and this is something I really can also only recommend to everyone has people there who are in charge of that and to help drive and find out about the things but they really took a deep dive and informed us about ok this is what we found out you know what do you guys think does this mean and we found exactly that that there are specific points that players you know turned heavily and they didn't transition to you know other elements of the game that we thought where we wanted them to go like this so then we could start looking what are the you know problems in the flow where do players get stuck and often it was you know really simple game design decisions just in balancing to tweak a little bit just to be able to level up a bit a little bit earlier and have something available that would help players to transition okay so the transition point thing is there really vital Canterbury's and as interesting kind of looking at a game in that basis

I mean if you have any kind of particular tricks techniques you do in terms of looking at kind of the player themselves I would say it starts with a decent attribution platform you need to know where your users are coming in from and who they are at least at least at the point where a player who found our game well where we were in early access they behave vastly differently than they did when we soft-launched it and to some parts we actually based some of our decisions based on you know less than ideal data the only way we discovered that was by turning on your site position and acquiring users in by a page Equus rather than just the organics because the people on Google Play god bless their souls who find your game by the beta part of the Google Play Set are not like any other Google Play players and so if we based our entire business strategy around sort of oh wait you know great numbers throughout then

then we're basing it on like a tiny cohort of a special tiny cohort that we then extrapolate to see like these are all the players in the world it's really important to to be able to take apart the top of funnel data because that has a triggers an effect throughout your entire analytics thing and your entire engagement stick if if your base so so I would start there always and be aware that the players would find your game while you were in early acces are not likely to be your your you know global audience we were consulting on a game that had Krishnan and a Ronaldo name backing it and almost every single user that came through his tweets were useless actually utterly appalling ly bad yeah so it was criticized him but that's that's a great point right like you you would think that people who like Ronaldo and are on Twitter might also

lie but that turns out that's not the case like his recent the audience he had on Twitter born not a great match I'm sure there's a great match for people like Ronaldo and like mobile games but that was the wrong that was the wrong channel to capture them right exactly and that's that's a good finding but it would be a terrible terrible way to base the value of the game itself on so he was heartbreaking for the client like I'm telling you yeah but because that's that's likely part of the deal right we boy I've worked with licensing before so that that is that is heartbreaking but I would definitely say like Chava phone start there and also another tool is cluster analysis because it helps you work out you know work out stuff you're not smart enough to ask yourself so sure now channel some data and see where where players cluster clustering based on are they engaging more with the game then see okay let's split that up into clusters and see why are they engaging more with the game are they playing a specific game or more are they playing a specific map more let's thing with specific gun more other more social than

others because then that have sure that helps you understand your users better and not it doesn't force you to try and work out why they're doing what they're doing it lets the players answer for themselves basically through data make sense Millis I mean what what you have a particularly honest response and cohorts but what this for us crucial crucial data decisions around bouncing and matchmaking because this is this is for PvP I can never ending story and you you optimize and balance on the way and it's really interesting because basically you fight with a few things you have people

which are in the beginning of their journey in game on the end these top players and you have different liquidity in different time zones and and of course simplest solution is put in place some books some players which are controlled by computer and and this is this is really like interesting environment to find way way forward and what works for us because we have still dislike ambition be we as a sport that be very transparent then if there are players both then we are saying it is two to two players then I understand okay I'm playing against both and don't wait for the microphone next next 30 seconds I can play immediately then there is that is this this specific for PvP genre and and it's interesting interesting thing fine balance and of course how I am saying to you guys here

some form of unbalance is is healthy because because people people looking for for new strategies new ways and be better in the game and this is this is all the stuff but but huge it is balance of course of course one was always looking for exploits with PvP I mean it's kind of part the fun is how to break the game so running data analysis at the same time there's having enough space for people to explore what the game can do oh that's a toughy it's fun but it's a it's what makes the making games all that so much fun but it's tough yeah and now we are solving issue in this other dimension because you have of course tokens which combine two players but this Underpants combined combined players from from like different times of lifetime the players and this is a huge challenge for

designers for us and we are using using data for our decisions Abby's not in our predictions it's tough I mean I'll buy you another time about a model weapon of choice which basically self try to self balance quake so that newbies could play experienced players that could just a way to make sure you don't break critical mass in the game like that is so tough so max and self any particular tools and techniques that you you want to share with us I just disagree with everyone I'm gonna spice it up and just say you know no surprise it no I mean everyone's kind of covered kind of what you know I'd say but instead were did it way better but for I guess it just again it all comes back down to just like know who you're designing for like if you're designing a sequel as well you're probably gonna go more to the your existing player base versus a new IP in

an understanding that you know it's it's a big thing for it but no like you know everyone everyone stole the words again guys thank you I appreciate that I'm gonna disagree for my next one I promise you well I'm gonna I'm gonna kick off the last question we disagree for the last one now everything's broke this one so what's the one tool or technique do you think anyone who wants to use data to improve their design should do throw me the hardwood that's a really good question that for me I think especially this is a level designer because we don't really get access to tools that's known people like something in Jasmine's or meals or we losses positions to decide on what they are but for for me the best thing to say especially as an LT someone who's aspiring to get into the industry is an LD is make sure that you do just get like player sessions in doesn't try not

to bring your mum or your brother or your father and obviously because it'll be very biased but do bring in you know people to test your stuff and just sit there silently just watch it's the best tip I can give to two young aspiring Aldi's you know I know it's not technology but just the simpleness of that can really help your your levels there so yeah I mean for me I think if I can not be in the room and I can observe them by camera through a glass yes and have somebody who has nothing to do with me introduce it to them anything it takes the the user away from me yes some better the information I get out of it no I agree like sometimes I'll come in partway through a test just so they don't know who I am or my relation just sit there sounds weird saying out loud sit there and watch people but you know for the sake of research that's why we're there so yes I can't help interfering and as soon as I interfere everything's easy great yeah you're doing it wrong it's all your fault

so what sort of one tool or technically you'd ask people to take away with you I have to come back to the culture thing and I just think you know design culture is something that's often you know just not maybe more recognized nowadays but maybe still something that you will struggle with because you know just as Amos said it doesn't matter how much data you have a bailable what sets if you don't have the by and from your team from your management it's important you know that to understand you know why are things they did and you know as you guys mentioned in the start it's not the single only source and it doesn't give you answers per se but it informs your decision so I always thought that involving everyone just not only make people give people a chance to understand how we do things but also gave them a chance to participate and you know in the end that is what we do we hire people who are you know most often smarter than us and to

be involved in what we're doing no matter the position you know so culture culture culture and but I really also appreciate definitely the the points that max made about you know it's really hard even for senior designers to not tell people in the room how to interact with things it's really a challenging thing no matter how experienced you are so it's always good to remind yourself and the team how can we remove ourselves from this so that we are not leading people yeah absolutely I mean yeah I would say think very carefully about what face your product is in and set clear goals for whatever kpi's you need to set to make sure that you leave that face at the right time it's okay if the KPI goals feels uncomfortable that means that you said it rights but you really need to think long and hard around what KPI goals you need to set that also forces you to have a frank discussion with your management team it's to your

xx around is this the right time and what's the expected timeframe to reach that goal because then there's clear KPIs and you can also have start to have a frank discussion around what what do we need to prove at each life like each cycle of the product life cycle and yeah 100% agree with my ex it's it's terrifying watching people take a game of part one of the things that we've done is when we go out we always open with saying we're testing this for another studio we didn't make this that means that it's easier for us to sort of be an unarmed saying we come clean of course by the end but yeah but I would say like really think around what stays your production is in and make sure that that's aligned and the expectations are aligned with your management and your team and of course like that itself being stabbed in the hall but you got to invite him you gotta want it yeah Millis I think that's technique before for people in distress playing games and I

think this guy's from bi and frankly best hires which we did to Jeeva we're exactly like guys who sit down play game and to look to vote what was generated in data and start understanding and connecting these dots and and I think the construction also is something what is really really useful to other games and learn from it how they built what do you think works for you as a player this is combining with flavors think again I think that it's exactly exactly daily life and balancing looking to the data and display experience emotions which which you have behind game as a player it's is something very useful and the guys who like understand at the beginning there's a line that Stephen King has in on writing he says you're not a writer if

you're not writing two hours a day and reading two hours a day and I think the same thing applies to game development if you're not playing two hours a day and designing two hours day you're not a game designer and and I'd add to that if we're not taking data in that design we're not a game designer either but then now I'm being a bit unfair on that I mean if it was gonna give one bit of advice though the biggest lesson I learned was actually just knowing what question we were asking and asking the consistent group of people that question so looking at people from the point of view of like day one of them starting to play so if I compare responses from people who've had the same number of days playing the game I'm gonna get much more useful answers than if I just ask a blanket group of people I need a common starting point otherwise I don't understand who the players I'm talking to are so that's us I think we're out of time I

don't have any questions a Martine hopefully we haven't got over too much I thanks very much guys I think you did an amazing job and Martino did you extra votes for the great panel yeah we were a little bit over time but that doesn't matter because you were the last panel of the morning sessions did I know that