Best practices in testing with new partners

[Music] well you a quick intro one at a time just tell us a little just tell us your name of course your role and just company and we'll get it to take it to some obvious questions that we've already discussed earlier put the answers up or gonna let it happen free fall so I see everything from corporate branding to community management to you a in the marketing and the last year we spend time testing the operation a Sony

I'm from Mobile which is a Twitter own company my current role is I'm a pleasure and I help with adoption of all kinds of ad format especially the engaging ones both on the supply and demand side and my background has come from a network DSP Twitter owned and operated and now over hey guys yes so I'm Matt Saunders I'm the you a Directorate Network so Network it's a game publisher we have we have our main game is ledger to game of Heroes previous that I was a you a director at Zynga and over my time eight years or so an industry I've probably tested pretty much or every major network so yeah I've ran you eight for eight years basically yeah hi my name is Jonathan I work for miniclip as the head of music position so I'm not looking all day using

position paid marketing efforts across multiple titles globally across a long range long list of different channels hi I'm for Han I'm ahead of you am none of it I've been a year or so and we have a team of eleven 3ua manages six artists and I was a managed community and our major titles a Hollywood story and my story be an interactive story outlets do predictable moment and we scaled you a a lot there and in the past I was that social point where low time titles like dragon city and monster legends where we ran God knows how many networks so like mom I think of a lot of them too great so let's start where most people start the conversation when they think about you a and that is with Facebook Google other self attributed networks that are obviously up-and-coming just an open question I

know some what we have are pre prepped for this conversation some people had noted that they wanted to chat about that others will chime in but the question is how do you view those self attributed networks particularly the duopoly of Google and Facebook when you start your you a today how might you think that might change say 12 to 18 months from now it may not change any of you it might based on what's up and coming elsewhere in in a media channel so I think Matt had said you would talk about this so yeah how do you view it today and what do you think's coming yeah I think it's um I think it's really interesting right so Facebook and Google is pretty much where most people start Andrew for us as well when we released new games we start with those channels and that's typically because they can drive the highest LTV because they have the most information on the users I think but if you think about what's happened in the last kind of 18 months the way you buy on those channels is fundamentally changed so it used to be you'd go in you would find as a

targeting you would find loads of information and you would manually put that in and then and then said a kid now it's moved primarily at least for us to value automation value optimization an app event optimization so the you a man user input is really changed so instead of us finding loads of information our input is essentially a budget a goal whatever that may be and creative right so I think that trend will continue and I think what you see with the other networks is they're all trying to follow suit so pretty much every major and mr. DSP or an SDK network will have some kind of all to optimization product which has similar inputs right they are yet to be anywhere near as successful as google in facebook my belief is that because they just don't have the level of data we're still going to have but I think over time that will that will start to change and these products will become more and more successful I can see the duopoly of Facebook and Google changing because they just have the only reason that changes is if the user base itself changes right so if snapchat release the same tools and

their user base becomes mature like Facebook's then I think that can potentially change but I think the way you buy on the other networks will will change and the next kind of 18 months I don't know what else his opinions are so we started testing Google basically the beginning of last year there is a weird sort of sort of conflict between Facebook and Google where both of them have really good things but like none of them have the exact right model I think where is Facebook easy to do small things quickly and see if profits come quickly Google you need to take a leap of faith in their system but if it works you don't really have to work that much anymore if it works it just does its thing and like where Facebook hands on but you can get things quickly so if

someone can combine those two two things you have a winning model I think the only one thing I do want to add here is I've been hearing a lot from yesterday and today is debunking her bidding and in happening and I think that's the only nuance to the network piece which might evolve hopefully significantly and quicker this year 2019 as you already know more pub is a preferred partner with fan and we are testing advanced in appearing with them so I do see firsthand the sort of investments and changes that are on in buying real diverse is the programmatic traditional way I think that's the only change that I feel will come in 2019 is when fan fully scales on the inner head opening piece so you're mentioning fan but are you also just suggesting outside a fan and once a prominent name like a fat dots a trend of in app header bidding it's tough fathers not to follow sweet so I think to stay competitive they will

have to move along with it yeah we were hearing the same thing right the advanced bidding header bidding they're interchangeable will change the landscape outside of the self attributed networks and fan included right because fan is not part of Facebook's so attributed wall so that's interesting so what I'm kind of hearing is that Facebook or Google they're a good place to start but we see on the horizon that the other channels have an opportunity to gather than the implement the data necessary to sort of start to emulate what happens inside of a cell for chip it'ld is that sort of summary though I think most of the channels typically follow suit right you can excuse me you've followed a market leader right and they are behind and the reason they are behind is because of the data right they just don't have as much information but over time these malls become more and more sophisticated right so yeah well I believe gules of it they're behind Facebook in the sense that switch from AdWords to more a black block

solution with UAC and it makes it really tough for smaller developers to spend on Google profitably because the budget required in order for the UAC algorithm to work is quite significant so it was a certain add a bit we were mostly spending on Facebook over to very recently where we made enough money where we could throw money into Google that that kind of men Google has like that huge reach and obviously the scale is potentially there but it's really hard for many developers to actually get into that space with with Facebook you can spend a couple hundred dollars and then see some results that's just not possible on Google it makes it hard to when you're launching new titles to test use Google as a testing Network because it's yeah you're in a position where you don't know if it's when that's come profitable and it's whether it's a good indication at all because you have to put so much money in there so Facebook is always to go to platform and

that's why in facebook it's a lot easier for for developers to use than Google but having said that rules and X behind that and we use snapchat heavily but that's a very specific type of audience we have like young females so our scale and snapchat it's it's larger than Google but I think yet there's gonna be fewer networks coming out and stuff like tick-tock which is a huge steam massively that probably will be big and Pinterest is growing but um yeah overall Facebook is leading the way with vo and and that's why people putting all their money the point I'm glad you brought that up in terms of you AC and the challenges that might exist for an early-stage paid campaign I hadn't really thought about that thing Saints for that that but it's sort of budget restriction restricted and it might not be the best thing out of the gate for you know an early stage so cool how hard to recommend for smaller companies I would say like okay well move on to the

next question and so this panel is a bit about you know we started with Facebook Google the obvious itself attributed networks but what about testing partners like how do you go about that there's a I so I didn't mention in my introduction but I run sales for our key in Americas and oversea amia flew in from beautiful Seattle by the way so a bit jet-lagged but it's ok um but anyway I talked to customers all day long and obviously I'm wanting them to test us as a DSP and I run across all different types of organizational structures and orientations around how they go about doing that some will orientate themselves in a silo I'm Anna Geist itíll exclusively so vital across social search and and networks if you will some will create a team that's social only for all titles there's a lot to it and I find that it

there's no consistency around it and that's okay we're still very fairly nascent industry but I do want to hear from you guys in terms of you know what do you look for when you're when you get the multiple calls a day and emails from networks DSPs etc what do you look for in a partner and how do you go about that yeah so in terms of the channel makes you a channel mix it's very important that you a managers are careful about what channels they're adding in order to ensure incremental value for their channel mix so the danger going broader with networks outside the typical channels such as Google snapchat or Facebook is you will hear about lots of network companies that engage in the same you know ecosystem and the sense I'm gonna add inventory and you step also into areas where a thought becomes a bigger risk so it can be quite overwhelming especially

if I'm a developer who enters the you a space for the first time talking about Google and Facebook all the good starting points but if I go beyond that that loads of companies you know thirdly you know more than 200 networks any given time my active running UA campaigns how do I know as a starter which one to trust which one should I prioritize this there are very few things you can really judge from from the outside so it's about really knowing and asking the right questions around what kind of inventory in what kind of space those networks act what ad formats how do my campaigns how are they shown to the user having that right set of questions right from the beginning helps avoid a lot of mistakes I think - completely right but you take that a step further you know we have and I've always used this I have essentially a

set of questions which I asked right which is a pretty standard and I think having a standardized kind of questions that you will ask it's really important and that way you can treat everyone fairly right and I think the crux of it is traffic right if I'm going to buy from you what might actually buy it who was who am i buying hammer buying if I'm books for you it's what woods view right if you need if you want more Facebook traffic then don't sign up a DSP right if you want DSP traffic then don't go in sign up a Facebook agency right so understanding that Anna and I think you know it is hard to understand who's I mean I I always use this example I get I have maybe 200 300 LinkedIn requests from salespeople who I don't know and on you know and it's really hard to cut the cut through it right and actually when you think of where the traffic is owned it's actually a relatively small number of partners mostly people who just are just reselling on top and you can use use events like this where it talked to people use your network to build piripi R like review is common in any industry

and it it is true about ad networks as well right I think and you know people you can find out really quickly who's good and who's in who isn't or who's a bit shady it's also not about the quantity so it's this is also sense of okay the more networks and work with the more exposure I'm getting right but but it's not really like that I've heard about a load of established UA teams who really have a short list of partners below 10 partners and they reach the best results for that while others who kind of go out and work with 20-plus networks they they struggle do not actually get what they need I think it depends on your game right so like we was we over the last year we've been spending like 100k a day with nine partners right we don't you know buddy if you're a maybe a more casual game maybe by and you have you have a lot more partners I don't know yeah when we were at social point for Dragon City Muslims we were running like you can say

to my partner's like twenty thirty at once it was absolute nightmare to manage and then when Ford when we started tracking fraud to realize the half of them were fraudulent and then there's lack of transparency as well that's how we used to really go at the beginning of like who's transparent so when you looked at like a blob and a vongole they would say okay that it's coming from these sources and then you'd run would one other you couldn't tell us much about what creative is working and where the source is from and then you starting to think like how much money invested in these guys that are you know either doing like some kind of click injection or or like delivering some kind of fortune installs so like their social point for Dragon City for example we brand like let's say 20 30 partners at once and there's a rotation as well between knocking back in but we weren't running Facebook because it wasn't working for that game which was strange but the reasoning was that the CPI was too high and it wouldn't back out even with the added conversion but then now obviously I have

that none of it where we've run pretty much three or four networks and I'm spending seventy thousand dollars a day roughly on three or four networks which is what we talked about earlier snapchat UAC and Facebook and not running that other networks because generally they haven't backed out for us because of our ROI targets being very harsh initial period because we need to have low retention so but we've had huge scale and we're like now top 40 years of today for my story in the download charts in the US so like you don't necessarily need to run more than three or four networks to be like high in both the grossing in the download charts I would like to add some point which is my method actually when I talk to new people is this is a very good test for me is if the using this partner means that I need to sync with them every week

does that make my work life better or worse so it's a very soft test of do I want to talk to this person even and if that usually that passes like like I I'm very strict about that if if I do like like this if I know that a meeting is coming today it's like okay maybe do something else it's not nothing is that critical yes usually that's at the point and I think I'm gonna zoom out and experience some zooming out and thinking more from a supply standpoint and the reason I bring that up is we see both right networks and DSPs plugged into the same supplies also I think with your coin there are very few organic supply sources out there everything else is just pretty much being recently sold so I think the sdk-based solution is first of the easiest the least hard work for you a manager's because it's not a server box so that's one take off the second thing

I've been noticing at least from 200-plus DSPs that are plugged in and I think we have about 15 certified ad networks that we work with we don't work with anything beyond that so to your point there very few that I was plugging in what I do see is that's free entry approach from previous um managers where you go with 200 networks and then hone in on what works I think that's dissolving which is great to see I think by now everyone knows who they work with word of mouth matters and then from what you brought up I think that's a really interesting point too and I think a lot of people are not educated on the DSP side when they're trying a DSP for the first time because your first go-to is always fan and I think the way you're used to capitalizing on a KPI is very CPI heavy so when you meet a DSP that making you buy on a CPM it can be nerve-racking because you're not used to seeing a figure on a CPM basis so now you have to do a reverse math so I think it's it's really important for this

audience to understand that there are types of DSPs as well there are no types of networks they're obviously video first networks are not but their mechanism of how they buy and logic is the same DSP is managed leg and a self-serve leg and I think I feel his pain because when you I think there's certain criterias you have to look at in terms of maturity of the app it could mean you have a robust team that runs you a and you want to take everything in-house and that's fine go use a self-serve DSP you don't have a robust team you don't have a creative in your attribution is in-house it's not to a third party those are key factors for why you should be using a managed service is you don't have a team yourself too much effort to run a chaotic system like ours second you don't have a third party tracker like kochava add just to post back to your time information to the DSP to make it work so I think it's really crucial to understand that the DSP lense is not one type there is a managed service and I

think what will help you in not engaging with that managed service is a lot like I give you the budget I give you the KPI do it come back to me less headache but then the self-service you're super involved yourselves then you expect it to have a team that runs it and looks at it day on day so I think it's crucial to understand when you're going from the network to a DSP per impression buying mechanism like choose the right partner there are two hundred-plus of them I think the you a forum or using like I think Jun has this certification process where they tell you who are the best partners of where they are seeing the best performance results or you have partials like mobile that have division that's dedicated only to get the advertisers to jump start and and educate you on what works best for your business so I think those are channels for you guys to start looking at as well thank you that's great I just want to revert back to one thing that matt said about so with at our key we love the hard questions and I love that you have a script of what to ask and we encourage

that and where I'm going with this is there are lots of flavors of networks DSPs is sort of this intermingling word but and tarik is pointing out a - is like there's not one way to do it and but we like to be asked the hard questions like what is your fault QPS volume what you know convince me you're a hundred percent programmatic right that's a very important question to ask a partner because there's other ways to sort of cobble together a network or a DSP and we particularly champion the idea of being a hundred percent programmatic right buying in a unified way across many exchanges we believe that that's what will be disruptive and what we talked about earlier when we get the data that can emulate or simulate some of the things at Facebook and Google as a programmatic space so I love what Matt said so I want to make sure that the audience you know take

that in like ask your part if someone comes to you as a potential partner ask those hard questions to make sure you look under that hood I've worked at two networks a two at two DSPs kind of been around the block I understand what's under the hood and and so I really encourage you to ask this okay I think being skips cause very being skeptical I think I'm British so I'm not even skeptical but I think it's I personally think in this in this situation it's it's pretty healthy right and I another rule of thumb I have is if the salesman told you that volume take it at least 30% off and you might be roughly about there right but like you know the people are selling to you and just like in any form right just approach that with skepticism so we're gonna get a little bit more nitty-gritty on that what I've heard so far is you know we started we're not normal normally everyone starts Facebook Google and their self

attributed we talked a little bit about asking hard questions outside of that space but let's talk about when you get through the 200 incoming emails from the sales reps and you decide to take the phone call what kinds of things do you consider to prioritize and I'll throw some stuff out there so there's different pricing models outside of Selfridges there's CPM there's CPI some folks will throw in some credit to help you get started there's many different ways it's really not a consistent marketplace but talk maybe give a couple examples or tell me about how you do prioritize when you've narrowed it down okay fine I'll take your call and maybe you set up two or three calls in a week what do you look for to get something so in terms of the pricing model while acting in the performance marketing space wherever things are tractable we obviously focus on getting those CPI deals or CPA deals even so we see it like an evolution that goes

beyond just the install going forward forward you know similar to to Google innovation right so going beyond the in-store trying to achieve results in terms of in-app events so these are the things we look for generally speaking it's a CPA or CPA obviously we haven't really talked about it campaign set up in general we need to make sure that all the technical requirements are obviously checked so you know the tracking support and and all the you know tools that you need to have maybe you have the API with 14 that you can fit into your reporting dashboards stuff like that you always check for and one note on the ad credit or test budget I know it's it's it's obviously a nice proposal and whenever a network comes and says hey 10k free free test it sounds nice and you think you're great doesn't know no risk but you have to be careful because some networks may also

use that as a bait to then you know get you really great resource and once you sign that are you and actually pay money things suddenly turn around so there are some horror stories and I would be very cautious with proceeding or just being blind when there's a free budget coming when your way people to because the CP I think the Jonathan mentions just had to be a bit careful there because for example we we spend three dollars an install more on Facebook and it hits our y targets but then we might use video networks like lights of on go out love in unity and we have a CPI of $1 there and not yet the ROI is worse so you like three to three times a difference you have Facebook is stronger and the reason is obviously because the conversion to payer we see is far stronger on Facebook because we're optimizing towards purchase so now some of the networks are optimizing towards purchase or other events and it's important to be able I think those those guys are going to be the ones that do all because that's getting closer to

Facebook if you're able to target installs based on optimizing towards either purchase or ad revenue depending on what kind of game you've got that's gonna be important but in general I I'm hesitant again like man Jonathan is with most networks and I go with those guys I've worked with in the past all the people I know in the industry in it for long enough but when I am open to new guys like I started testing a new networks that do snapchat Instagram influences I mean I did know that I he was across installed before but yeah that they were there we paid like one tenth of what we found Facebook so even though the traffic is convert six times less than a dozen Facebook it's similar to organic traffic basically we still get great ROI so you got to mix and match these things correctly but obviously checking all those fundamentals like fraud and all other various stuff but it's a difficult place if you if you don't know the industry well it's difficult to know where to start and it takes some experience I

think I think when it comes to pricing models with its CPI CPA a CPM it's all about your attitude to risk right so if you are buying CPM the risk is on you the app developer if you're buying CPI that's lower risk and more risk is on the network and even more so CPA right so if you are confident and if you have good data and if you believe that the algorithm is there and I think CPM can I'm we in our world because because the game we have to quite hardcore so our CPI is a super high right so we found a publisher which works actually we because we want that first impression would I start with a network doing CPM bidding no no not unless I knew them really well when I trusted that doesn't happen but over time I think you you can you can move you can move towards that right but my opinion is you should take is lower as lower risk as possible in the day when whatever the DSP is or whatever it may be it's it's a lot of it's coming down

to the way you want to work with them to your appointment it's a managed service or self-serve they might be but it's basically tech it's how effective is their tech at buying versus and other DSP well this is another DSP right so when you never tried or you're like leaning heavily on organics I think your approach is absolutely perfect like you should never take that risk to start with especially when you're trying to gain your first traction of foot in the door to gain peer installs it's just unhappy to go with the risk your mortal touching another DSPs and program alia what Matt said for me that's risk because they often love them and say we need $25,000 to get stuff to work and that's for like us iOS and that will be spent okay spent over two hundred fifty dollars or two hundred dollars a day but it takes time to for the exploratory to find the CPI and then find the conversion but that's a lot of money to commit you know even if you're making large amounts maybe you'd rather put that on Facebook where you know you can

get it back so that's a hard to test the ESPYs particularly I find and because if you put that money in then it doesn't work on that DSP you think okay I'm not gonna use DSPs at all yeah I get that a lot by the way you're absolutely right because we do this sort of a consulting thing where we get you guys to try DSPs who've tried networks and I think the best approach that they come up with is we know our audience and I think to Matt's point earlier once you're confident of your audience and confident of a CDI /r by that you want achieve from these users the typical way that I've always encouraged anybody to go into the DSP world is do a look-alike just go blind with what you know works and find something similar in that space because that's when you're not burning money it's not like the exploratory phase is what costs the DSP a lot of money and costs you money because if they need time and energy for it the post backs have to come in at a certain threshold for it to be statistically significant right to draw coverts from but if you

already have that data from facebook and you can upload that into the DSP and say start from here I think that's less of a risk and I've just seen that as a great tactic that works without much of a response I think if you're discerning about who you test you might not test that in any way I don't actually put that what I do go into the test I go into it with this should be successful I want it to be successful right so I always work really closely with any network on how we can help each other be successful right if your traffic is in if you are really strong in the US on Android well let's test there let's not try and you know I mean it's and and and so finds what it does cost a lot of money right so you have to go into it with a strong sense that you can work together and and it's gonna work and and work to the network's traits as well as yours right don't try and shoot wanting to be something they're not yeah I mean I love where this is going this is kind of a breakthrough conversation in the sense that you know thinking about those hard questions to ask right we I will emphasize again I think from the programmatic space it's still nascent its new we're doing a lot of education in the market but we'd love to have you uncover those things with us

which means like are we stronger in Android or iOS what can we do together be successful I think it comes from a fundamental understanding that we believe programmatic will work one day right and that how can we lay some of the fears that you know kind of has has maybe been there but asking those questions like where where can we start to be most successful I have those questions a lot I will segue though what you said about you know the weekly meetings and the you know the big sigh of I have to have a weekly meeting because we need to keep track of how it's doing i right before I joined our key last year I was at a facebook agency I get it right I know ads manager it's like plug and play Ayoo off I go and but so we want to work together and trying to figure out how you know P working to consider a test with someone that's outside of those networks what the next question actually what kind of things are you really hands-on would you prefer to be hands off would you love it if you know that space operated very

autonomously for you I'm for us at least we have great success with Google body took the leap of faith in the beginning we are in the position to take that leap of faith without crushing the company and it worked and when it works it's pretty pretty good but just it's finding the reliability there that whenever you stop something new it's gonna do the same thing it's it's really hard to be confident in that but when it works of course yeah it just does all the work for you that you would be doing in Facebook hands-on so and we have like there it worked really well that we had a contact that we synced up with and then they were real helpful and sort of heard our pain and understood if something went wrong we could sort of trust the fact that they will meet us in

the middle if something goes wrong nicest important I said at the start I think the role of the you a manager will kind of change right so go from picking out all of these targeting things yada yada to essentially this is my target and I do the targets correct this is my budget this is how much want to spend and this is my creative input right and if in that world as you amend you're actually you're relatively hands-off right your input is creative and you might use technology to do that you you can do many different things right I think an ideal world that would be great but we're not there with the vast majority of networks yet and maybe they'll get there over time right so at the moment today if I eyes still gonna have to be in the data I'm gonna have you know my team will be and will have to be is this working is it's not working blacklist whitelist whatever that may be but I think my eventual God would be great here's a budget who's creative I trust you right but there's a long way to go to build yeah and maybe a look at a dashboard if we know then and then the who was that

you imagine moving budget rather to be most working what is right rather than being individually yeah ID like place to be and I think you know we all aspire to be there that's I think in everyone's mind you mentioned something briefly we didn't talk about before but making sure that your target is right you just said that like you know one day we picked you will have a target that's a interesting point because I run across folks that have a gamut of understanding of their monetization right and so therefore what their target is any any thoughts on that in terms of you know early-stage game that maybe not be as saturated as another do you feel like that there's a spec you know here's my target today but here with my target out here in the future I think you should always go for is deeper target is possible where you are confident the results will be correct right and so what I mean by that is when you put a game with two years of data you have a really good out of you code you know what it is when you don't

have that you've got a brand new game you've got nothing to base it on this so it could be a continue games your company it could be your first game right you need to go okay so what so what can I look at what can I look at quickly and how do I maximize that metric whatever that metric may be where there is potential revenue add served or whatever that may be and I think you just always have to be buying against the kind of the most complex right and don't be afraid to review it constantly we review our so I mean we use machine learning and our LTV additions get better every day right we've got a lot of data but you know we've got games and software we don't have that data right so we'll start buying against cost per whatever may be cost per day seven retained user for argument's sake right whatever that may be because that's where we're confident we have lots of data which we can get an early when you start to weed things in a night but then over time that will change to payers and that kind of thing but it depends on the goal of your game but I think yeah just go for his deeper targeting can wait within timeframe reasons I was just gonna say I think that's it

can be quite hard to know who to target a lot of games companies once I vote out they don't know really actually who their target audience is they can't specifically define it and it is quite varied and it's not one kind of person who plays that it's not like a 16 year old boy who plays it this can be quite large range and it's understanding those those guys and they can be worth different values as well and these days with the look like audiences that you do bidding on you you don't really even need to know who they are you can just take a look likes a few payers and then and then you like this is similar to what we know pay already in our game and with them target those it's that's why Facebook and some of the other platforms are kind of had it because able to target those guys while look-alikes haven't been very prevalent I know now use tarika says that it's now a DSPs are able to do that which is the way it has to go because when you have that broad targeting if you're just targeting a hyper casual games maybe they have their users that could that could purchase in

in your game but you go for such a broad chunk of users that you don't know what's happening there so targeting is a tricky one because I think love games developers don't often know who they should be targeting they just want everyone to play their game and this is this is also where I feel like the marketers or the UA managers have to really be very close with the product team with the developers and the designers of the monetization of the game you can't you know be in separate rooms trying to achieve your own goals you really need to have this discussion around okay what are the product updates so you mentioned the soft launch I feel like it's a crucial element of to have a saw soft-launched to really explore what what my game can achieve within a small environment test environment we've talked about how Facebook is that it's a safe environment where you can control your budget the low scale with good results creative

testing etc and always keep the product teams involved because they will be the ones who feedback as well on on things like user journey why is day seven retention not as strong as the other game why is it sort of little compared to the benchmarks we set and then really go step by step you have to do your homework and only if you do that you can go to their networks and inform them about what you want to achieve yeah that was actually I think we are are we close on time is there someone in the room oh we're out of time can we get 60 seconds sir okay so the last two questions we were we didn't have was how do you interact with your VI team's product teams of etc so thanks for touching on that and then you know how do you organize your buying team some of you have a single title some have multiple titles any thoughts really quick on you know do you organize your and orientate your team based do everything social search and networks etc or you know so super quickly so I oversaw changes inga from when we went

from owning products to to networks right so we went through that change so originally we had I ran the natural motion games and then some guys in San Fran magazine became zone we became one team and we switched to running networks and the reason we did that was was kind of to your point so being close to a product is a really good thing right and if you own that products then that's that's great but actually we found it on the u-ace like we lost out a little bit right because in every know the contact and it was just getting a bit messy and what I find is you're at sat in the office with the product - right so we what we do network is we are responsible for channels but we always have a product champion right so it's kind of still the same thing and that person is responsible for the strategy of the game and updating the product team every day and stuff like that but from a network point of view they know I can speak to Matt or speak to a colleague eggs that's how we do it it's exactly we operate as well we have we have you a manager's dedicated to a certain origin element top-level view for specific networks because those networks become

also very sophisticated and require a lot of you know detail expertise so we try to have the social channels specialist for the UAC or search it's definitely out there ones just super quick to add to that I join it's quite good to rotate people around so people pick up different skills right and that's a really good thing so we'll have someone who runs Facebook for X period of time and then maybe they'll swap to Google and then that especially if they're juniors staff because it helps build up their and that's something I've always found that's a smart idea it shouldn't be overlooked because buying facebook UAC and programmatic frankly if you look at the kind of the three it has there its own elements to it and teaching someone Facebook is not an easy thing right I mean that isn't undertaking so I like the fact that you're rotating them around did we have time for questions no all right thanks you guys appreciate it thanks for joining hope it was good for you [Music] [Applause] [Music]