Session: From Chaos to Clarity: The Journey to Building a Money-Saving BI Tool - Kamil Kufel.

[Music] from chaos to Clarity the journey to building a moneys saving Biol with camel thank you [Music] [Applause] it's probably me okay uh so do you hear me properly perfect uh thanks so much for coming to this talk uh I know it's pretty late uh all the parties will start soon so I will try to make it quick um the deck the deck is pretty interesting because I will show you how we made made a path to the tools that we are using now at boombit How I build them how I started building them a few years ago and how

they involved uh in the solution we have now allowing us to sell that platform to also our uh competitors to the other companies uh let's begin so to present the problem uh we um as a people working with data very often have to struggle with multiple sources of the same metric like Revenue so uh we can see that there is some Revenue in MMP like up R singular um also some revenue is passed by pob to the networks you see some Revenue if mediation partners and also slightly different Revenue in N networks directly there is also some Revenue in analytics dashboards like Google analytics Firebase um there are some data in the market intelligence tools of course uh and finally you see different Revenue in app stores Google plays uh and the bank accounts that you receive money on and there are also web shops

now additional source of Revenue that is very rarely properly implemented with tracking okay so why there reasons for discrepancies very quickly time zones not calculating net data um different methods of calculating cohorts like rolling or 24 hours or daily uh there are subscription renewals that were often AR missed uh refunds that are not tracked at all uh human errors QA plenty of them so how to calculate Roi uh we are losing money a lot of money because of stupid mistakes like we had a lot of profitable campaigns that were not stopped in time and also good campaigns that were killed too early because we didn't eager to wait for they to actually earn um obviously when have people working or something from time to time there are bucks so we made mistakes in bits and budgets as everyone uh obviously they can be bugs in updates they can be uh not working creatives

they can be different performance plan country that you didn't notice and didn't change uh or just wrong postar configuration okay so we had some requirements requirements to build something internally uh we wanted the data to be always ready in the morning uh you have some solution currently probably uh so ask yourself when you have the data ready the complete data for yesterday that you don't have to like wait if few hours in the afternoon uh for the complete revenue for instance uh we wanted to add re Revenue data aligned with what is on our bank accounts so calculating all the net revenue excluding taxes store Cuts commissions of our networks of multiple times uh to be as accurate as possible um sometimes third party providers have issues like there was a wrong CPM in waterfall or there the service was not responding and in the tools available in

the market you are not able to uh schedule a back fill Simple Thing pulling the data once again for the specific period of time we wanted to customize create very customized alerts on slack uh our company is purely based on slack we don't do emails at all everything is going for slack which is amazing uh but also we wanted that data on slack as well not as a links to dashboards or just PDFs exported that are not very readable but alerts proper alerts uh it must work well on Mac and iPhone uh we have some managers checking the phone performance on the phone and we wanted to provide one source source of true and we found that the unfortunately dashboards available the market didn't offer them um so we created something by ourselves this is the dashboard I created four years ago uh in 220 at the at the end uh it's not very beautiful it's gray but it

already sols the idea I had at the time combining all campaigns and the single view with no um weird paddings margins a lot of wide screen um scrolling freezing browser those are the problems that many dashboards currently have because they are not well optimized uh for actual use normally in the dashboard you will see like five six roads on the MacBook 13 in and you would have to scroll uh this is not the whole screen but usually uh we aim to providing such big tables with a lot of data it's not it's very rough here but it's it will get better uh here in the first generation we also have this um first idea of the bit files so bulk update of bits in the networks uh so here you can see the screen of the page when we basically show the differences between what we

have set now what change we'll make and there was just a button to uh deploy all these changes to the network okay great uh a few months later we made a second versions We had some time to add some colors so this is when the color indicators show um have showed up giving very clear information TOA managers okay this is great campaign this is terrible campaign kill it uh scale it whatever uh and also we had a first colation of um more advanced filters because in many dashboards uh you have do a pretty many clicks to make a simple thing as filtering you have to click to open the filter menu make your selection sometimes confirm sometime confirm even more uh instead we find the most useful and uh quick uh filter to be just one click filters so here you can like easily iterate over all the networks all the important countries at

the time uh by just using One Click by clicking those Network icons or the uh country codes um and this is when we added the idea of drill so whenever we wanted to deep deeper uh check the performance per day you just have to click on the row and the whole detailed performance uh per day um showed up this is also where we added the very nice feature link to network with that campaign actually Act actually selected so here we have some campaign on plin I think uh so by clicking that button you are redirected to uplo with that Campa visible so you want we were able to make changes very easily without copying IDs pasting searching for campaign just removing Cal fields for making mistake um this is also when we added a few like uh uh other funny things like um when the data was reloading we just

displaying the this icon of the coffee so um in a few minutes the dashboard automatically changed the newly refreshed data that was the moment of the update third generation new framework um uh definitely more filters adding icons it's finally looks like something commercializable let's say um this is also when we start experimenting with a dark mode because uh personally I'm a developer everything on my phone and my MacBook is dark so I wanted to make my dashboard black as well so this was the first iteration of the black dashboard it was not bad and finally the current generation the the current one so um I will talk about the technical stuff in a moment how exactly the textt looks but here you can see the optimized filters the switch allowing you to uh

switch between add only revenue or add Plus in a purchase Revenue we have added more sections to the um to the menu we have added the adsets countries and sources split uh you can see the bids and budgets for the companes directly here on the dashboard you can see how much budget you are using currently uh you can see profits and everything is very configurable the the list of the columns the list of the games the list of the networks countries everything can be sent up uh and this is the generation introduced the custom dashboard so uh feature known from MBA tools uh but simplifies as other so uh here's our most popular timeline dashboard allowing you to quickly compare like 50 different kpis for like a few networks it's very U very common case for us we have like four games that are similar Style or the same product owner so we just choose

them from the list in the menu and we see all the important kpis on the timeline uh with the line for every game separately um and here um is addition other dashboard uh a cohorted one basically uh showing what you probably have in your systems the so the cohorts retention arpus Roy average revenue per paying user um as well very clean very fast the whole dashboards if you change filters like takes less than a seconds to load uh so that was great Improvement for us and uh what is also available since this Edition is the changing bits and budgets directly from the dashboard so we are no longer forced to uh be redirected to the network use this network button which is still there behind the drawer uh but we can change bits uh Change Daily budget change status change name of the adset cine or or or site uh directly from here change

the budget and bits history as well um all that combined in the one screen without leaving up the dash uh and we added a tons of built-in dashboard that are always aligned uh with the current performance working for all games one of these is store health check so very simple Dash comparing revenue from stores with Revenue in your MMP uh um so for example on this case when I beautifully mask the game name to game one we have like 1.2% uh difference uh on uh Android and 2% difference on iOS and if you do if you handle the purchase tracking on mobile you know that anything uh less than 5% is amazing actually because all all that struggles with tracking um and you made it mobile friendly finally

in that generation uh as you can see everything can be easily scrolled on your phone without freezing up the browser without crashing it uh you can try opening Google ads dashboard on your phone um and then moving to alerts uh some of the alerts we wanted to create on slack because as I said before we are addicted to slack uh an alert that uh is basically saying that the particular creative uh Roi decreased by 10% or uh at aru decreased by 71 and here you have the hourly uh report of the profit for the particular game so basically our product owners uh receive every hour the detailed report about performance of their games to their slack um containing uh the most important metric which is in our case the Revenue uh spent and uh profit so if

there is an problem with the game or with the UA usually product owners know is this really fast and can pink uh other people um assuming they didn't s that alert already uh obviously we do a lot of creative tests uh as a publisher uh so all the results of those creative tests are also coming to slack so some thumbnail CDR ipms and most importantly very clear indication that is it good or bad number so the green broken hearts or yellow uh everything is crazy customizable oh and that one uh as I said we wanted to minimalize uh mistakes made in the dashboards so Al like this can be sent if like bit in the particular country or asset is changed significantly so here Victoria changed bit by decreasing it by 53% which might be intentional or might not but uh we know

about it because the alert was sent Tex stack uh so it was initially built by me one person then the team uh increased to uh like five people uh however we kept this very recently very simple backend made in Python very modern um with the fast up for handling all the requests and salary for handling all the task because there was some orchestration needed because we pulling data for all those providers we have around 50 plus connectors uh counting only at revenue and spent and so that have to that have to be managed the front end is made on react uh my most favorite uh front end uh uh framework um with the and uh design and typescript and finally how it is deployed uh we made some integration with uh bit bucket and uh pipelines so

basically every comit change in the system um made by my Engineers after code review uh and uh automatic test testing I got automatically deployed um by a simple thing as pressing the button uh and everything is currently hosted on Google Cloud run which allows us for scaling again um basically automatically so uh some information about transparency so how things changed uh previously we needed days to notice that we made a mistake uh or just performance changed now it takes hours or NE minutes if you reacted to alert uh we struggled with our MMP that they don't sometimes don't didn't have the connector for particular Network we started working with and now it's like just a few days or one even one day to add an API to to to our dashboard so

that is definitely a progression uh now we are able to create an custom dashboard matrics the most important metric for us is this net revenue so uh still none of the mmps provide you the net revenue uh you have to make that calculations on your own and um for us is absolutely crucial the cost of the queries uh you obviously have to store data somewhere we used big query um so um in the past when we are using different bi made on ler or data Studio we were not in full control how queries are built will are they optimized enough uh so that was pretty expensive by building our own query engine uh we decreased us a few times uh in up Revenue discrepancy decreased to 1% in most of the games uh which I showed you uh on the previous screen and number of slack alerts we had zero because we hated those PDFs and screenshots uh and now we have 300 um at

least 300 alerts sent to channels every day and that doesn't include the uh the alerts sent every hour because they are automatic reports uh and data is now available in the 8 a.m. UTC plus Z and quality of life of our employees uh increased significantly um so uh wrapping things up uh for our partners we offer full transparency if you publish the game with us you get access to the same dashboard as do including our bi um of course uh all the features that we create for boom bit that we proud of like custom dashboards or soon we will be launching the creative automation uh I mean replacing the creatives in the in the networks uh based on the performance uh is also shared with the partners as soon as it tested um internally uh and data security as we again want it to be slightly different than the tools available in the market we create a separated Cloud instance separate uh dashboard with the dedicated

domain uh URL uh just for for every client separately to keep things safe um and that would be all uh I hope you like it um if you have any questions I would love to answer them we have like one minute um thank [Applause] you thank you thank you yeah great presentation thanks uh one question you mentioned that there is a lot of uh sources of revenue uh which do you use on your dashboards then and if I can also add another question uh about the alerts you mentioned that there are 300 or more alerts per day isn't that uh like or in because I'm also doing like an alert system and what

I experienced it that a lot of alerts can be overwhelmed and ignor then thank you um yeah so when comes to source of the revenue we use uh we decided to go with MMP but we apply additional validation on top and we recalculate the values of the to the net so we made sure the uh Revenue in m is correct because this is the revenue that is sent as post back to the networks and then um again apply net and show it in dash and it comes to alert uh we not sending all the alerts to all the people uh so usually uh alerts are personalized so only the user user acquisition manager receives the alert for the game he is owning and his manager uh and for product owners again um we are not aiming to send all the alerts to everyone that would be crazy we we tested some tools that offer to like huge number of alerts and that

didn't work well so absolutely okay if You' like to learn more please come tomorrow by our Boo or catch me at one of the parties today thanks thank you very much