Level Up Your Jenkins Reporting

[Music] my name is adrian yu and i i do have some credibility behind my name i've worked in the automation industry for 12 years 11 of it has been in the gaming industry so um from that standpoint um i guess i'll actually go back one slide i do i guess i assume everyone here knows the benefits of continuous integration and you guys are here for jenkins because jenkins is in the title um i start off as a software engineer and test so i focus on automation scripts and then eventually my team took on the builds at ea now i'm currently associate technical director for the development and release engineering team so what is theory my team is a community as a distributed community of embedded devops engineers focused on continuous integration and deployment automation we're kind of a unique team within ea we serve a lot of the game teams

and we're effectively this uh the build engineers of ea now what do we do well our name is kind of not not that obvious but what we do is we set up automation for your usual continuous builds um for those that are smaller startups and stuff like that the setup i'd recommend for new automation would be from left to right uh just because that's easier so canadians build static analysis then once you have those running you set up your build verification tests uh then eventually you you add metrics performance metrics build metrics uh then you start focusing uh on plug and crash reporting as your game gets more advanced and more mature then you write miscellaneous tools if you need to to help with automating uh things throughout the process and eventually you start doing custom reporting because your system's stable the game is stable your ci system is good and then at the end of the day our team owns uh the nbn ci so we're able to

improve the productivity of development teams by taking on that task that everyone here probably understands is kind of maybe a little tedious at times but by having a dedicated engineer which is not always the case you're able to identify optimizations throughout the pipeline and if you're lucky enough to do multiple iterations of a product or releases you get better and better and you start thinking of ideas by staying focused not as a side job but as a dedicated job now as far as i'd like to provide a little context on the team our team first started in vancouver um ea sports i've been in vancouver for 11 years uh eventually we did all the builds for all the sports games and then over time the past 10 years um everyone kind of heard of us within the company and they started reaching out to our team and then we eventually started um i guess adopting and helping out other studios uh so it's kind of a unique situation we searched we first started off as a team of 15 in vancouver uh we're now over 100 worldwide uh just

within the company so it's kind of a unique situation um it's kind of a double-edged sword with this kind of expansion that we're lucky to have by this we were able to take over and and inherit existing automation systems in different studios as you know ea acquired them or at least they started reaching out to us through this my team experienced you know rapid growth and some of the pain points which was the technical debt that came with it so a lot of these studios had custom ci solutions built around their game and their workflows this is good for them but also bad for sharing amongst all the all the different teams so one of our strategies of our team was to actually adopt open source to help tackle this and help remove some of that technical debt so that's why i'm up here i'm i'm trying to preach that adopting open source is okay for your ci system it's probably the same thing to do especially if you have to grow your team rapidly i don't know about you but games

industry we make games we're not necessarily tech companies so writing our own ci system the people on my team made that mistake in the past and they we kind of we were finally able to deprecate that and now the majority of our companies running jenkins um to give you a little scale we have 200 plus jenkins masters uh over 2 000 build notes uh so that's a lot uh a lot to handle um i'll go over ways how we can handle that you know throughout this presentation um you know if i were to ask any of you what the most important trait of automation is most of you probably say speed reliability or flexibility you're not wrong but you know i think it's everything combined good reporting is a combination of reliability speed and flexibility if you don't have good reporting nobody knows any of that because they don't get the results they don't understand results they don't see it so you definitely focus on that once you have good reporting you can use

that to help manage expectations so for those of you that are administering the build system you'll probably have to defend the system the reliability and you know as new people come on the team they're always questioning that's the automation error right and that gets really tedious to answer time and time and again so you want to have good rapport and help manage those expectations once you have that you can now focus on scaling your automation responsibly because you can now see the stability of the system people trust the results and you spend less time talking about it and defending it and everyone just sees the value of it so um since this is gdc i decided to apply some reporting levels to what i'll be presenting a little later so i kind of like applied the world of warcraft leveling system to it so when you have basic dashboards i kind of call it level 60. that was you know when the game first came out for those that weren't around then and then as you add log parsing and a little bit of i guess if you want to do machine learning and

stuff that you start to get a little bit more intelligent reporting and that's when you really get up to level 85 and then ultimately when you start getting scalable reporting solutions integrating bi tools or oi tools i'll start going over that a little bit later so first off to start off with good reporting i'm going to cover four different areas one is log levels another one is the default reporting features and then ways to diversify the views that you have and then also incorporating oi into your stack so first off to start with introducing log levels you need to introduce log levels into your system because if you don't your output's going to be really really bad and it's going to be really hard to read it just adds to the it just compounds the noise problem of your system and any subsequent reporting or things that you want to do with it

afterwards so some of the things that you want to do is introduce the log level so for things like log4net or or gradle or maven all those things have ways to toggle the log levels so i definitely look into that for things like unity i'm a little less uh municipal for me so perhaps there's ways to do that hopefully there are if you do manage to reduce log noise you can increase the performance of the system and you can ultimately have longer history retention which everyone seems to love because when you have automation you run a lot of it everyone wants to go back in history to make sure things are running properly now that does have drawbacks the drawbacks include you know you're increasing the complexity of your system by adding different logs and usually or sometimes the rare case is actually you don't have enough information um and then you're forced to actually incorporate toggles into your automation to then rerun it and output more information so that you can rerun it effectively

the second one of the things i recommend is leverage the default reporting features for this talk jenkins you should you should leverage it so i'm going to start the lowest level so when i talk about configs i think i use the unity naming conventions here but you know you master release debug different code compile configurations you want to have separate jobs for each one actually and i'll go with that a little bit more then you add certain you start incorporating regions for those of you that have multiple regions or skus you want to add another layer on top of that and then have those kind of chain the jobs underneath and then you know if you support multi-platform well this is an example but you then start adding projects or schedulers for that within jenkins so these jobs are nothing but shell jobs that trigger things and help summarize the results and then at the end of the day if you're doing multiple branches or streams or whatever source control system you use uh you add one level for that and then this kind of gives you the layer architecture uh for reporting that

you'll be able to leverage later on it also gives you vertical silos too so if you want to look at the specific configurations you still have the freedom to do so this is important to keep your automation levels modular so that you can leverage it later on now once you have that i'm gonna i break down views into two different types one will be the dashboards another one will be direct feedback um once you've applied that layer concept you can start diversifying your views into these sections um because at the end of the day what a qa manager wants to see what an engineer wants to see or a producer they're all going to be different and in the past my team tried to design an automation system that satisfied everyone at the same time and that was horrible some people are happy there's always unhappy people so you want to break that out you and for us we actually broke out our dashboards into primarily two points one for job overview so you look at all that job reporting at a massive

scale on some on a dashboard of some sort and then second of all you want to add health monitoring to your system as your system grows you want to make sure that it's it's in a good shape and that it'll continue to run now for direct feedback for the engineers for the content creators for all that stuff we basically set apart uh three different ways uh primary ways slack is one of them uh setting up dedicated slack channels uh is useful i'll go more into later then email of course everyone's used to email people hate the spam some but i'll go into ways where you can control some of that later and then incorporating log parsers so that you don't have to manually debug automation failures as much so first off going into immediately an example open source plug-in called the extra columns plug-in i highly recommend it we use it for it's useful for build engineers in this case i've obscured some of the information but the job information is there so the interesting columns that you can add immediately are

the console output plug-in or the the icon actually so instead of doing a traditional click you just click there and jump straight to the console output saves you extra clicks then you have this neat little field that you can also expose and actually dynamically populate with your groovy and in this case we use it to highlight a link to a network share with you know published builds or published artifacts and then one that's really useful for build administrators is the slave allocations so for those of you that do incremental builds and you want to hard allocate or dedicate machines to specific jobs this is really useful because you know immediately which job is run on which machine if there's a problem so it's really useful for administrator now the second one that is also open source is the build monitor plugin so this is a a lot of information but it's good so we use these actually on tvs all around the studio and people's desks they can they can cycle through it i use it with a combination of a chrome

revolver plug-in or extension chrome extension to revolve around different tabs and for each tab this is actually what reporting of automation for one particular stream so if you have a lot of streams you set up one for each one uh the text is highly configurable in this case you can see the job names are really bold you can change the text size um the bright green is actually showing a build running this is of course a screenshot and the bottom you'll notice that there's actually failures and it shows up red now my team has modified this slightly but we haven't committed the changes back to open source so apologies so far but we incorporate the ability to uh interpret uh the log parsing results to change the color for automation failure to be yellow which isn't shown here unfortunately but the bottom of the red if you can't see it actually shows that there's n-unit test failures and that actually gives you a reason um it also shows you who committed to that build so you can actually shame people if you want to so that's so you don't have to be a build cop system can do it for you

next moving on jenkins master health so monitoring jenkins master health no matter how many masters you have is very very important and especially when you're adopting different plugins we actually use four different methods um so i'm just going over the stack so for jenkins the application we use the monitoring plug-in which is open source we also use the splunk plug-in i'll go into some of these uh or actually all these more detailed later on then at a different level on the operating operating system level we use zabbix you can use anything else you'd like there and then for the hypervisor level we use vmware so as you can see we have multiple layers of monitoring so in case the jenkins application becomes unresponsive we can go down to the next lowest level we have many different levels to go now the first one the monitoring plug-in this is actually my favorite screenshot it's a section of it through this you can actually see the ram usage and the system usage the cpu

usage there's 22 or more charts that you can actually look i won't show you all the screenshots to save time but it's really powerful it takes all of like five seconds to install and configure there's no extra work that you need to do but it gives you a heap of information just for free really it uses the java melody on the background so or the underneath you can actually trigger your heap dump so in case of all you guys installing different jenkins plugins and you're having problems running up memory you can do heap dumps and trigger it and then analyze it after the fact if you'd like we made a lot of mistakes in the past so that was actually really useful as we started to adopt jenkins three four years ago it helped us tremendously now at the os level monitoring uh this is just some screenshots of the zabic's alerts that come up so all these alerts are like they're reactive so they're just responding to actual events that happen uh they're not ideal but they're a good place to start and then of course you can also get performance metrics in this case it's a

cpu utilization so if the jenkins monitoring plug-in which is rendered through jenkins is unavailable you can fall back to something like zabic's and the history and actually go through things that way to identify any performance impact now at the hypervisor level uh we're able we're lucky enough to use vmware so we're able to actually go and use the vmware monitoring or the performance monitoring in the vm and vcenter or vsphere this is a screen grab of some of the information now this information is really granular which is good if you want really quick feedback but each of the systems show has different we configure different log or history retention so the level of detail gets is smaller it's a lot smaller than this one so you can get real-time really good feedback from this one at the hypervisor level monitoring you can actually look at individual cpu usage as well for those that aren't aware whereas zabex and jenkins mod and the jenkins monitoring plugin will look at the os level so it's aggregated cpu usage so if you want more information

you can get it through the hypervisor now moving on to the direct i guess the direct notifications specifically slack plugin this is of course open source common theme so this is a use case i think some of the talks uh or there went about this for the live ops and stuff like that but if you hook up specific deploy jobs and jenkins and stuff like that you can post a slack channel and then people the live ops team or qa or whatnot can actually monitor it so instead of having email all the time you can use that and actually disable your emails for that you can have both it's all configurable right so i'd highly recommend it for those that use slack to actually start doing this and then that way you can start shrinking some of the emails noise that comes into your inbox creating multiple channels is also recommended now for the job summary plugin this is the one of the few plugins we've actually written this is actually this is actually derived from the email extension plugin which i'll go over

shortly so in this one it's a lot of information but it's good information a majority of our jenkins don't actually use pipeline yet so we were using build flow and with build flow we used a stepper we were migrating from different ci systems in the past and we liked the concept of steps or stages before pipeline existed we already had it the middle section i think well actually it has a link so it's an optional view so the installer plugin gives you optional view in jenkins and then this middle section here um actually goes over uh details all the steps underneath so i showed you the layer architecture before this is kind of grouping and distilling that in one place and you can immediately see via the log parser which i'll go over later you can it highlights the failure and shows you what the failure is in that view and then it also shows you the detected output or the errors so and that shows you an icon on the side as well so we started playing not really a blame game but made it more

uh obvious that it was a gamer so we put a joystick so this game team compiler or something like that it appear right away so people know if they're looking natively at jenkins that it's uh it's a it's a it's a specific team's fault really now the log parser we use we chose to use the build failure analyzer plugin there's a lot of different plugins out there that you can leverage that's open source that you can use oh sorry let me go back this one's actually a popular one with us because of our scale it was one few plugins that allowed us to uh use a gui to actually edit it and actually you can actually connect it to a mongodb on the back end and then therefore when you're any jenkins master you can anybody can add a failure cause regex based um it would then post the essential mongodb which we have multiple instances of in terms of uh to track the information we did some modifications to this plugin to make it a little bit more optimized for our scale because we ran into scaling issues it was probably the only plug-in that ever caused us

problems on jenkins uh we're running like 64 gigs of ram on some of our jenkins and when you're log parsing and you have really ugly logs or a lot of it it will chew up through a lot of ram so that's just a lesson for everyone to be aware of but this is really useful in sharing it and it's probably the biggest game changer of our plugin for jenkins or any log parser really because they'll we went from debugging um things manually to just using this and then just adding further causes and then basically just manually tweaking it and training it with different regexes and we're able to share this across the entire organization now the email extension plugin which i mentioned briefly if you guys ever were lucky enough i say lucky with most sarcastic terms this is the default failure email that you get with jenkins it's all text based it has links and i did a hello world and i assume if you have a lot of output it'd email you the entire contents of the output which would be horrible for

some very large games and so and then subsequently the fit the when the fill or the failed build succeeds you get this really really useful email with a link um so it's not really ideal so what we actually did was uh in the email extension plugin you can configure much better notifications for different culprits and stuff like that we we configured a lot of different rules um so i recommend that and then i didn't mention before about the the job summary plug-in but we actually use a groovy script to generate the content so the exact same groovy script is used to generate the email content so it looks exactly the same well a few caveats but at the same time one would ask if you send emails to different culprits and stuff like that do you have customized emails yes we do because it's groovy base you can then just condition it to send out different chunks different people and different audiences this is just one example

now and then follow that following that the failure email is also just just as useful or i guess it's improved it actually shows you the change list that went into the build that made it pass again so you can actually verify uh without having to click anywhere else which saves everyone time and of course this is only sent to the people that contributed to the build so it's not gonna add it's not gonna be noisy for everyone and then yes the changeless i've hidden the names for you know uh to protect people um and then the part i get really excited about is the operational intelligence portion of this talk um and the pros and cons of it you know uh the scalability of do of it so the way we're able to report over 200 plus jenkins masters or at least a report across them we had to incorporate something that's better we didn't write anything ourselves i'll show you a little bit later but it enables you to debug the issues faster because now you can search across multiple masters all your job logs you're not stuck opening tabs and tabs

and tabs and scrolling you know that's really bad i did in the past i learned from it it gives you greater extensibility because you're now it gives you options to decouple the reporting stack from your actual automation system so that you don't have to have people looking at jenkins if they don't have to it enables you to do metrics driven decisions in your ci system it also definitely helps manage expectations which ultimately gives everyone administering jenkins a lot more time back when you don't have to have those those lengthy conversations with every person that doubts it and then the cons of course are you uh similar to the other ones you have increase complexity and increase maintenance because now you have another stack to maintain but i think the pro is far outweigh the cons in this situation and i guess for those that aren't that don't know what operational intelligence is it's basically tools that help make sense of large sets of data it aggregates log information and makes it searchable by indexing

yeah and then i'll go into how the reporting stack kind of changes because of it as you incorporate it so you can see the familiar xavix and vmware layers um i went over the build failure analyzer plug-in and then it connects to a mongod database the log sash plug-in which is also open source we use that to actually inject information from jenkins into elasticsearch cluster that's uh that's on-prem for us and then i mentioned earlier the splunk plug-in as well so the splunk plug-in i'll go into more detail later on elastic is free it's free to use in open source they have enterprise options just like jenkins does as well so we use it primarily because it's free the splunk we have a limited license so we use that as much as we can i'll go into reasons why later but we don't set on one i think that's a key thing is never to really set on one if you don't have to they have different benefits for each one now while incorporating oi tools

makes things easier it people often understand underestimate the effort to generate the reports afterwards right so it's very similar to safe programming if you don't sort your data you're going to have to load problems writing things that handle the data afterwards so the layer architecture all that stuff i went before you need to add something to it you need to actually define job naming conventions across all your jenkins masters you know i kind of set on something like this so depending on the complexity of your studio we have different projects we call them titles you have different studios your branches platforms automation type in this case is going to be like a test or a build or deploy and then you have regions and configs when you do that some of it can seem very tedious but it'll all make sense when you actually generate the reports later on in your systems you definitely need to define an infrastructure naming convention if you don't already have one because you need to know where things are running if you have multiple studios around the world

then you don't name it separately it just makes even you report on it it would make no sense and then apply the layer concept i mentioned earlier so i gave some examples in terms of some of the names we have mainline so ml all skew trigger so this is a it's just a trigger project it has no real logic underneath other than chaining it to a bunch of other jobs um you can do this a lot easier in pipeline now but we we did in build flow we don't have to use built flow you can do a freestyle as well if you'd like but that's definitely very important when you do that you can then report at any one of those levels and that's really useful to create those different visualizations i mentioned earlier for all the different audiences now the log stash plug-in um as i mentioned earlier it goes into elastic the configuration is really easy it's about five lines as long as you have a cluster set up that's basically it uh using and then there's another step afterwards you can configure you have to add a post build step to each thing there are scripts available

online to just mass add it to the jenkins script console so it's really easy if you want this mass log every single build log of all your jenkins jobs on under master you can do that and then this is the post build step you can specify how many lines so you don't have to do all of them you can do a you know you can do a tail so in this case it's i think 4 000 lines which is usually way more than enough and then uh this example is for all those people in this room that's encountered random compile errors and automation system um i don't know about you but if you had to debug that and track that and tally that up manually that would be really hard and really no one would want to do it in this case we used we shoved everything in elasticsearch and one of our engineers created a graph so the top graphs denote the frequency of the errors that happen off this particular search string um and then the bottom the bottom charts kind of give you a visualization how widespread the problem is

so once you know how to you know how many occurrences are happening from their top charts you can kind of focus it a little bit more so the in the inner pipe uh is graphing how many jenkins masters this problem is is having is is or it appears on so in this case i think it was like five from this graph and then the other one this secondary layer pie chart denotes what kind of jenkins jobs it's been seeing this failure so not only do you have the master you have the jobs that they're occurring on and the frequency and then the last one is the specific steps i think so oh no sorry the nodes the build nodes so you can actually find out all that information just from this just by shoving your information in and then doing a search uh in this case uh the random compiler error was caused by antivirus locking files on the on the build machines uh so i'm sure everyone's sort of encountered that it's painful well in this case you're able to use metrics and reports to actually prove that it's a problem to people that control

control it's able to help identify which build nodes have maybe corrupt installations of antivirus that you have to go take out or you have to go you have to go and mitigate so it it makes it all possible without the oi 2 or elastic search i don't know if anyone would be able to get this manually or at least convince someone that is a problem now moving on to the splunk plugin i get really excited about this one because i use this one a lot the splunk team i believe actually uses jenkins to build their product so they actually their team if i remember correctly actually authored uh the plugin for jenkins itself and as well as the app that visualizes uh the data from it so in this case the configuration of it is almost very similar to logsdash except this one happens at a global level so once you configure it all your jobs send information in it actually has a tighter integration uh to jenkins so it's not just log information i'll go over that a little bit uh shortly um

in the chase if they you so you installed that jenkins splunk plugin and it shoves information into splunk and then you install the app to visualize that data in your splunk instance and then you get this so this is a build overview of multiple jenkins masters this is the screenshot's taken from the the plugin page so if you want to go see it you can see it afterwards i can't show you our stuff but it's basically the same thing but you're able to visualize jobs across multiple jenkins masters it would work for one and here's another example of a pre-built chart that they provide and this one is the jenkins health so it's actually taking jvm information and then graphing it in splunk as well so this is your fourth method outside the other ones i mentioned so there's a lot of redundancy in the system but it's cool that way it actually has the master log as well so this case is really short example but you can actually search for jenkins master log without having to ssh into it if you have multiple jenkins masters well you can search across all of them

so it's really cool that way uh saves a lot of time and then lastly it's kind of cool but they actually built a way to interpret uh unit test results as well so if you have standard jnu unit or stuff like that to plug in it'll actually be able to consume the test report and graph it for you as well with then all of this was just install the plug and configure it and install the splunk app to visualize it and it's all there this is zero customization from my side um really powerful stuff and if you haven't noticed the bottom left that was kind of like as i go along you're kind of leveling up your reporting so that's a little thing i did there now once you have that once you have those uh that data hooked up to your oi systems i i this is where managing expectations becomes really easy because we're able to aggregate all the job run information from your jenkins all across so instead of just having your trends view you actually can just do all your jobs at once now yellow is bad in this case um it's an automation

error classification um i give you the legend now so it makes a lot more sense orange is bad data so that just means the log parser wasn't able to determine a failure cause and then there's no script errors that appeared there's also very few border builds uh the game failures are denoted as you know compile errors linking errors stuff like that and then success is just to build past so immediately from here you can see the system is either super reliable or super unreliable so the big yellow part actually looks a lot worse than it actually is because if i hover over it in splunk it's actually just one so this was a weekend for uh for a really small pro like for the project so nothing ran only one thing ran and it failed so it looks really bad but you're able to just hover over and tell right away that is no reason to panic now on the exact same data so this is reporting off the build failure analyzer data that is in mongodb that we ingested into splunk in this case this is the exact same legend same color scheme same legend this is really noisy

this is actually reliability on each project jenkins job so each vertical column is a jenkins job but right now you're able to tell which jobs are stable which ones aren't from the color coding and it's really powerful for the build as an administrator of jenkins and for my team to actually go and address problematic builds now this example there's full drill down capability so if you hover over something you give you a number of instances if you click into it you actually get to draw not the build logs but at least the the metadata that is uh collected by the build failure analyzer plug-in and you're able to determine uh exactly when it happened and other information as well this is really powerful that way you can of course deconstruct reconstruct the jenkins job link but because this system this data is in splunk and not in jenkins if your jenkins log history retention is really low you can configure you can have a lot more longer history in in splunk and the fact that splunk and elastic both compress the information keep a lot

more uh so it's really good now earlier you saw the graph about the overall project reliability of a bunch of automation jobs well you can break it down by stream so if your particular team or qa or producers or or lead engineers focus on a particular feature stream or something that you build a report for them and then they're able to consume that information so you can have multiple ones this is just these are all screenshots of specific reports you can have a dashboard with multiple ones so it makes a lot more sense but in this case this is an example this one actually has project names that you can actually see so that's really good for them uh subsequently the same set of data i can generate pie charts to visualize which problems are the most frequent uh in this case the big green is actually a linking i think a dependency error so it's not actually our fault uh you can filter it by different failure categories within our system so if you're a jenkins administrator we can look at specific failures for your game team you can look at other ones it's useful that way

once you have managed successfully managed expectations of everyone and people trust the system and they really like it they're never happy they want things to be faster all the time right so off the same set of data noticing a theme we're able to generate or calculate the mean max and min job times and give up and plot it now this plot is kind of ugly to look at because it's got all the jobs or a lot of jobs so i actually have filters in the top if you don't if you can't see that that you can filter by job names and stuff like that so you can filter it down across different jenkins masters this one i limited to just one because looking after everything would be would be a mistake you wouldn't be able to see anything um now that helps we also do the i guess we also calculate the standard deviation which is really useful for people that like tables and numbers because job times not just speed but people want consistency you know when qa has to plan for when when you start a build when do they have to have people in to test inevitably as much automation

automated tests you do you can still have manual qa come verify things uh we're able to track you know how consistent the build time is so you know if it differs two to three minutes or something like that that way you can get a better ballpark how long or when to schedule people to come in so it's really useful that way um subsequently you can also just pull a line chart now the line charts in the middle are kind of ugly because there's a weekend there's no data but you know many types of visualizations that were built you know just a fingertip from the same set of data all these graphs would probably take me maybe five ten minutes to create and then they're shareable across all the data a lot less time than talking to individuals i might add and now once you have that that was on the build data you can also incorporate other information not only jenkins but your performance metrics so if you want to ingest anything text based you just shove it into either elastic or in this case splunk and then you will plot in this case i believe it's the memory uh the memory high watermark so on a cl basis on the access

you're able to tell for a very stable game as ceos go in and out and through the automated tests the metrics you're collecting you're able to detect if when the performance impact has happened and people are able to address that a lot sooner than if they did manually so it keeps the performance high i don't know if it's clear but i mentioned jenkins console log so all the build output that the system creates can be indexed now you want to be careful with that i mentioned introducing log filters there's a lot of data but if you can imagine being able to search across all the jenkins job results in one interface without having to open different tabs there's a lot of power behind that and that's what really makes me giddy when i when i look at the system i'm able to use it for multiple reasons i mentioned it helps improve debug or helps reduce debug time for failures in this case i'm going to go over the example of the perforce errors so in this case i can't really read i'm

trying to read my screenshot but in one search of this simple search query i specify the index where the data is and then i specify the type of data or the source type and then i just type air force password i was able to identify 104 different perforce connection issues over three different jenkins masters over 15 jobs and a total of 43 unique job runs from that simple search and that kind of highlights i think it summarizes the kind of power that you would have in debugging debugging something at scale if you incorporate this type of tool now splunk does cost money but you can do this in elastic as well um a lot of these things i talk about there's a lot of stuff that's junk and specific but the concepts can be applied to anything really um and not just games too now the second example i'll go over is you know i actually use it to optimize java server builds with doing with knowing nothing about java and the server itself

so i i joined i looked i became involved with the team uh relatively recently and i had no idea how their system was all i knew was the server was java based and i knew it was maven so i was like okay great i have a start and i was like i actually used splunk to figure out things for me and help me on board me so in this case i want to optimize the server build and i'm going well i know maven i heard rumblings from other engineers that maven was downloading packages from the public maven repo constantly so like okay so we had an artifactory cache or artifactory server locally so okay well if we publicly or what if we just you know cache the stuff locally and have a local artifactory cache and like okay so which repos do we need so since i had all the job logs i was able to search across all of them i did a little thing called extracted fields in splunk which is able to interpret job logs and actually isolate and create fields unique fields and i created one that would give me all the unique urls that we were downloading

and then that kind of highlighted there that was kind of like the extracted field that i created it's actually called maven repo you know really aptly named and then i was able to construct a chart or actually not a chart i was yeah this is a chart in a line chart but i created a job where my engineers created a job that one built with the default settings nothing changed the second one was pointing at our local water factory cache and we're able to track that on average by just looking at local maven repo we were able to save one minute build time for our jobs and that was without looking at the code just using the job parsing and i thought that was really cool and then i think the maven repo went down a couple times i guess uh earlier so it was really useful that we were able to at least isolate the system from the public maven repo so that kept our build stability up high even higher now the other one is once you have that data in splunk offers a machine learning kit uh

and so does elastic as well i think you have to pay for the elastic one and splunk you have to pay for two but it just shows that it unlocks a lot of potential and it's kind of the next steps for my team will be to look at this stuff i don't have any examples i did i did uh we were starting to experiment with um looking at the log or the drive space so whenever you're syncing and the artifacts and try to predict uh do more predictive reporting rather than reactive reporting now once you have all that you can now probably scale your automation a lot better so my recommendations for that are going to be really loose and they're going to be a really high level but you know this is really important and if i spoke to a number of you before then i apologize for you know ranting again but you got to factor in the talent so in our industry uh ci is not the most glorious thing the companies that we work for may not value us as much as we think they should um

so you know you should factor in the town that you have that you can retain i don't know how many of us contractors how many of you guys are full time but there's not a lot within within ea it's definitely improving to do that you need to definitely minimize the technical debt within your stack and using leveraging open source or existing enterprise tools is definitely the way to go there's also transferable skills and then you will definitely want to apply configuration as code principles as well as infrastructure as code principles to help scale your automation responsibly i won't go into too much detail for those ones because i could i'd be up here forever but on our team i'll go over the high level stuff i mentioned we used the billflow plugin before that we don't we still use it in some of the leggy systems but we're migrating to the pipeline plug-in i know that i was there for the whole ride when they kind of deprecated build flow and they they hired all the people that made it into developing pipelines so that was kind of cool and bad at the same time

you can't is extensible and powerful the fact that you use the groovy domain specific language to do basically anything you want um it offers the stage view uh which is very similar to our custom ci tool that we built six seven years ago at ea it actually looks very similar to it which is kind of ironic and then offers optional blue ocean pipeline editor now this view is kind of cool it is i think there there's other ci systems reviews like this as well but they realize jenkins realized that you know the way that you view results is not the best and they started trying to improve it so that's a good good for them and good for us as well next moving on that isn't the only thing you need though because that only solves what when you actually run a job well what about job creation well we use the job dsl plugin so i'm not sure if any of you guys are familiar with that but it's really it enables denial or dynamic job creation and that's really good

um and once again it uses the groovy dsl as well so we're able to use the job dsl to populate our jenkins master jobs and then those jenkins master jobs when another pipeline freestall we use that afterwards so it kind of it's kind of a chicken the egg thing i know that there's many areas where you can do scripted or non-scripted in pipeline you can put your properties in both which is kind of confusing but you know it's up to you guys how you guys want to use it i don't think we've found the perfect mechanism within ea yet because there's a lot of people with a lot of opinions but it gives you a lot of options and that's that's all good for what we do this example is taken from the the jenkins wiki um it's an example of the job dsl plugin for those that aren't familiar with it it will you point out it at a git repo and it will generate a unique job for every branch you have in it and this is an example this is cool and if you guys have if you guys are using git you can probably use use this right off the bat

if you're using perforce i don't think they have something like this or maybe they do maybe my team does i don't know i'll have to check on that one but this is just an example of how powerful it can be um and it will auto scale your system now once you have the ability to generate a lot of jobs make them as complex imagine you know using code you're gonna you're gonna need things to run it on right so uh for us we're using in-house we're using vmware for our vm stack and then we use chef to help configure the software installations on it um and and that's i'd highly recommend doing that uh the chef is using a figure our zabx for our xavix configuration so it can do a lot for you it'll save you a lot of time um we do use jfrog artifactory we have we're not completely adopted it yet but we're starting to and that will help us with tracking our build our tracking our builds better than just using network shares and jenkins itself i would highly recommend you not using jenkins to store your repository because

it's just going to blow up and you're you'll regret it later uh because we do um and then next you know we're exploring with kubernetes um we're using kubernetes environments to build our server most of our server stuff right now uh for what can go on it and then of course docker containers um but uh there's one thing i'd like to mention is we are experimenting for like you know aaa like the ability to use containers for that but you know the jury's kind of out on that in terms of it will give us proper returns or not because it adds a lot of complexity to the system i'm not sure if the performance gains or the flexibility is worth it at the moment you know in summary your automations is only as good as your reporting you know the speed and flexibility and reliability isn't the only thing the earlier you invest in your reporting stack the the more you'll thank yourself later on uh you never know when you're going to go big you never know when your system's going to go wide so as long as you have that

flexible reporting stack and scalable it'll just help you out in the long run and also it will help reduce your technical debt by by doing that and leverage open source as much as possible or enterprise tools if you if your company has the money to do so um you know that's really really important and then of course once you have that i'm sure everyone here has had to deal with engineers you have to manage the build system use it to your advantage manage the expectations and then last but not least the i think i said everything else was born but this one's the most important factor in the talent so you know if you have a built a big team a small team a permanent team use those in in deciding what kind of tech stack you want you know what i said is one option of doing it it's our constraints and what we live with i understand that it's not perfect for everyone so definitely you want to factor in the talent that you have that you'll retain you know if you want to stay in the job industry and stuff like

that when you're making your decisions custom tools is good for certain cases but um for it depends on the situation right so all the all the links i had for all the plugins are on the pastebin link here as well as some of the technologies i've listed so if you didn't have time to jot that down you can just go to that link and then there should be links for that and that is it so if you have any questions i guess if you have questions please call the mic hi um you said you were monitoring memory uh and that's something we often have problems with it uh just tuning the java heap um running out memory uh are there any kind of tips you can give on keeping memory usage down don't use build flow

if you use build flow or anything like that it does class loading and the class loading will bump up all your memory it doesn't always clean it up i mentioned earlier we have 64 gigs of ram on some of our masters not all of them have that it depends on the scale number of jobs run concurrently it really depends on the complexity of your jobs and how much you use it so there's no master solution for it but the monitoring plug-in i definitely recommend you using and you can try to get some information from that if you use log parsing it's not the perfect thing it's a it's a double-edged sword it makes it really easy by using the build failure analyzer plug-in or other log parsers but you are on the jet you're using the jenkins master to do it i don't think i found the jenkins plug-in that will offload that to a node yet now that we're shoving more information to splunk and elastic we are thinking about ways of just not using log parsing altogether inside jenkins just using the oi tool itself to do it because it's powerful enough but we're not quite there yet okay thank you

hey thanks for the great talk that was very good oh good i found oh thank you and the question is that you said the jenkinson ci thing is not a glorious thing to do as an engineer so i just wondering what's the biggest motivation for you oh that's a that's a great question so i i say it's not the glorious thing but i think that was more for everyone else maybe not in this room um for us i i find passion in helping others i've been in this doing this for 11 years and i find value in doing it and i take pride in it and i take pride in having a stable system and for the other day helping the engineers enabling the focus the game engineers to focus on the game itself that brings a lot of value it doesn't mean i can't contribute i contribute ideas i don't have to do it so it's kind of the best of both worlds we also don't pull as much on our team we actually don't pull very crazy hours at all i think within ea we have one of the highest retention like

internal retention like we don't have no turnover rate really uh it's really low i mean outside of the stuff that we can't control like contractors and stuff like that we try our best the company is warming up and doing a really good job doing it they recognize the value um but i mean i think for everyone in this room it's it's really valuable that we do what we do and to share with one another um and i think going open source definitely helps each other out because we can then share things like this right yeah yes thank you very much hey there uh thanks for the talk uh so quickly just uh two two questions um i guess first for midsize companies uh we have enough trouble just convincing our clients just to at least use ci cd in any capacity um but you didn't i i may have missed it you may you may have talked a little bit about uh sort of your deployment uh best practices and when we're working with our clients we're already trying to uh convince them

to you know use jenkins but what tips might you have for uh deploying sort of you know hundreds or or tens of gigs of data to uh you know servers or or wherever i guess for companies or clients that might not be comfortable uh deploying them to say dropbox or you know google cloud or something like that that's that's that is a very good question so i i'm i'm lucky enough to not have that problem myself personally with nea we have our own a team called rpm release and preservation management so there's a talker there about preserving stuff we work directly with them they also manage the distribution network uh so i know that we use a dedicated hardware to do to do that to uh to securely transfer data like that encrypted to we do it to i think amazon so that's just an example of how you can push things to the cloud using dedicated hardware now the piece of hardware is quite expensive but when

you're looking at securing things and not increasing your software side i mean other than secure ftp i mean that'd be the the free stuff to try to use to start with yeah let's start with we also within the company we use uh wan accelerators wide area network accelerators hardware so that's also expensive but i mean there's there's hardware options for optimizing that and those are specifically used for you know your code co-developed uh studios and stuff like that and your big games right now as far as convincing people to uci it's hard it really is um there's nothing harder than convincing people that don't want to change the quickest way and that's why you know if you have time and you can invest in in elastic if you're if you if you have money constraints definitely go that go down that route first hook up the data start using it generate the reports get buy-in with those reports because execs non-technical people love those um even technical people love that because it numbers shut people up

so if you do that then i then that's the way to go um and then i mentioned a slide deck about setting the automation set up the builds first and then your static analysis those are the easiest to do and they'll start getting you some returns the the test will be the most difficult step i think because you need to decide on a test framework every game is different if you're using unity or uh or unreal i think you you have a your leg up that they have test frameworks that you can leverage uh i know some parts of our team or our company do that so um i think you just need to analyze it but the free there's a lot of free options i said there and that was kind of my inspiration for the talk was we also use free stuff and we're you know uh my team we will be looking contributing some of our stuff back to open source but i still need to go through most of the red tape okay so yeah so the um i guess leading into the second question right so our we've thankfully convinced our clients to use ci and cd it's just necessarily trying to convince them to put some more money into the hardware and kind of

upgrading uh but i guess uh thinking sort of stepping aside from that do you have any tips i guess i don't know if you're involved uh sort of in the hiring process at ea but do you have any sort of tips to try and attract talent that is in ci cd because we're having issues trying to find more like-minded individuals and i guess uh you know it is there a discord channel or something out there that you might go for to find someone who's interested that's uh that's a great question i know that uh the automation roundtable automated testing round table is happening at the same time um i think there's individuals here like in this room here our our community is quite small especially in the gaming the gaming community so i've this is the first time i think there's a dedicated ci talk or somewhat like that so i'm excited that all you guys showed up actually i was like this is a big room i hope i have some people come especially at this time especially this uh this time slot um i don't know if you want to set something up i'd be more welcome to help join but i was i didn't mention i had my slide deck i forgot a lot but i was

involved with hiring like majority of the like 30 40 engineers on the team the thing that i look at for the most on our team is the passion you know technical ability is never for our our stuff it's nice but the passion is the most important one to do because passionate people they always learn and my teams have i've evolved with my team and our team's evolved constantly so i think that's the expectation to have to have those individuals i started reaching out to local universities from vancouver where i work and trying to do that i haven't had that much time but i think it's it's a hard one because nobody no school really teaches it i did have that's kind of part of the issue there's no yeah there's no class about it so hopefully by us by me doing a talk and by us talking more as a community we'll start garnering that intention and actually bringing that you know there's actually a career path in ci within not just the game industry but at large

and it's very important so cool thank you adrian yeah that's it thank you everyone