Game Preservation Best Practices: A Real Life EA Case Study

[Music] my name is Jared Lee I'm a software engineer Electronic Arts and my work encompasses all that is preservation I handle the preservation of some of our headlining sports titles and I make it so that they can come back again one day whether it be 10 years later 30 40 even a hundred years I make sure that that is possible my initial role though while do engineer work day-to-day I sure got my start developing tools developing tools specifically in a technical art space so I kind of have experience with both making games and kind of doing weird preservation work with them so that's my background that's what I do and you'll see soon that how that actually plays into preservation that said let's kick things off what is this thing called preservation many of you probably know what it is some of you might even be doing it let's talk a little bit about it so preservation is simply put the process that puts a game away all the things that went into the game all the

files all the data and all the documents in such a way so that it can come back again and many of you right now have seen the direct success of preservation who here seen any of these titles at on the screen right now excellent you have seen preservation success but preservation is not something that is always perfect and there are stories of tragedy in the preservation space and we'll get into those in a little bit but preservation is much more than simply this what I just said we see the end goal of it but there's a lot that goes into this and actually one expo is what we Electronic Arts do and what we look at preservation as in form of a maturity model and simply put this will allow you to see where you sit in the realm of preservation at least from our eyes so let's open up this discussion with simply a tier-one archive now a tear when our kyv is what we see as take a box any box it could mean SSD it can be a hard drive it can be anything you constitute as a box you take all the things that comprise your

game or your work that can be the tools that can be your data that can be your code and you simply merge the two together to form the archive who here is doing this right now okay some of you that's awesome that's a great start now some of you might see this and go that's awfully familiar to what we call it back up and you're technically right to a degree a backup is very similar to this and anyone who's currently doing a backup is already on their way that's excellent the difference between a backup and a Tier one archive released to us is that do you archive your documentation about your team photos what about your server tech how much do you preserve of the actual game besides just the content that built it if you preserve the documentation as well and all those kind of exhilarating --zz those are what can make this a Tier one archive as opposed to a backup but ultimately speaking we want to preserve things because there's value in what we can bring them back for this while it is

valuable because that is what the game was built as it you still have to have a familiar knowledge with how to take this jump of stuff and remake that game again with it so the value in Tier one archive is limited so what we did is we said was the next step was the next tier of value that we can control with this that led us to the tr2 archive now to or to archive starts a bit differently we actually start with a machine we take the exact same things again but we do with something a little bit different we take those tools and we install them we configure them how would an artist use the artists software how Dannette build engineer reproduce builds what software did they use how was it configured all of this needs to be done so that's they can interface with this archive exactly how they would have in production the data can this machine take a brand new piece of content and inject it straight through the pipeline into a new build of the game if you can excellent but can

you also build a new release of the game if you want to patch any patch an archive title let's say someone discovered a security flaw in your build and ten years later you're like oh boy we should get that fixed you need to be able to reproduce a patch with that archive and there's many other things you would do with it - you're - but simply put can you reproduce the game if you would have stand this machine up could you produce a new release of the game new content for the game and your DLC for the game that is what it's here to archive is for but let's not forget there is one specialty requirement of a tier 2 archive something we probably would all fail to check for right now can it all be done without any access of the Internet who here says all their work would 100% function without internet exactly the point of an archive is so that this thing will it function in let's say 50 years if you have internet access right

now you're probably interfacing with network shares server let's say server programs the works maybe domain accounts if you turn your machine off you prove can this work independently amongst itself and so that is the ultimate verification step for a teacher to archive and it's obviously speaking quite difficult to do but it is a valuable step because this truly allows us to push the game further now that's tier 2 and a lot of the time we at Electronic Arts are able to get to this point but even for us it is a very very challenging thing to do here to archive is not easy as one can imagine so we try and aim for tier two but a lot of the time we do end up at tier one but like anything else and many tools developers in this room will know we want to go to the magnum opus you want to get to the end goal the greatest possible method of this as we can so we formed what I like to call the mythical tier the mythical tier starts the same

way as a tier 2 archive you get a machine you get the same things of the game but do you use online services all of those need to be preserved as well now let me hit home how complicated this is for instance if your game runs on a server can this machine or run a local server that you were using and can it connect to that server can you run a client and connect to your server ok that's stage one if you can awesome sage - most people have gone there builds to a distribution service for instance we use origin could you localize origins so that you could technically build a new build with game with this run it through origin locally and use origin to distribute it again with that single machine do you have metrics gathering tools analysis tools the list goes on and on if you can get this archived you've truly achieved in mythical progress but that is incredibly difficult to do and

Electronic Arts we strive to try to reach this model but is it not simple and those are the three tiers of archiving those are the three tiers of where we believe people will sit and kind of the grand scheme of archiving if you currently sit kind of close to tier 1 or even in tier 1 excellent if you're just sitting just before tier 1 we're not doing documentation push a little bit more it'll feel really good when you can look back and say wow we built this 15 years ago this is incredible if you're sitting at tier 2 awesome let's have a conversation I'd love to see how are you doing it if you're at tier 3 I have a lot of questions for you now that's preservation as a concept let's see what Electronic Arts does let's talk about like preservation at Electronic Arts so Electronic Arts many of you know us you probably heard our names in the news and simply put we have about 35 plus years of titles both archived and analyzed for decades what's great is this both allows us to see how games as

a medium has grown over those years but also how has preservation grown how has preservation grown since let's say the early 80s as opposed to now probably quite a lot but I'll be doing an injustice to games if I didn't talk about how big they are who here works as a game developer who here is act knows fully well the complete size of everything you built for your project perfect some of you that's awesome it's a rare question to ask because if I asked you how big your game is let's say for Electronic Arts we have these monolithic numbers of builds released daily you output a lot of data and we're talking about preservation and we look at something like this we wonder how we supposed to preserve that in what right mind do we preserve that that is an extremely important question to us Electronic Arts but simply but there is not one person to do it in fact we have three groups I like to see as the responsible partners

for the preservation process let's can look at those three partners the first part are the content creators the content creators are the programmers the artists the animators the producers the ones who really just make the game they are also in charge not solely of just making the game how can they better the process of archiving can they embed tools in their workflow that make the archiver's job easier so they do have a hand in this but they ultimately create the game and now it needs to be preserved so who does that who handles preservation Electronic Arts we don't technically have a single sole archiving team we have many people from both game teams themselves or technology teams like myself who are responsible for performing the process of archiving and so we kind of call them archivists and preservationist s-- now the archivist and preservationists are solely responsible for taking all that built all that went into building the game all

the tools and software that they use to preserve the game and using that to produce an archive whether it be in tier 1 tier 2 tier 3 or some other unknown tier they are responsible for that process as you can imagine what we dig quite a bit into this and to see what this actually looks like but these are the folks who really dig into that what happens next who does archiving after that they simply put you can't just archive a game and leave it in a box somewhere so we have a dedicated team who managed the archives there the preservation managers they are the team responsible for where do I put this and how can it survive for a hundred years what happens if the entire server goes down and all of our archives get lost do we have backups how do we maintain them are we able to see technology grow and still say that these archives are functional they are the maidens people of these archives and these are the final holders of those archives now many of you probably have either a small

department or just a few people who handle archiving and as you grow you'll notice that the requirements of archiving becomes so great that at this becomes more reasonable but what is actually goes into archiving I've talked a lot about archiving the concept the teams that handle it I want to dig in I want to reveal exactly what I do day to day to preserve any one of our games so let's talk about the tasks of preservation I'm gonna go over exactly what a tear to archive looks like at Electronic Arts and all it always starts the same way this set up the set up simply starts with a research who here loves research all right I appreciate it research is absolutely the most important thing as I mentioned before the archivists and preservationist group they don't always work directly on the title there are sometimes a technologist like myself we do not have familiarity with the title so how would we be the best ones to reproduce the environment for title this

gives us that information go through all the documentation you can find speak with some members of the teams and gather the information do you know SDKs the game were built on what about the platforms what's technology do they use how was their builds produced what game engine to use the list can go on and on and on ultimately this is just to gather and make you as prepared as you can be for what you're about to do what comes next is getting the hardware for this as one can imagine as I mentioned before and especially those who know how big their games are you're gonna eat a lot of hardware for this simply you need terabytes and terabytes of storage data for this process and it's taking understanding and this is actually gathered in the first part you need to understand how big your game is first before you request this so once you have all these like crazy machines set up you have all these terabytes of data or of hardware let's give them set up now this step is completely impossible if you do not do your research how did an artist interface and build content for the game

does anyone in view in this room know how to do everything possible to remake the game from scratch probably not well some of you were like I know quite a bit about it that's awesome as you grow in your game gets bigger this becomes inconceivable to be able to do everything so work with your artists work with your programmers get the Machine set up maybe you've asked them to play with it once you have all the software installed you're certain you've successfully set up now you can move on to stage 2 requisition requisitions a short phase technically but it all starts out with the same way acquire absolutely everything literally everything you can find now this is awesome and easy if your data was stored in the same place that is a dream and that does not happen in the real world most likely people will store it may be on the main perforce server or the git server but some people will store it in their onedrive they'll store on their local desktop which is truly the worst

people will not always start the same way Sudheer due diligence scan through everything you will eventually find all that you need to find get that on to these machines now get it to a local SCM server SCM just sends for version control if you use perforce and get all that data onto perforce if you were to use this archive to operate upon that data you have effectively altered the data that was use in production you do not want to do that you need the original files because that is what the production people used if you toy with it you've rendered it not production-ready or you've rendered it not exactly what they did get it onto local server so you can make copies with that you can play with those copies and it's also better to have multiple copies than one once you've done that great you have everything the machines set up to do stuff lets do stuff reproduction reproduction is truly the longest step because this is the first step reproduce every release in many games and

especially in the larger triple-a companies they will have a lot of releases alpha-beta final that models pretty much standard then you have patches one 3x DLC one 3x perhaps you'll even have variants of them for instance if you released a Japanese version of the game that's probably a wholly separate build than an English version of the Brazilian version what do you need in 10 years what do you think the people will need in 30 years every possible release has to be verified that can be produced by these archive machines probably a lot to ask of a and so to give you a figure if I if I for instance asked you to do ten patches per region less you have six regions per release let's say there's eight releases that's probably a lot of build you need to verify so it helps to make sure this machine is quite capable of doing it all again but it's not everything though obviously this machine needs to be able to function and create new stuff not just reproduce the old repeat is all the

content as well this means taking an audio file putting through the pipeline and getting it into until when you build a game ask an audio an artist or an artist or if an effects artist make these new content files on this machine if they can great they should feel at home they worked on this game they can work on this machine that's perfect match made in heaven that is an important step and America member there's a lot of realms of content so don't just associate art and audio as the only two parts of content there's a lot and lastly you gotta verify remember you're not doing this for yourself this is the person in 30 years from now is gonna be opening this hey what you probably might not be at a company or you may be somewhere else at the time do your due diligence verify everything what I'd like to do get a build onto a console if you produced a new and I know patch 18 build or whatever get that on to the console and verify that it runs if you've created a new asset let's say you made a new character get the character into a build get the build

onto a console that is probably the best stress test of the reproduction of stage as you can get and so as you'd once you've done all that you've done everything this machine is pretty much perfect you're probably tired you want to go home let's get to submitted and you can so this mission starts off with cataloging I know you probably haven't heard that word in a long time Electronic Arts we have this great tool that we use that allows us to catalogue everything it's a tool that we use so that it can be accessible by any developer at anytime it is filled in by the archiver's and it's able to depict details about what was our tired for instance if you we made a title in 2001 we can say what was our cut what was put in that machine what tools did they use at the time instead of dredging up that machine again I can just look at this like webpage that's all I need to do I recommend if any of you are really investing in the archiving space get a catalog of people who actually really want to see that for those artists in

the room wouldn't you want to see what you did five years ago see the original files maybe even play with them again it'd be cool once you've done that you catalogued everything submit it send your final email get it to the responsible parties who manage these are these archives the most likely peripheral perform their own verification two-sided verifications perfect don't want to just have you be the final person to verify once it's done transfer it to their final storage place mostly with us we try to do a cloning will clone the actual machines into our final storage so what you have is merely a copy with that copy and you've got it stored your game has done everything with an archived you show everything everything just did for months and months just trash it it's been cloned it's been put away destroy all your machines put the hardware back into the pool you're done and that's what your to archive who feels they're ready to do this right now it's probably

not this is difficult it's very very time-consuming but I believe all of you probably can if you'd look into it and you try however we weren't able to get this the right away this tear especially not tear 3 but this tear was very very difficult to get to and it wasn't just one person saying I'll do archiving this can be easy it spent years of conversations and discussions to convince people that this is important it wasn't until we created a cultural change and got the team on board with this that it finally started and all that always starts with the same thing why do we archive why do we put so much time and money and thought into preserving what we've done we'll simply put there's a lot of reasons we archive for remastering purposes if we want to bring it up into a legal matter for patching if for instance again security

for found you could patch your game you want it analyzed what tools were available by our competitors as opposed to this thing we built ten years ago once you have the fundamental understanding of all the realms archiving can play in you've created a great foundation for the reasoning behind archiving but I really doubt that just showing LSA and executive producer or a lead the people who can make those decisions to adopt this this is not enough for them they want cold hard facts and they will come at you with cold hard questions let's talk about a really common one who here has heard this question before or even you yourselves have said this before you probably have seen this question said and it's totally fair it's absolutely fine you may never want to remaster your game it may be something that's just not applicable to your work but I highly doubt that these titles and these companies thought the same way do you think when Starcraft was made that they're like oh yeah we're gonna release

this again in like 15 years probably not even Dark Souls it wasn't released that long ago in known perspective I really doubt that they thought that as well it's normal for companies not to think of archiving remasters are not a thing you typically think about that's not the only thing leads will come at you with again they have a list of questions it's too expensive it is expensive you have to dedicate someone to it you have to make sure it's good there's quality checks involved the bigger you get the harder it'll be it is expensive but it's also it pays off let's take let's talk Spyro for example I love the headline there's actually the headline says this little purple dragon flies in the top chart in the UK despite all the remaster topped the UK sales charts so much so it actually feed out red redemption to a title that was going for Game of the Year that year was beat out by Spyro a remaster there are facts like this and

is normal let's talk about Resident Evil the HDS really a remaster it was the fastest selling digital game beat Capcom has ever made of at the time I think monster entered now just beat it out but think about that two examples of a remastered title topping sales charts they've effectively made their money back twice that's some pretty good income if I say so myself but ultimately speaking we gotta go further we have to do more push even harder to convince people so I have three suggestions these are high level suggestions and I want you to think about it and your own work you got a first critter realization what happens if you don't preserve where as a dark side of preservation let me share some stories who here has heard of Blade Runner the game that actually had more than I expected awesome so blender in a game more more

people know about the movie but there's a game made years ago it was a wonderfully made game he had a really good solid cult following during their liquidation and they're moved from Los Angeles Vegas to Los Angeles like Mad Dog magnetic platters or HD or hard drives that held the game completely vanished they lost everything everything that comprised that title went missing let me hit this home and an interview with gearbox when they wanted to trade the rights over oh what do you think the number was estimated as to replace that data forty million dollars forty million dollars with a Content lost as you can imagine that can't be franchise died and there is no possibility of a remaster you'd have to remake it from scratch but many of you don't know Blade Runner and that's fair let's talk about an example we probably all know about who's here heard of Starcraft probably a bit more bit more of you so we think that triple-a titles

are impervious to the issues of preservation no-one is impervious to this Starcraft and actually you can go look up yourself an interview with the art director during when they're doing the HD remaster of Starcraft it was figured out that we lost all the art content what do we do then actually he states I had to tell my artists to eyeball everything back into existence using YouTube videos copies of the game and photographs off Google I wonder how expensive that was so yes this affects every domain but let me really hit this home because it's while those are matters of things just straight up not being preserved even if you do preserve you can still run into trouble who hears heard of Silent Hill oh yes the list goes on Silent Hill is a title that was was preserved they did do their job the problem is that and again

once an interview with the director they'd love to talk about the stuff an interview with the director it was stated that we realized that what we preserved wasn't the final release of the game we only preserved the beta so effectively everything they did post beta was gone I wonder what kind of bugs they'll have in the game now as you can imagine that's not good and so it's a precaution you see stories convince your fellows and show them no matter where you are how big you are or even if you already have preserved there is still danger now that's a theoretical portion of preservation let's talk about the actual cost of preservation are you capable of preserving how many heads counts are you gonna have for an archive Electronic Arts we typically lean between one and three depending on the complexity of the title are you prepared to dedicate those numbers alright how

many software do you expect him to use that list alone covers maybe 15% of the number of software a game typically is built with can you say that those one to three people are capable of interfacing with every single one of those to verify your archive is completely no capable of being used or at least you have something answers as to how to verify that does all software can be used okay so let's say you do let's say you you have a client I can bring some artists on board you can get them to play at the archive how long do you think is going to take how much money you think is going to cost it's not cheap can you give me a number if you add these things these three things together and you look into a little bit more you form a plan you can create a cost and tag that to an archive that is a number you can give to a lead you can get to a producer and executive and they can say okay so this basically if I want three archives it cost this times three that is pretty good data to give them again they're all

about data so I recommend taking some time to analyze how long would it take to for you to do an archive but archiving is still a technical process you still have to do the archiving work so you got to make sure to reduce the manual labor or it's going to be terrible and frankly said archiving sucks our Chi ving is a slow egregious process that's based purely on the tools that were used before so how can we make this better a lot of people will typically say let's just throw people out the problem well increase the headcount we'll say let's take our senior engineers or our senior members and say we need you to work on the title you guys may have some great archiving experience and that's great that's awesome you're probably gonna gain a lot of time from that but your tools are slow they're not plentiful and they'll have perhaps issues in them that will make this take a long time you're not going to gain anything it's going to be probably be the exact same as it was before so that said let's talk about some tools tools that I like to use my

own work tools that we use Electronic Arts and tools that I believe anyone can use to help better their work both an archiving and probably anywhere now the first one I want to talk about many many of you probably know right now who here is heard of virtual machines perfect so let me let me kind of Express the origins of archiving back in the day before this kind technology existed let's take effort by physical PC now let's say we want all the aspects of the game put onto that PC how large you think that PC is going to be a lot of terabytes in fact is pre normal for us to get 10 plus terabyte machines now let me pose a situation an audio engineer comes up to you and says I want this I want the audio environments not gamey we hope to be preserved do you think you'll be happy when you say here have this 10th era by Machine have fun probably not very happy what can we do to rectify the situation

that's where we can pull in virtual machines now again virtual machines start with the physical PC what we can do is we can split up the hardware to create these small virtualized environments that all function is complete isolated machines they all can be happy tools installed to them they all function in isolation and now we can say each one of you take an aspect of the game let's take at virtual means say machines say you are the art you are the audio machine you express all the audio data they audio tools the audio software will have another one for art's another one for builds another one for a server the list goes on your compartmentalizing everything in your game so now let's go back to the situation ananda our engineer comes up he wants to see the audio environment how happy do you think he'll be by saying here is literally only the audio probably a lot happier and so this is a great tool that's really provided a lot of value to us i recommend everybody here going exploring a little bit the barrier of entry is not

too extreme and can provide a lot of value but ultimately speaking while this helps us set up archives we still have trouble when it comes to getting it set up further again setting up the audio software the arts software the tools that production used how do we make that easier well there's a wonderful tool called chef who hears heard a chef before after you a few a few less than the less of you but chef is a wonderful tool but let me again let me pose the situations we all can be familiar with it artists are a fun group of people artists have a wonderfully large amount of software in their tool belt too many to speak of I'm looking at you artists the problem with the situation is that how do you preserve that many things for instance if I say an artist has 20-plus software and I as a billet engineer needs to go and set those up how in the world am I supposed to do that I have no clue how to operate them so if

I were to install them even if I was following documentation I'd probably be led into some human error i probably set it up wrong I miss a path I'd probably mess it up I wouldn't know how to to resolve any issues with it and so it becomes an inefficient and inefficient and ineffective process so we've done is we looked at chef by saying okay so typically speaking I'm a built engineer what can I do to use my skillset to better the process of archiving can i code it the typical engineer response can I just code it doing code yes you can in the world of chef chef allows you to create what's called cookbooks yes I know the naming convention is wonderful you create these cookbooks and each cookbook has a resist of recipes the recipes all defined a software and let's say the software version and a set of instructions because you are about to go with this a set of instructions define encode exactly how to configure install it and place onto the machine you work

with an artist you say tell me how I need to install this I will write it in this CODIS recipe it will be done through code effectively just codified my entire game and the practice is called configuration as code the beauty of this and while this is great if you're working on a single title the more your company grows the more tiles you have going on at once the more you need to see that your technology grows with you because this process is codified and is frankly just a bunch of text files I can literally just copy in and within 10 seconds make another one with a whole nother set of software listed in all different versions that's wonderful and with a company like Electronic Arts we have so many games going on at once this is an invaluable process to us and so I highly recommend it once again you can access it right now the barrier of entry is a little bit steep but I do recommend it they have a wonderful set of documentation and while ultimately this provides a lot of value for us today what does tomorrow look like what does a

future look like now reap repaired for it the future is going to be weird it's going to be intense and extraordinarily quick in development Electronic Arts we know this and in fact we talked about this in a 24 page report from our CTO Ken Moss he talked about what's called a project you can all access it it's publicly available and talks about yeas giant pushed to the cloud so far we've defined a processes about defining building games without cloud services what does preservation look like with cloud services are we ready for that in a interview with Bloomberg our CEO Andrew Wilson talked about right now while we're doing yearly release models in some of our games what happens if we bring that down to monthly weekly daily it's already challenging to get games done in a year in terms of archiving how can we do that when it's so much faster

these two things are terrifying concepts when it comes to preservation but no matter how scary the future looks we have to look back at the basics why are we archiving in the first place we're archiving because we have history we have history we have games we love both as users as developers we have thousands of hours sunk into making sure our players are happy if we don't preserve if we don't preserve making sure our games are recorded in history everything you ever loved in games will disappear now that more bethought out of the way let's wrap up don't be known as an industry tragedy I've told you stories about what happens if you don't preserve I've hopefully hit it home that that's not a good thing I do not want to put you on this list next year automating build produced manual

labor right now archiving is awful it is a slow process our tools suck how can we make it better is up to you looking to it let's have a conversation there's always way we can make it better but again you have to be prepared archiving might be stressful today it's going to get a lot weirder and a lot harder in the future so with that said thank you for coming all right I believe you have time for some questions we actually have a lot of time for questions so feel free to stand up to the stand and ask me what you like hi so how does this process differ if you're like a one-person team all right so one person team is a great question so let me rephrase the the question that you were asking is how does this process apply to a one-person team now when you talked about the historical preservation

of things do you want to see your games again in ten years most likely yes then you should be doing it if you want to see your work in ten years you should be preserving not you don't always have the understanding of whether you want to remaster and that's not relevant what's relevant is that do you want to proceed your games again if that's your if your answer is yes then you should probably do be doing preservation if you're in terms of process though if you when you're doing a one-person team tier two and Tier three is pretty hard to achieve if you can get virtual machines up and running however a tier one I think is a great start I'd say a Tier one and then if you can grow to a tier two that answer your question oh yeah thank you thank you I think it's kind of related to that one so all right small team right the probably zero days off between projects you know what what's sort of the what sort of tier

zero right like what what is the like okay you can't even get to tier one what's the like worst case scenario just get something in there process for you guys so the question was small like small team again what happened what would the the baseline be that kind of a good sure that's a good way yeah so what would the baseline of preservation be where where could I start if nothing else it's here I would say backup is a Tier one if you use version control or if you simply have your files stored on Google Drive or anything like that right click hit download put it into a folder everything you've ever built put it into a folder that is I would consider it to your one yeah and then and then like a weird archivist 50 years from now might piece it together exactly a totally yeah so yeah again tier 1 and tier tier 0 inter one is really difference between the axillary aspects like documentation if you want to just get it record a record of your stuff yeah just hit download that's all I need to do that's a good entry point cool thanks thank you hey thanks for the great talk

I was wondering during the process of internally in the company introducing this archiving did you have to at any point for example involve InfoSec did you come across with bad practices you know something that you had to globally change for example during a a like taking practical example like one content team just using a pirated software or know something that you know was just completely incompatible but you kind of like accidentally found it because you had to review the whole I guess during the archiving you get to see all the practices the company does currently so before I rephrase the question I just want to clarify yep do you mean in these practices of preservation or the practices of the developer breakfast of how the company develops software and how you know what tools use so the question was during my work in Electronic Arts and during the work of preservation you typically come across or new come across bad practices is that a fair yes or do you have any kind of experience or got just like even though you wouldn't start archiving

process now maybe issues still within your company check are you also doing this perfect so yes I have definitely come across bad practices is evident everywhere not every developer is perfect not every developer lines up with preservation a common one is that where they store files it is a common thing for developers to store files in places that is not known whose archiving it and what is a very dangerous terminology we use is tribal knowledge it's a tribal knowledge is when a team has knowledge strictly with amongst themselves they have not shared it it does not exist outside of that that is a massive danger to preservation because if the person preserving preserving it doesn't know it exists it won't be preserved and so yes there are a lot of bad practices tools that require let's say non perpetual licenses are a great deal of a threat to preservation and there are so four have perhaps external vendors none of that can be

archived so that is a red flag and so yes to answer your question there are a lot of cases where I think thank you thank you so I think that you just kind of touched on it at the end but knowing that software be Amaya or anything else as licensing servers and now console development hardware has licensing word after 180-day as you get locked out what is EA doing to try to ensure that you have some sort of access outside of that area alright so the question was with licenses being perhaps not perpetual and they do expire after a time or software stopped working after a period of time what is Electronic Arts doing to combat that what are they doing to making sure that they can still operate with that in mind that good recollection ok so the question and the answer is that a lot of it's a weird a weird situation a lot of companies are working with us to make sure that what are your needs what can we do for you and a lot of companies do opt for perpetual licenses or they will

kind of work with us to get one okay that's great there are cases where inevitably we cannot provide a license for instance audio software are known for that it's expensive licenses and it's troubling to just put that into an archive and so there are situations where we cannot say that this will 100% function okay and so yes there is an inevitable end of we are trying our best we are constantly communications other companies to try to get to work but there is that kind of end that we have to expect thank you thank you I want to ask about do you have a preferred storage medium tape blu-ray glacier and then how do you keep track of what's in this giant storage medium evolved archive data alright so the question was what is our per forwards towards medium AWS glacier hard drive yeah and w-30 had a follow-up point to that as well yeah I was like so you mentioned the catalog which is kind of like a web page list of what's in each individual title archive

I guess but how do you like if you have you know a bunch of glacier buckets or a bunch of hard drives on a shelf do you have a different way or is it you just do it in the catalog of track of which hard drive has this title that kind of thing that's a great question so the follow point of that is how are we keeping track of the data scale of this stuff and so it's great question thankfully enough technology has provided answers in terms of tools we can run masca queries based on the storage I'm not too keen or privy to the information regarding what service we use for the storage I want to say glacier but I can't I can't confirm only because I don't know personally yeah regarding track and scale there are a lot of command line options you can do you run a couple day long batch yes a couple day long batch and it'll just track and scan everything I mean we can record that we were doing archiving and we as archivers when we do our work we are constantly querying everything for instance perforce as I version chill system has a built-in answer for that and so we probably know that when it

gets put away we most likely do a tool based scan cool thank you hi is there much carryover between the work that ei does privately with like public organizations companies that are looking or groups or companies they're looking to sort of preserve the history overall of games as a medium you know D is there information that you share like that so the question was do we work alongside other companies who are kind of in this space of preservation and do we work on their processes or share our information that kind of a fear so we definitely we have a great interest in this space for example I'm here trying to meet other people who are doing preservation we are going to conferences around the world trying to engage with partners there are preservation meeting strictly amongst preservation engineers to discuss this very topic and so we are definitely going out and we are learning about other developers and learning about how they preserve we have had conversations with historical Mersey museums in fact some of our people have actually had pictures with these famous articles lulz games we definitely have

an interest in going out of the way in fact I'm here for that very reason that answer your question yes thank you thank you any last questions anyone want to roast me a little bit all right I think that is it thank you very much everyone for coming you