Working Smart - Machine Learning in Art Asset Creation for the Games Industry PG Connects Digital #2

our next session um which is with uh kelly vero of so real digital twins um and she is going to tell us a little bit more about working smart machine learning in art asset creation for the games industry and kelly is already with us hello hi um i'm kelly vero i'm based in switzerland at uh soviet digital twins and i want to walk you through um something that we've entitled working smart which is machine learning in art asset creation for the games industry it sounds like really amazing future gazing science but it's actually a really great simple

way of being able to make your process faster so and without further ado i'm going to share my screen right now and we can go through it together please feel free to ask questions at the end of the panel presentation um yeah let's go take it away so you can see my screen and as i said i'm kelly vero head of game development at so real digital twins what does that mean well i've been working in the games industry for 25 years so i've seen everything move from analog to digital in both a really fast way and sometimes in the games industry we like to hold on to the traditions and we like to hold on to the values and principles that make our industry really great but we don't have to lose those things in order to move forwards um and and so to get started

let's talk about how machine learning can be used in the games industry and why the hell should you care about it well simply it's about speed it's about speeding up the process of being able to get to market much faster than we can get to market right now so that means creating individual or group algorithms that manage an entire process particularly i'm focused in the art asset creation section of the business after all at so real studios um in switzerland we turn something like a simple bulb um you know that you would find in a light or whatever into an actual digital game-ready object and that can be used in the unity engine or in the unreal engine it's also available for vr so any kind of vr experience or ar experience we're effectively doing something called

digital twins so we have to treat these objects in a way that we keep the state of what the original object is in the digital as it was in the physical so because of that we have a series of automated platforms that we need to dive into within the studio not just concerning the end product which is potentially you guys um or the end user but also the automated experience from the aspect of the physical object and how it handles this also gives a better experience for the end user because they're getting something especially on the soviet digital twin side that is real so they're not getting anything that's cad created they're not getting anything that's been concepted they're getting an actual object and i'll explain why in a few moments it's so important and then we're creating something that's solution developed

so we're not creating a solution for a problem that you guys don't have we're creating a solution for a problem that everybody has or is going to have and that's about the user experience so that's why these guys asked me to come and work at soviet digital twins because i have an understanding of game development from start to finish from about 1995 don't tell anybody all the way up to present day the subset of of sorry machine learning is the subset of artificial intelligence and so we know that ai exists um in games already i've worked on a couple of games which i'm sure if a few of you know about um some of the games i've worked on by the way you can just contact me on linkedin or reach out to me however you want but i've worked on some aaa stuff also some mobile development and ai tends to live in places like um camera handling

so we know that accidentally ai appeared through um the forza development um they fell and stumbled upon this amazing ai and then just cast it out of the development also i understand we picked it up when i worked on transformers universe and we used ai to handle the cameras and especially the vehicle handling um in that mmo game unfortunately it's quite short-lived but that's the games industry so let's talk about image segmentation for art and games um we would use it simply to examine simplify speed up the process learn from and predict what objects do not just what objects do for art generally but for what objects do for art in video games so how they can be handled as part of the overall user experience after all as i said um we've got to remember that the user is at the heart of this

so the user for us can be anybody from the client to the actual game player themselves so we need to make sure one that machine learning gives us the ability to examine each object in turn and so what it does how it behaves and how we can simplify that process in terms of the creation of more of these objects because in the case of the light bulb i don't want to just create one i want you to be able to have 40 to choose from 50 or 40 000 depending on what your needs are and they might span from modern led light bulbs to something that is a gas powered valve so what i need to be able to do is make sure that the machine learning on our automated platform is able to handle that information can examine that information and bring that information to the fore faster than using cad or doing concept drawings modeling

measuring etc so speed is a massive part simplification is a huge part because i don't know about anybody else that's watching this talk one of the great things about being virtual now is the fact that we don't want to have any human contact whatsoever apart from in video right so if we can simplify that process by taking all of the humans out of the equation that's a great thing and then we want the machines to learn so we want them to get better and that's what machine learning is about it's about taking segmentations of what the object is learning from it and then predicting what the outcomes will be in terms of the object itself so the practical use cases in the games industry could be and there are like a million ways that you can use machine learning i've already mentioned too about uh vehicle camera handling all that kind of stuff but in art i'm really sort of interested in asset storage and creation and i'm also really interested in things

like predictive marketing and research that's really important as part of our job as well but obviously not on the digital twin side so vertex object examination analysis is really important and interesting because sometimes when you're optimizing objects you can lose quite a lot of quality we don't want to do that so we would write algorithms or use the machine learning platform to be able to enable us to keep that quality throughout the entire process of the object selection development and placing it in the scene also from the prospect of pipeline performance and from that perspective that's really interesting because i'm also a project manager and i know that a few of you that are tuned in today project managers too so you might find it really interesting to create very valuable use cases either from an agile standpoint or just building a better studio you can

do that by digital twinning and you can naturally do that through machine learning if you have in the case of say jagex or in the case of um epic or square enix montreal you've got millions of people that you've got to manage you need to be able to find a way to manage all of those pipelines and manage that performance to a really high end you can do that by creating digital environments and you can manage that by using machine learning and then finally qa qa is fantastic and it really does need the human touch but as we move a bit further forward there is not a massive needs these days to ensure that we've dotted every time every iron crossed every t we can certainly use machine learning to a greater extent to manage everything from unit code testing unit tests generally and towards the more focused user experience

end of the business if you get the algorithm right you get the machine learning right you get the qa right how does it work first of all we use trainable segmentation so we train the machine or the algorithm to look for exactly what it is that we want to do in the case of objects in motion what you see here is like on the left hand side there's a buff in view there's a lot of input there's a lot of hidden segments and then there's an output and then i put on the other side a non-buff interview because what i need to do is train our system into finding the objects that i need for my clients in this case and from now until the end of july it's so real we're going to be looking at over 750 objects of the same things ballpoint pens iphones even pomegranates even goddamnit hair dryers but you've got to ask yourself a few questions what are you looking for the detail which changes from article to

article is what you should be looking for if i want to work for fallout and produce objects for them i'm going to require a different type of object than if i'm creating something for eve online right so these are at two opposite ends of kind of a time spectrum so a hairdryer in fallout is going to look like it's made out of bakelite it's going to have fabric leads but in the future in the eve online world we might actually see that they might be using something like a dyson hairdryer or they might be totally wireless or cordless so using the non-boffins view that's what i see there you can see that the trainable segmentation pulls out what the input is so i tell the machine that i want it to look for specific vertices so it pulls those vertices out for me and says these are the things in this image that you might want to use these are the

things that are closest to what it is that you're looking for it's a little bit like having databases or things stored in tables and setting up search functions to find that stuff it's just a little bit deeper because what we're dealing with is a different type of data we're using segmented data so it's data that's not completely stringed it's not completely finished it's part of something bigger we cluster that information so we need to set data and threshold techniques we have to turn things like um grayscale into binary um the the key of this method is to like select the threshold value or the values when multiple levels are selected and that's what we want to do when we're turning grayscale into binary we can also do histogram data and thresholding and in this technique we compute from all the pixels in the image and so that's the peaks and the valleys and we use them to locate

the clusters inside the image and then finally you can use the otsu method for ct and that's the maximum variance and what oxy does is it takes the raw image and for us you'll find out why that's really important in a moment and we break it down into segments before reconstructing that's super important i've just done a really great scan actually on gundam and what the um scanner did was it automatically up sued everything for me it broke everything down into very specific parts of the raw image so then what we could do in our automated system was use the automation to put it back together again basically clustering is asking questions the algorithm should ask the questions for you and it should be asking those questions continuously and you should be setting what those questions are there are things in your art pipeline that you are going to want to know either from how do you store massive

files to how many vertices does this object have this is what an algorithm does for you it asks those questions so the simplest method of image segmentation is called thresholding and that method is based on a clip level to turn the grayscale into imagery so everything from grayscale onwards is a really good way of clustering i i just put up like a regular algorithm here for mlct so we are using machine learning of the basis of um computated tomography ct so we're using computer tomography which is taking an object and we're scanning it the reason for this is because our clients require us to provide objects especially for vr and ar which are x-rayed so you can see inside the object this is really important because it means that as we grow by 2021 we will

also have physics built on top of this our ct is not photogrammetry related it is pure ct we use photogrammetry and pbr really just to make sure that we have dotted the eyes and cross the t's of the image itself but actually when it comes down to it by 2021 we're going to have physics based um object movement so interestingly for you guys um if we take an an example i'm going to show you in a moment of sushi you'll be able to pull out a squished sushi or you'll be able to squish it squash it break it unravel it unwrap it put it back together and you'll be able to do all of that in the engine by next year using our automated process so while we're going to be constantly improving on how we manage that data you should probably stay in touch with us if this is something that's interesting to you and something that you want to do for

your um art pipeline for sure how do you do clustering for motion graphics and moving image well again this is moving towards where we want to be in 2021 which we're doing right now so this algorithm simply is written based on the behaviors of the moving image and from a to z and that allows us to look at the differences between a pair of images it's literally spot the difference we assume that the object of interest is moving but the difference will be exactly that object so perhaps the background stays static um but the object itself is moving maybe it's a car or some kind of scene and in early trials when they did this they actually used a robot to just keep poking and pushing to try and create the motion graphic from a more physical angle but actually the way that we are now is we can create that in a digital twinned environment um and this is constantly being developed

as i said in terms of our platform we'll have that ready by 2021 what types of things go into the scanning technique the machine learning and the automation we break our scans down into taxonomies states textures materials it might just look like a machi roll but it's whoa so much more we can count how many pieces are inside that piece of sushi we can remove the artifacts that are sitting on top of that nori and we can dive in a little bit deeper into the cucumber funny story there's a company in switzerland that have taken one of our um mackie roles and they've decided to create a rave next year on top of a mountain no word of a lie but if you want to rave inside our piece of sushi you'll be able to do that next summer next spring stroke winter so how do we use a machine learning in

ct this is where the magic happens guys the main rules are don't try to fix that which isn't broken unless it's an object volume graphics segmentation and then reconstruction we follow a bio or med tech process in what it is that we do and that's to say we segment both manually and we use machine learning this allows us to use vgl more accurately for the purposes of reconstructing the sushi from an actual grain of rice so we can build california rolls and mackie rolls on any type of roll that you want using machine learning that just the machine has to know how many pieces of rice and it needs to know whether you're using nori or whether you're using sesame seeds whether you're using cucumber or whether you're making a sashimi and this is exactly how we started we started with this actual piece of sushi we scanned it it became an stl we textured it um we we cut down actually the speed

with which we do this stuff from this mackie roll took about between 15 minutes to an hour to be created and then we render it so it looks really pretty and gorgeous we twin real objects and that's really important to mention i understand that in the games industry you guys you use concept drawings because i've been there use concept drawings cad drawings you find models you use asset stores you do all that kind of stuff technically you shouldn't really need to do that if you build your own platform and you use machine learning in your art pipeline you should be in a place pretty soon where you can get the machine learning algorithms to do that stuff for you here's our deployment platform it's top secret so please don't tell anybody um i know you will but um if you don't want to and keep it to yourself and you want to talk offline i'm available to talk about this a bit more we know that ct data is hard and so we

realize quite quickly that it doesn't have to be we've got tools and standards parameters and decision making in the initial test environment before we go automated we create a digital twin of what the digital twins will become so we've already created an environment and we know that it's hard but basically here is how it looks from my designer head the raw data goes in we pre-process it by asking the questions we treat the object by actually making those questions like be asked of the object in the algorithm then it post processes out so that means it reconstructs and the object asset is formatted to do that we need to do like a series of bits and pieces everything from literature search pre-processing and the mathematical treatment of the computer vision itself in the segment we have to prepare the object

so when the raw data comes in it comes in raw as you saw it's raw like sushi literally so we can dive into it a gazillion times as an object but we can really only look at segments as i said we use the bio and mech sorry med tech version of what it is that we're doing so we break everything up into little segments and eat those segments in our machine then we put it back together afterwards and also the otsu method though it's the simplest the algorithm returns a single intensity threshold that separates the two pixels into two classes which is the foreground in the background that means that we can render the object quite quickly and we can cannot render in the crack cloud at the moment we're not doing that and the reason why we're not doing that is because our objects are of such a high quality we have a pipeline that comes online next year that we're not allowed to talk

about so if you do have hard questions for me at the end of this there are going to be opportunities where i'm just not going to be able to answer those questions i'm so sorry but we're coming to the end of the presentation now you'll be pleased to hear so you can all go and take your lunch here's some helpful things we looked at the prague online texture segmentation benchmark you should do the same if you want to build this the berkeley segmentation data set and benchmark is really important slicer is great for taking objects and breaking them up for semantic segmentation if you don't know a damn thing about machine learning go and look at semantic segmentation for dummies it's amazing and then that robotics motion graphics slide i was talking about you can have a look at robotics.tu berlin and um that will give you some interesting white papers on how motion graphics can be handled in machine learning for your art pipeline so i'm ready to take some questions now if people have got some

and i thank you for taking a few minutes to listen to me harp on all right kelly thank you so much um that was uh well i have to say i need to study a few of those links that you shared um because as you awarded it i don't know a damn thing about segmentation than that but i was also very grateful that you had the uh the example of the sushi which made it all sound um you know for someone who doesn't work with this in a day-to-day basis it helps to like put it a little bit more into into perspective um so yeah um thank you so much uh we currently do not have any questions because everybody's absolutely blinded by the science of the future i think so i think that's what's going on but if people do have questions later i'm going to be on um the machine learning panel along with the guys from unity etc so if you want to just go and digest it

over lunchtime and then you think oh i've got some questions i'll be back online this evening i think it's 6 30 right um yeah um i don't have the full schedule here um but uh yeah i mean there's you know talks running through throughout all the afternoon and um yeah let's be uh yeah i'm trying to i can't look it up that quickly um but yeah it would be great absolutely and uh i i you know definitely invites uh all the other viewers here to to them join you uh on you know listening to that panel as well and um and ceo what other people also have to share on it there's one question if anybody's got any questions like outside of here as well they can just reach out to me on linkedin or you know you can drop me a little line and kelly h and just ask me any questions based on the panel and the talk today i'd be happy to to share anyways um if you want to write that down you images you can also share it in the chats next to here so it'll make it a little bit easier for

people um and of course there's the discord channel so if you're already in the discord people can chat to chat to you there as well uh we did have one question coming in the meantime from baron harris um he said hi thanks for the talk i was wondering how much data do you need to generate an object i think it that's a an interesting question but it totally depends on the object as i said things have to be segmented so they're segmented in various forms so in terms of data you need to look at things like the object structure and the taxonomies those taxonomies can be broken down between maybe four to 16 variants so that creates quite a lot of data then if you want to add more data on top of that like vertices etc that's even more data so you're effectively doing that um sorcerer's apprentice thing where you're just chopping a bucket in half and it just keeps like reforming so

set yourself like um the parameters of what you want that object to do for you ask that question and then build that data from there all right great thanks so much um well no other questions came in in the meantime um but yeah your email address is shared um so yeah people can you can follow up um then i'd like to thank you so much for for your time and for the interesting uh talk uh and we'll be seeing more of you today winter yeah look forward to it thanks very much for your time guys and uh have a great day you too bye