This Machine Has No Brain, Can It Borrow Yours?

[Music] my name is omar i'm from nvidia um i'm here to talk about uh something that i've been working on for the past three and a title half years talk is this machine has no brain can it borrow yours um does anyone have any idea what this is in reference to has anyone seen these signs before irl this is actually from fallout but um these these signs are whenever whenever you have a machine um that can kill you uh this sign usually um is in some form next to it um and i'm here to give a warning to everyone because we're facing a crisis we don't yet see uh in about 10 years from now um we're gonna have robots everywhere um so we already have some form of robots in our personal space right we've had um for example cleaning robots hang out for the past 10

years um but now we uh you know we're beginning to see some changes in the effect and the amount of robots that we have in our personal lives or doing logistics for us and that's mostly because uh we have amazing reach nowadays right we have um innovations in motors and um in uh in microcontrollers that can do stabilization we also have innovation and dexterity so uh grippers that that uh used to be something that was very special specialized are now uh we can now manufacture them uh pretty much in mass manufacturing even for like the most uh specialized tasks so you would have um rubber uh and silicone in uh injection molding which is a mass manufacturing technique but now it's very common in robotics so you can do soft draw robots but you can also have um

uh robots with tendons um that have a very high gear ratio so over here you're looking at a robot from the cornell organic robotics laboratory that's holding a can right so it's very accurate for a machine to be holding a can without crushing it and this is done pretty much with custom motors so this is you know like we're definitely going the direction of precise machinery and we also have ai um as you well know and the infrastructure is also going uh pretty much to personal personalized robots we have data centers at the edge which means uh you might have a data center in your uh next year traffic light or just by the base station of your mobile phone so the economy is pretty much not going to allow it to be anything else we are headed towards a completely autonomous future in several fields and this is a problem this is a really big problem because

what's missing in this chart is that we don't have good design for any of these robots so no one's designing their behavior and if they're going to be interacting with the elderly in healthcare with food preparation with delivery and logistics they all need to be designed to work around to work around humans and so far it's a hack right we don't have a stack to actually use robots interacting with humans and we need to think about how we're doing it and my proposal over here is that simulation is the right way of doing this um so let's start with a basic question i know you all work in vr but how many people here know these images okay these were the world's first ray traced images they were created in 1979 uh by turner witted it with that was a paper that he released while he was at bell labs um he called these uh very imaginatively figured six

seven and eight um they were part of a paper where he's describing also how ray tracing works which he described in figure one figures one and two um one is a description of the actual algorithm the other is a description of snell's law which is a way to describe how light moves through different media we haven't gone to simulate actual optics and therefore have so sort of like photorealistic rendering until very recently right um if you look quickly at the evolution of computer graphics we you know had our first um popular 3d engine that can run in real time in 1993 on the other hand we had our first ray-traced film in 2006. uh well if you ask the people at pixar they would say that a lot of the ray tracing there that i would consider fully raytraced was a hack i consider it fully retraced they actually managed to simulate a whole model of light um but something happened

along the the way we have this now right and with uh with gpus we can suddenly do things like create game engines with which approach uh photorealistic simulation in this case we're looking at a buoyancy kite demo by epic games which did a lot of photogrammetry around the environment but nowadays when we've reached a rage racing in gpus we can move between what is basically this all the time to this right i'm gonna do that again because i love that apologies to everyone um yeah but what this actually means is that suddenly like we have we've achieved several levels of abstraction throughout the years right we started out by being able to only make games via basically assembly language and then slowly evolves into what we have now in game engines right like unity and unreal um but the same thing happened in

graphics in general right if you worked in a film in the 1980s and 1990s you'd be doing everything with command line interfaces now you're using photoshop and mayan nuke and this also happened in robotics because um when you're working when you're working on robotics in the 1970s you were building all of your own circuitry uh around the 1990s you started having microcontrollers that were very expensive and hard to program and nowadays the stack is almost completely solved for you you have operating systems like ross that do all of the all of the robotic control um on one hand on the other hand you have very good hardware you have gpus on the robots themselves some companies make them i'm not gonna name names and um the underlying idea behind all of this is that from engineering from well a culture that is supposed to build machines for other people to to have ideas on top of we now have design frameworks

and from the idea of modeling a well making a mathematical model to uh to solve a problem we're now just teaching computers to do this right so in a way you can talk about machine learning being the abstraction um to how you design human robot interaction so i want to talk about machine learning a little bit so we're all on the same page um i think this is the the first interesting component of um of deep learning um that was ever introduced that this is this is i think this is the first paper um from stanford in the 50s about neural networks uh at the time they they could deal with one um basically one solver which had a bunch of inputs and would output a sort of like a linear result is this above a plane is this below a plane as a result you slowly evolved into these right so you can have multiple um layers of of many input nodes

and um because of gpus we can we evolve to this this is google net from 2006. um no from 2016 i'm sorry 2006 what happened to me all of these contain subnetworks which contains the perceptrons from the previous slide this is a very very big network the problem with these is that there is an underlying thing that fuels all of them it's data right and we have evolved to needing data everywhere so if you are building any kind of system which has which takes um resources from the real world and makes a conclusion about something new it is using some form of data and there's a market for data as a result because uh everything that used to be modeled is now learnable and learnable with data that doesn't necessarily belong to the company that made it

and if you're using social media you are also part of that market right you you are being used as a example in learning this is a this is a photo from my house just before i left here i i just got a new sofa last week so i just removed the old sofa you don't have a sofa in this photo this is from a camera that is connected to the internet and it helps me know if someone's at home um but along with powering this camera i signed a a user like a eula end user license agreement that forces me to give up all of the data from that camera and help the company that makes the camera train its algorithm its models on um whatever it wants actually so it can sell my data to other companies it won't release the raw data but it can use it

for training right i i'm i'm not i'm not extremely happy with this um but the reality is if you buy a camera attached headset or if you buy a camera attached car which you are doing um they'll have the same kind of end users license agreement like your data won't belong to you and your data will be used to train newer models for uh the future of robotics there are multiple problems with it but i think the biggest problem that everyone is ignoring is that no amount of data is enough when we look at data that's coming from users we're looking at the normal behavior most of the time it's very hard to catch anomalies anomalies don't appear in the data set or when they do appear they appear very sparsely right so if you're trying to train a self-driving car you need to train it for stuff like this

that's not an image that you would typically see driving down the street unless you live in new york like i do um this is a very hard image for a neural network running inside a self-driving car two parts it has multiple lights it doesn't know where the sources are this is a rare occurrence like i would call this an anomaly but anomalies exist everywhere in the data right you're not training for this you're certainly not training for this right so some data sets don't exist right now but some data sets also can't exist it's going to be very hard or unethical to get the data sets in advance for machines to actually use when you're when you're building a product that requires um a mathematical model running behind the scenes so i have a solution this image isn't real um it's part of our uh

it's part of our uh simulator for self-driving cars that call drive sim um we we've been using this for quite a bit at nvidia uh the pro the premise of synthetic data seems reasonable there's no data in the real world you just have to make it right and there's two things to attack here one is generating imagery enough imagery for ground truth to provide data for um for the networks to learn if you were in a scenario where you have a lot of red lights or if you're in a scenario which has complexity in the road um but the other one is generating interesting scenarios creating images like this one um pretty much at random solves for one it's also for the ground truth it doesn't really solve for a scenario so we have to think about how to create scenarios for robots to learn how to behave um when it comes to things that are hard

to capture in the real world this is one example of something we've been doing this is a bunch of robots together in our in a product called isaac these are all simulated and they're learning how to um how to play hockey right so um all of them just have a stick and they all they know how to do is just move the stick and they also have a score function which tracks the puck and tells tells them how close they are to the goal after enough iterations those robots learn that they get a better score if they move the sticks such that it hits the puck and they move the stick such that it hits the puck on the way to the goal and when they do they get a super high score so that then they know how to repeat that again and they get all the observation from the real world um and just use that to make that one specific uh action now if you think that's strange um you need to get used to it because that's of how all of robotics works right now and that's the most intelligence robot will be for a while

unfortunately what are we actually doing here that hockey thing that i showed you that's the environment right your environment as a robot playing hockey is a field there's a goal in there there's a puck somewhere you see all of that and you as an agent are looking at the environment you're making an observation after you've made an observation time to make some action right so you'd be moving the puck it doesn't happen just once it happens at very very high uh frequencies right so you you'd be moving the puck a little bit then you've you're observing the world again it's my sorry not the puck the stick uh is my stick too fast is my stick too slow until you've reached a good speed and you've reached a good direction it's time to hit the puck and you've hit the puck and stopped and if you got something good out of it well what happens actually there's something missing here right someone has to tell you that you've gotten that you've done something good so there's also an interpreter in the process and if you've done something

good they'll give you a reward so this is a the basic idea behind reinforcement learning and if you um look at any kind of reinforcement learning paper you'll see that there's always some you know system on top of this but this is the basic idea there's there's an agent in the world they make an action and um and they they get a reward as a result now i want to ask you how many parts of these syste of these systems can we debug right now so um if you would be um a reinforcement learning uh expert what you'd be debugging is the interpreter um you'd be looking at uh what the robot actually gets rewards back from and you'd be tweaking that reward function right get closer to the goal cool maybe maybe getting closer to the goal isn't enough maybe you need to hit at high speeds so you tweak the function a

little bit so that the stick uh that's actually moving um you get a higher reward if you hit the goal and it's a high speed right you could be for example saying as long as you got close to the goal that's already high score so do that um so there's a lot to tweak there to debug something and that goes into the agent but there's something really big that you're not debugging that way and that's the environment right so a lot of things happening with the robot when they're interacting with the actual world you can't solve because you can't change the environment too much you don't have a debugger for that so we made a debugger for that this is the first example of a vr experience that um that i created um this is for siggraph 2017. um what's happening here is you're playing dominoes with a fellow called isaac this is isaac um he has domino's i have dominoes i'm going to pick up a domino on my side and i am going to place that piece of domino

on the table in front of him so as i do this i pick this up this is done with a with the ue4 it's already two and a half years old so it's been a while um so i'll place this piece domino it's got a four and a five and now i'm gonna hit the button in isaac's neural networks what's happening is that he needs to evaluate they need to evaluate the type of domino they need to evaluate what the legal move is and then they need to act on it and in this case isaac connected a five and a five so it's a legal move great and it was almost geometrically legal as well so good on him uh now i put a six and a six together so he has a few more options on his board um so he's going to look at it again and perform the same action this time he's taking a what is that taking a foreign connecting a ford f4 okay so what's going on here is uh this is a very controlled environment in

which we can check or we can intervene with the environment for the robot to play in right and i want to talk a little bit about how we did that so the first thing to do is to train the vision network for the robot we generated a lot of images of dominoes so this is also done in unreal engine we pretty much simply threw a bunch of dominoes in the space and we attached to each image as we outputted them uh the ground truth right what is this domino and the neural network can then learn um what uh what number and and where is the domino and we also added a bunch of distractors so um the uh the neural network wouldn't pick up on everything being a domino now you'll notice that these images are not photorealistic and this is another very important point we don't know we actually don't know um what what neural networks consider important right this is a research topic um so we start from the very basic things and we slowly improve upon them the next

step after we train the vision network was to just take a very very simple agent and to play a lot of games with a robot we played about something like 50 000 games until the robust reinforcement learning network learned how to make the right moves and then we debugged it in vr so at every step that we thought that the robot was good enough we would play a few games in in vr and now we wanted to do something else actually at any point in this process did we assume that the robot was virtual we didn't right so if this can work on a virtual robot it may as well work on a real robot that has the grippers and the right uh physical configuration right do you think it's gonna work yeah it works so this is uh this is a robot that we actually deployed in uh siggraph 2017 um it literally is running the same networks when we first put it up

um nothing worked and i was really upset and i then realized i put the camera 90 degrees off so i rotated the camera and suddenly everything worked because the robot just had the same vision network and it had the same gameplay network so i could just play the game the only thing that was different about it was that in this case uh we were not using the same inverse kinematics as in the um the uh control system that powers the the grippers um but everything else was just the same so there is no difference when you're actually getting it right between a physical robot and a virtual robot and this is a very important takeaway here right so um here are the goals that we set up when we when we started this we have to use vr interactively with machine learning um most of the team that we had weren't roboticists at the time i was not a roboticist um we can't use end-to-end training we have to modularize the entire system um and we have to do it really fast in

fact we we had two weeks to do that um so building the system was a challenge just just from a time crunch perspective the way to solve it is we use the best component for each layer right we started with a um just the uh the image recognition system for for dominoes um completely separately from that after we assumed this works we had separately trained um a reinforcement learning uh network on the gameplay and completely separately from that we were testing a physical uh control layer or a virtual control layer for the robot right so at the very end of the process we were testing both vr and a real robot at the same time the reasoning for this is that we can debug the virtual robot really quick and the physical robot is a little slower otherwise it'll kill you pretty much um

so here are the tech stats on this just to so you want to know that the production itself was two and a half weeks uh we had separate teams uh doing each part of the com um of the process there was the vision the policy policy is is how the gameplay is played uh the physical robot the hri and the art i led the um the hri and the art and the virtua and the vr um and um we trained a vision network with uh 30 000 domino images with our own um our own simulator called isaac data studio the policy network used i'm sorry i thought it was 50 000 it's 80 000 uh games to get the robot to the right position i tested it after about 50 000 games um it wasn't good enough uh we also had to redo some of the components at the time this is about um two and a half years ago at the time um the training was quite quite slow as in like it took 24 hours to play

uh 80 000 games um which when you're in a time crunch actually matters so testing it early uh virtually really mattered to us um and the components were never integrated until deployment right the other thing that's really important to say here is that all the teams were separate i was in my home office in new york um some of the uh uh ai people were in toronto and um some people were in austin and the physical robot was la i think not sure um so so we were all completely separate and got it and we could get it done right i'm going to skip this so uh the other takeaways here is that we built this in such a way that people know what to expect when they're testing the robot right this is not a smart machine so far one day we'll be at a point where this is a smart machine but so far it's kind of like a five-year-old child learning how to play dominoes so you

have to be patient right so by putting this inside the story people wouldn't make smart moves at first we can actually debug using the simple cases um if you're in vr you get bored right uh but if you're in vr thinking you're playing against a five-year-old child you get bored less quickly because you're the what's going on in their head right is this a good move is that a bad move um so that's really important to keep in mind when designing things like this um the other thing that that's that we realize is really important is that players don't remember um if you tell them oh just just don't put like um just don't put one of those tiles uh sideways because that's that's a move that's an advanced move but they will remain remember game mechanics really well so you know we've been playing games all of our lives so we we get to know what um you know a game of dominoes looks like and we kind of listen to that we tune into that um we don't really tune into instructions like oh no no just don't touch that right but if you

make something in the game that makes you not touch that really helps in the simulation um obviously i talked about it when you modularized all components we could really speed up because we weren't dependent on each other at any point i could generate fake data in the first week um and have everyone test with fake data and then i could bring in real data and replace the fake data and it just worked right and the other thing we realized as we were doing this what we're actually building is a debugger right so we need the ability to cache previous games and we the ability to switch between and a later version of the framework to an earlier version and test which one is better right do a b testing on that so using this knowledge you can actually go very far um you can train very advanced networks to do things like self-driving cars these aren't real images again what this is is um output from a neural network

looking at synthetic data that we have inside drivesim our self-driving car simulator i want to talk about it a little bit just so you know what we're actually building when we're building uh gameplay environments for robots so in this case we have um this car model it has a bunch of cameras all these cameras need to inform the car on how to make decisions in an autonomous driving simula situation right so cars would be driving around and they get some anomalies right um so we need to emulate all the sensors in the car and we need to recreate dangerous and rare scenarios do that um and so we can manually inject inside the environment we can inject fox no rain and we can make some some arbitrary traffic congestions uh we can add really backlit scenes with fog um so so the sensors barely do anything but they can still react properly because we train them on many extreme cases and we get this and we also apply some distributed computing to this because we don't have

to train one scenario we can train a bunch of scenarios at the same time and kill them with the ones that we don't like right the ones that we don't like didn't perform well in like a rainy street um okay redo that right we'll we'll use another um and this this approach should work right do you think it works yeah it works um so this is a test that we have from um about a year ago yeah it's about a year ago uh we did we did a test with um one of our autonomous vehicles uh this one is in um in the south bay uh we're running a track of like 80 miles um and this is real data that you see here so so what's what's actually happening is the cars driving around and it's picking up on all the road marks and so on in order to validate this information what we're doing is we're recreating the same track virtually so you see on the left here you see drive sim drive sim is our self-driving car simulator and it's using all the information um to recreate that ride and validate it

to see that we can inject more dangerous scenarios in that case maybe the car wouldn't make it so we want to validate all these we add this layer to it right we're just changing the environment and we're not telling a real environment from a virtual environment as far as the evaluation network is concerned because they're not different so the layers that we have here are simulation on one hand emulation on a set of sensors on the other hand and at the bottom we also have imitation we're adding more data for the the the machines to be able to use without exactly understanding what a human is doing or what um you know other players in the driving simulator are doing but still be able to do something with it right like if you if you don't know why a machine has done a certain thing but you're still able to navigate around that well mission accomplished pretty much the same way um this is one example of what i mean so

this is a uh demo by a colleague of mine named madeline gannon she built uh this playful robot now this is a this is an assembly robot from abb robotics this thing can definitely kill you um it's very big and it's very powerful and if you get too close to it um just one wrong move can break a few bones um but in this scenario she made it look cute and inquisitive right mimicking humans in a way and this uh so this this field has grown quite a bit in recent years this is a work by daniel holden from about a year and a half ago or two years ago uh he made a neural network that uh combines a bunch of different motion captured uh humans uh to recreate motion and over here what you see is that this this character is getting a bunch of scenario terrains that it hasn't seen before and still reacting properly because a neural network is mixing all of the different um terrains that that uh were available in training time it's just doing it intelligently

so the thing looks like it's acting reacting pretty normally the same university also released about a year ago um pretty much the same type of network with some modifications but done on a wolf right there's some improvement there also you see the motion is very very natural please don't ask me how the motion captured a wolf we have been doing the same sort of things only with reinforcement learning so over here you have isaac gym it's our own training system just for reinforcement learning where we're training virtual characters to just get up and um tap each other um there's definitely a video game in here somewhere um there's no mocap here just physics and um we can do this with imitation right so one character here the uh the the physics simulation that you see on the left it has all the characteristics of a human body so it has the biomechanical model of a human body and it's observing another motion

captured clip and it's trying to imitate it so what we're doing is we're providing real world data real human data into the process and then another machine is just imitating that and it's getting a good score for doing it right right it's not actually using the motion capture data it's learning to activate its own actuators and motors to move like the motion captured human and it's getting a result of is getting a score on the end result not on the joints right is that clear cool and we can make mass games with it so in this case we motion captured a bunch of um a bunch of different positions and we also inserted some some synthetic data into it in this case they're trying um to do backflips over here they're trying to do some mass games and you see how this is this evolves into the training of the character in a video game so uh next time you kill a monster in doom i want you to feel a little guilty um it's done so much for you okay so um so now we can create

imitation based data sets and we can actually do useful things with them so this is one uh other vr experiences that i created in which you're using the same robot that you had before but now you're making pretzels so in this case um the user gets uh gets uh the chef's hat and they get to transform themselves into the robot and control its uh actuators control scripters and and they'd be folding a pretzel so they're grabbing the grabbing a pretzel from both ends this is sticky dough uh it's it's also soft so you have to shake it off your hands you'll notice here something strange the robot doesn't have the same joint structure as we do so we need to think about what it looks like when we make a motion and sometimes we're wrong and that's when you see this this thing turning red like this virtual gripper turning red there's the real world which is the that virtual gripper and then there's the virtual world which is where you see the robot that was a legal move so this uh simulator is

only tr is only made to make legal moves inside um inside the vr experience and whenever i whenever my hand is green i can do this now i ring the bell when i'm done and now you have the robot imitating my exact motion so after i've recorded it the robot is going to try again and they're going to fold the dough which is fully simulated here the same way that i had just done so this is training a virtual robot to do a completely manual human task and now that it's done i have my little helper robot do a salt bae move here i'm this is mind you this is a year and a half ago i was very proud of it at the time um so now he's gonna bake the pretzel everyone's happy um i think this is uh a very significant process in um that's it's going to be part of

the uh of the future of labor in many ways we're going to train robots to do the small action that we are so good at doing but hate doing right it's gonna happen over and over again so i want you to notice a few things that i that we found along the process that might be interesting so one is it is totally non-trivial the way you choose um to control the robot we control the robot from first person because we thought it was interesting enough to make it such that um you would learn how the robot um how the robot can move from the perspective of a human right some some moves are legal um but the other options that we tried are grabbing things by the object so instead of grabbing the actual gripper i could grab some virtual point away from the gripper where the actual object the endpoint should be and have the robot attempt to grip it that had limitations we also tried standing at a distance from the robot so we can see what's going on and complete the task from our perspective because there's nothing

actually limiting the robot from doing the entire task upside down mirrored um we just happened to be piloting the robot from from first person we could be piling the robot from third person and it would be completing the robot the same task from our perspective so there's no difference really we try that too at this point you see this kind of like preview where there's a bit of a rope between you and the robot doing that um we also tried following the actual gripper with a controller that turned out to be weird it has definitely has some advantages but you still run into the same problems of legal moves as you had before from the first person perspective only it's harder to track and we also tried doing that at a distance right um i think the main conclusion of that is just don't don't make controls that are special for the game because you'll end up making them anyway and as a result um of thinking that way we ended up making the actual controls that we

needed so we had uh visible motion bounds for anywhere that the robot couldn't reach we also had the legal state indicator to see when the inverse kinematic system that solves for where the robot should be actually doesn't hit a singularity or doesn't hit a motion bound we also placed all of our scenarios inside the environment so people can get used to doing the action inside where they should be but that wasn't enough because actually folding a pretzel itself that task requires some training and so when people started doing it we we thought this would be enough to train them right by just showing them a how-to somewhere in the game but it wasn't enough so when we actually set this up we had a rope on the table so people can train on the real material and this really matters because when you what you're actually doing when you're inside a simulation is you're trying to use all your proprioception your natural senses of where where the world might be and um you'll end up relying on some

defaults that suck um so by having uh good intuition to what what you're doing this is helpful um the other thing that's super important uh when building a robotic simulation is to make sure you understand the difference between you and the robot in this case we had or that couldn't cross um its arms right so this motion was illegal whenever i was would fold a pretzel like this the simulation would fail right and the arm would actually stop somewhere here because there is a motion bound um however the robot is more robust than us in other cases for example there's no limitation that the robot could fold the pretzel when it's facing outside um so make sure you understand what the limitations of the robot are and make sure that you understand how fast the system can evaluate alternative cases in this case the reinforcement learning system can

search in a very limited space very quickly but as soon as you move outside that space it's actually not intuitive so doing things like going above a hand is not something that a robot can intuitively do you have to make it very very specifically perform the one task um yeah and you have to validate your assumptions other takeaways that we had from that users will defer default to illegal states so um you have to tell them that that a state is illegal we used like the red grippers to do that uh we also used motion bounce to do that uh i saw some people to do something like ah i don't want to do this and then a hand would be stuck over here because the robot couldn't solve moving the hand from here down back to where it should be the scale and the torque of the grippers and also the mass affect the user behavior so you really have to have visualization and feedback in place

they never got it right the first time whenever we tried it so you have to make tutorial levels like we did and the last thing super important that we didn't implement and i really want to implement is making the last action that you did easy to undo because you are performing an action on a robot that's affecting the environment as far as you care it's real okay so last slide i swear um when you when you're building human in the loop systems here's what i want you to think about make the system modular the more you can replace stuff in it the quicker you can move you have to state clear goals because any system that you're making that you're making that's human in the loop has people from different disciplines if they know what you're trying to solve they will work around what you can't solve okay so always communicate well between people and the team avoid false empathy i just covered that but it's really important to remember and remember that you're building a debugger right you have to expose

parameters you have to make agents interchangeable and you have to make states rewindable and remember that you're building a debugger and you're also building a game so you have to exploit all the games mechanic game mechanics that you can you have to move as fast as possible thank you questions all right oh question yes i'm sorry hi how do you come up with all of those uh test scenarios so basically it's it's a test scenario that you're pushing through um the whole process right um so i don't i don't necessarily think that i can call every single every single of these situations um a full test scenario i we were trying

to validate if a robot can complete a certain task right so we're like what is good for a human to perform that a robot can copy or um what is a simple task in the case of dominoes what is a simple task that we can play with um is simple for a human to understand and um uses the limitations of a gripper that the robot has we we pretty much we brainstormed until we had something that works there's no there's no guidelines except for well user creativity i guess i'm sorry if there was supposed to be a higher insight there no just use your creativity hi um i just have a question when you were uh showing the tapping isaacs and they were imitating the motion capture files um you said it's important to note that their score was not based on joints but their overall performance i'm just wondering how you scored that yeah so so

um there are many variations on the um on the reward function uh but the end result is overall pose so you can you can think about overall pose in many ways one result would be to project from the perspective of the camera and get like you know a least squares approximation another one would be to test the end effector positions but not the actual joint hierarchy because you might have you might have a small offset in the root joint but still down along the hierarchy there are some corrections right if you were to judge joint by joint um you'd create an overfitting in the system that is going to be hard to overcome in later steps so um there's a preference in some composite score that's undefined i'm i'm aware um but some composite score that would be more helpful for um uh letting the machine be more creative later on i guess that makes good sense i've done the same thing in dance

um okay uh so i do a lot of simulation work as well and one thing i find is with a sort of the old guard if you want to call them that the physical engineers there's a little bit of skepticism and pushing back i'm curious if you've had to deal with that and how you've dealt with that [Music] i think the proof is that we're getting results a lot quicker uh so i i have no pretense when especially when it's like uh 20 or 30 year experienced roboticists uh to to claim that i am um in any way improving their uh their techniques like improving upon their techniques i'm producing a new perspective that they couldn't do up until now because tech wasn't ready um vr just lets you do that quicker and you know like game engines and simulation frameworks allow that allow you to move very fast

and they they never could so um if you present it that way it's a nice carrot and suddenly your skills become complementary and they uh they they begin inquiring very quickly oh and how do you do this and can we also add some stickiness to it and so on so i haven't found anyone who hates me for doing this so far um yeah sorry i just had a quick point of clarification in the dominoes example uh-huh you said there was 80 000 games there were 80 000 games uh played in succession to train the reinforcement learning but not all in vr no no no no um so we had we had a scripted agent uh make legal moves um and we had a reward function uh reward um legal moves from isaac's side okay um but the the scripted network was dumb right all he could do was play legal moves when it couldn't this

mission would just stop right so very early on you'd get punished for making silly things but slowly uh you'd converge on first placing a domino next to another domino that's like the first thing and then actually converging to make a legal move takes some time i'm sorry you got cut off so it's a little bit i guess unclear to me in that example what the what was the vr adding that you couldn't have done with your standard scripted methods yeah so the scripted method only examines um the scripted method only examines what is a legal move to make in front of the robot but what you're actually trying to uh teach the robot to do is to play a game of dominoes like humans would play them right so i could be placing the domino not exactly in the right place and i still want a legal move out of it and i could be placing the domino sideways and the robot would then know how to do something like

like that was that was a one like interaction that we had just like say that you don't have a legal move um the other thing that we're trying to do is not to make to make the robot not make extreme motions outside their their uh their motion range so all these things have to be trained in a network that breaks the rule in a scenario that breaks the rules and so the last thing is then so how many vr games were part of that training set for the domino example so you put the number of scripted games and i guess that's what i'm trying to tease out here um [Music] all together probably low hundreds okay um maybe maybe like 100 150 okay yeah okay thank you