How Art Was Used to Create a Unique Cinematic Experience in Heavy Rain

[music] Okay. So, welcome everybody. It's very crowdy uh and uh I'm very happy to show you an overview of the hard direction and the production pipeline we use to create uh heavy rain. Uh being there talking from this side of the desk gives a lot of emotion. And uh it's quite a good thing because at Quantic we really love emotion. So the title of this session is how heart was used to create a unique cinematic experience. Uh so just remember few words of this title create a unique experience because when we start every rain it was really in our mind not just making a good game

but creating an experience sometimes it succeed sometime it's fail so uh First let's talk about uh Quantic. We have been established in uh in 197 in France. So I think that uh my accent speak for for myself. Uh we are 100 full-time employee uh and we have a in-house uh motion capture facility, a 3D scan and a sound studio. We also uh develop uh experience based on emotion. As I already said, every rain is a new format. It's called interactive drama. It's a blend between uh film and game. It has been released last year exclusively on PS3.

So, who am I? I don't even know myself, but I can give you some key points. Uh so my name is Christoff Russo. I'm the graphic director of the studio and uh I start in the old garage game uh time. Uh in a company called Delphin Software International uh they do flashback. So I joined them uh for the sequel. Thank you. Thank you for them. uh and I work on uh on motor reserf on PC and p PS1. I left the company to move to the movie and commercial industry and I came back uh to video game uh as a lead set in uh 20 uh 2001 sorry uh and I work on Fahrenheit the indigo prophecy in US uh

and uh on a rain. How many people play uh to every in this room? Oh, okay. So, uh I will just show you a quick trailer uh for people who don't play every rain. Are you prepared to suffer to save your son? You have 5 minutes to cut off the last section of one of your fingers in front of the camera. If you succeed, you will get [music] your report. Agent N Jaden, FBI. Mimi, [music] my name is Scott Shelby. I'm a private detective. I'm investigating the case of [music] the

Oregon. Hello everyone. Page 27. [music] Single Ethan Mars has had psychological problems since his first son died. He is haunted by visions of drowning bodies [music] a few weeks ago after one of our usual sessions. I found this on the floor. He must have fallen out of his pocket. Ash, I want you to assign every available man to finding Ethan [music] Mark. I want you to keep an eye on the train stations, the airports, the bus terminals. I want every cop in the city on his ass so that if he moves, we know about it. [music] You're going to tell you something. I swear on my son's way that I would kill [music] the man who just don't believe he's guilty. Your other son is a shepherd. [music] What was his name? Everything I did, I did for love. So it's a little bit darker but um

uh let's uh go uh let's talk about uh few main topics. Um first uh the story after that uh how we create uh the world of every rain and the characters uh how do we animate them and the fifth uh main character. So what is the vision behind? First of all, we wanted to create an experience that engage the player emotionally and explore the potential of uh interactive storytelling, but we are also targeting a much more mature audience. Every rain is unusual. It's not based on challenges. There is no boss to beat, no

solving puzzle. There is no game over. You can lose one of your main character but you can still play the game and continue to discover the story in the way you choose. Every rang uh story is a multiple path story. So we are to write it and also to produce it all the different paths. So some numbers because everyone is a massive project. 2,000 pages of script. It's very huge because we have to write all the path, all the dialogues, all the game play, all the mechanics. It means se 70 unique set, four main playable character, 20 secondary additional many days of moap shooting.

Hopefully we have an in-house facility so it's much more easier for us. Many body animation, many facial animation, a lot of localization and 60 scene. So every rain has a 2,000 page script. We start by reading it and believe me it's a huge task. We we [sighs] we do an initial list of mood boards to uh identify some specific places uh directly taken from the script. We do some visual re research um to identify a city a place where uh the story can take place.

So it start uh with some sketches picture some mood board maybe some K color board some rough art concept on some uh specific scene here it's the crime scene I don't know if you remember but it's a huge scene in in Everane. And um also uh color board just to handle the the feeling and the and the look and feel we wanted to to to to make uh on heavy rain. We identify Philadelphia as a potential city, but we wanted to avoid uh doing a French postcard vision uh of this city.

And believe me, it's hard because I'm not an English fluent guy. So, we use a movie scoot uh to tour to Philadelphia. We take a plane and go scooting, but we don't stay in the center of the city, the historical center. It's quite pretty, but it doesn't match uh to the script. We try to go deeper to the suburbs, from the middleclass suburbs to the poor area. The main goal was to avoid having a a kind of ideal vision of this city. So by taking pictures uh we identify some visual grammar but I

will talk about it later. many picture and sometimes some dendrants uh meeting because uh on a on on one specifically in this picture I remember that uh we we were taking picture with with a team and the the the scoot say go in the car, we don't notify that someone is coming with a gun and asking why we were we were taking some picture of his district. So I get in the car, never left it, and uh we do all the the scooting and the the

scooting by the car and from the car. No, I'm kidding. Just just for this part. So we also went to industrial area trying to find a warehouse whole power plants uh peers we also visit uh people living in Cape Code we go to a police department hospital office newspaper I can't put uh images in my slide because there is some privacy NDA but uh uh we are we are very proud to to to to be able to to scoot this kind of uh uh location because um without a scoot uh it couldn't have been done.

So we came back uh with a new understanding of a city and sometime emotion. We have a huge documentation with many picture, many video references. We also identify what we call a visual grammar. I mean uh I don't know if you notice uh in the picture but there is many warehouse bridges electric pole electric power plant uh many things very linked to industrial um I mean past industrial. So we took this element and we try to dramatize uh the vision because Philadelphia is just an

inspiration because is a blend of game and movie. we decide to uh recruit a design team uh very close to u the movie industry. So uh set designer and architect are in charge of creating uh and building realistic location at human size and very precise blueprint. Interior designer and researcher provide a precise vision and documentation for the production team. Set concept artist in charge of illustrating the vision and providing final art concept and specifically light setup and the CG artists were in charge of giving a virtual life to all these uh visual re

researches. So their goals first creating realistic location at human size. What does it mean? Uh usually in video games uh all the location and spaces are very large huge like like stadium because you know we need to avoid collision issues and camera issues. It's easier but we don't want to do that because we really want to be close to the to a movie. So we can't afford that. It's also uh very hard to keep a realistic dimension because visually uh when it's wrong you can see it very quickly.

We also uh find uh visually and strong atmospheric location specifically by scooting. We also uh have to ensure consistency regarding characters and gameplay hotspot because we are talking about a game uh close to to movie. It's a blend between game and movie. Uh we need to match uh location and characterization, but also we have to manage uh all the the game play uh hotspot uh specifically uh to be able to um to match to the main mechanics and the plot.

We provide K color on each set and seal. Um for people who play to heavy rain, uh there is they can notice that there is a huge gap between the beginning of the game and the rest of game. I mean uh before the accident and after. Before it's very bloomy, colorful, very clean. It's a kind of video life. And on the rest of the game, every uh color is color wash. It's rainy. There is few saturation. It's very diffuse light. It's chaotic and dirty location. We do that just to improve the feeling of the happy time versus the bad time. We also provide invisible storytelling.

So what is invisible storytelling? I imagine the first time you go to somewhere else house when you enter a room you can identify many thing at the first sight. Maybe could be a huge amount of books or comics, baseball stuff, domestic mess, TV set or no furniture style. All these elements provide you information on the people living here. This is what we try to do in Heavy Rain. We don't want to tell the past of a main character during hours of cutscene and dialogues because remember we had 2,000 page of script just for the main plot. So we try to provide visual background

small pieces of their life. And uh also we ask the artist to make it stunning but depressive. The initial uh design has been uh developed in uh 15 months. So what is the design process? The design process is quite simple. Uh we first build uh some sketches and blueprints for each location and we need to respect uh the gameplay hotspot. We built in parallel a 3D placeholder of a set uh it's a gray boxes uh

and visual documentation uh package in mood boards. We test the 3D placeholder directly in real time in the engine and when it's validated in term of game play animation camera uh we can finalize the art concept and the documentation package it and it go to production for outsourcing or insourcing. So here is some example of the the blueprint. So remember it has been done by architect so it can be built and sometimes we we had to stop them because they want to add details and two realistic details but uh saying that uh we need an electrical plug here and we need that and and no stop it. We are making video game. We are not building a house.

And also they have they had a a hard time when they came back uh to the movie industry because with video games the we are very strong. We can do anything just in 3D. When they came back to the movie industry it's hard. there is some budget and wow when you're trying to do an office in a church you have to find it you have to budget it you have to build it. So here is uh some uh mood boards uh uh and picture references. They provide a huge documentation on on materials for modeling and shading. It's also uh give to the artist some um messy places reference. I mean that it's

always hard to put some chaos in a set except in fantasy games. that uh every object uh object position has to has a reason in the everyday life. When you go home, you throw your book on the sofa, newspaper, on the table and it has a a a meaning. It needs to be real. So providing chaos need to be um to provide consistency uh on the set. And we've got a precise uh mood board for furniture too. You can uh see on the bottom uh left uh

the gray boxes. And here is a material board reference for shaders and specularity and the concept heart uh give atmosphere. K colors and light setup. So this uh concept art is taken from the Han house. This one too, his bedroom. And so it goes on. So many many art concept because we have many location

and sometimes uh uh light setup will change in the same location. So it doubles the number of u hot concept. Sorry my computer is slow. Uh so the outside let's go to the production team. So uh inhouse we are organized in small core team uh for each type of assets. Uh each core team is in charge of producing managing uh all the sensitive uh art and tech technical driven assets and polish all the data coming from the the outsourcing partner. So you can see that it's not a huge team. In fact, we are uh we have really um a small team

but with very skilled artist and we also do many months uh outsourcing. So the set creation process it's placeholder realtime testing production in the Maya viewer icing I mean I mean polish doing the lighting setup uh putting the visual effects and doing the post-processing. So, uh you will uh so uh a little video it's uh a placeholder um and you've got uh the data coming from the outsourcing and after that uh the polishing.

I think you can see the difference in this uh video. You saw the real time testing. So just do a pause. Um we've got um some uh dimension constraint because uh every set is recreated during the during the mockup session. So uh in this uh video all the red uh object um means animation maybe uh game play and interaction and for the blue uh surfaces it means hand contact. So when we send it to production,

it means that 3D production but also animation and mockup production. So when we we've got a a 3D placeholder like that, we just have to recreate it on the mocap facility and do the shooting. And in parallel, we can also produce the polish set. And this is a final result. testing. So we can uh also validate uh collision. Uh we can improve motion kit. Uh we can uh test camera shot. So everything it's a kind of uh a skeleton for the production. It's the same for for for the cinematic. It's a kind of 3D

storyboard. So we can uh solve some uh technical issues, visual issues, many issues in fact. Let's talk about uh the design of the characters. Uh so on the character side design is slightly different because every rain is based on emotion story and characterization. So characters are one of the main focus we need to find uh actors able to play to give emotion to perform a scene like in the movie. So everything start with a casting to identify the right the right uh actor and um we do the same for the

main characters but also for the secondary one or for sidekicks. After that uh fashion designer and art concept uh provide mood boards and a final vision of the character. Everything is packaged for production. So here is some uh picture uh taken from the the casting. So the casting is done uh from board by agent but also we've got uh video casting with dialogue and and scene to play. This is Mr. Shelby before it was he and this guy is not really an actor. Uh it's a guy who who come uh in a casting

a free casting a free open casting and uh we say oh it's got a good look for one of a secondary character. So get on board. This is a classic uh fashion design board with many details uh rings um all the this small detail matching to uh to uh the characterization and after that uh final heart. We also do um not in this case but we also do what we call the turnaround just to help uh the artist to do uh the body uh modeling. So on character the process is quite classical. We do a photo session, 3D scan session. We do the resurfacing in ZBrush and we produce and finalize uh

the skin shaders and specific uh visual effects directly in the Maya uh plug-in. The animation uh setup is also I mean rigging and skinning is also done uh in Maya. uh we we we do animation and and for body and for facial. So this is a classic photo shooting same guys. This is a 3D scan. So you can uh notice that it's not a magic tool. It's quite precise that we lose many details in some specific area like here's convex surfaces or airs. So it's not a 360 degree scan. So we are

not able to perform a full body scan. Concerning the resurfacing, we uh develop a tool that enables us to project a a face template to match to match the 3D scan. So it's quite useful and um very quick because it's ensure us that each face has the the right topology for muscles and facial animation and all the micro details of the flesh I mean wrinkles imperfection of the skin are added in a ZBrush. You can notice uh too that this guy is

going much more a badass. So because we we we think that uh his face was too flat. So we try to make him much more aggressive. So sometimes we we have the freedom to change a little bit uh to um improve the characterization. The body is totally built in ZBrush. It's finalized in Maya and in the QD viewer. And the polygon budget is close to 20,000 approximately uh 10,000 for the head and 10,000 for for the body. depending on the fashion uh design.

We also do uh facial animation. So uh we can test uh facial expression uh directly in uh in the QD viewer. So it's not uh blend shapes, it's a bonebased system. So let's talk about uh the tools. uh after Fahrenheit uh we've done a a huge postmortem and we asked the R&D to build a real-time viewer uh directly integrated in Maya because we really think that uh real time visualization is crucial in term of production but for artists

we use uh Maya 8.5 uh integrated an openG engine viewer directly linked to a PS3 remote. So we can send uh automatically the data directly through to the PS3 dev kit. So the QDP shed um it's a nal material editor. We've got our own uh shaders. We've gone we've got real time on shedding network and lighting. The advantages it's improve the artistic control. We can build simple and complex uh sheddders. It's an artistdriven sheder. There no need of code. It's easy to recruit because when you're coming from a CG and you know Maya quite well, it's the same. But there is some disadvantages. It means that you can build uh many many

many shedding networks and very complex. So uh artists need to become much more technical. The Maya realtime viewer in the PS3 remote. So we've got uh as I say re realtime feedback on shaders, lighting, animation, visual effects and particles. The main goals are uh always improving the artistic control polish decrease time consuming export. I mean when you have to export check in the engine so you you saw some bugs where you need to do some visual retext you came back to your package do the retext reexport. So it's boring and uh it spends too much time specifically on on Fahrenheit. So

we say no, we need to have a much more smooth pipeline. We also can do much more iteration. The postprocessing editor, so it allows us to manage uh postprocessing uh as a 2D composite layer. As you can see, uh the GUI is very close to the Photoshop style layer. So, it's really easy easy to handle for um for juniors and for uh seniors. We can do many iteration without changing any data. So we can add a layer for saturation for contrast, for light, for luminance. Uh we can add a depth of field and we've got the real-time feedback. So everything is

compiled in a in a in a kind of uh preset. you save it and you can send it to to the camera department or to the script and they can change dynamically uh the rendering. So for prototyping it's quite useful and for the final final tuning too. We also had uh two um tools. It's an ambient occlusion manager. Um I I'm talking about every rain because uh now we don't use it. Uh but on every rain we are baking ambient occlusion. Uh it was a huge Atlas sharing quick baking much more quick than mentor way. Um and we can do some retakes uh in 2D package. send the atlas to Photoshop, fix some bug and uh get back to the engine.

Uh the other tool we are still uh using it. Uh it's the ambience control. It's a it's a curve based editor. We can animate all editable parameters from any type. So, light intensity, meshes, specular power, skin shader, uh attribute, uh texture, texture map, mask, everything. So, animating characters uh I mean uh the body part. So, we've got uh 28 uh Vicon. I'm talking on often an old time because now we are uh 74. Uh the marker set is 55 and uh we have many many days of shooting. We also uh use some stuntmen because

there is some dangerous uh session I mean fighting falling. You will see a video. So um we also need to have some generic uh skeleton with some archetype uh matching to the stunt man. So the performance is split so we can reuse many uh small part of a session. There is massive work on preparing the props. Uh as I said previously, we can build uh for real uh on the on the mockup facility uh each set. So each set need to to be rebuild uh and we've got a script always uh taking notes of any changes during the session.

It requires a new approach of directing uh because actors have no references and also uh it's a huge uh investment uh from actors and long-term uh collaboration because uh we shoot many many many days and we really need uh to the actors in house because they are doing the body session but also the facial session. So on facial uh we shoot with camera. We've got a much more uh complex marker set because we need much more detail less day of shooting but I'm not sure

that uh it's the the I'm not sure that the actors agree. uh we do the facial capture uh and the voice recording at the same time. Facial animation is split from body. So it means that uh we can use any facial animation on any body animation. So it's quite useful specifically when you're talking about of a game with many branching, many story branching. So it's very useful. You don't have to redo all the animation because they are standalone. We also uh can do re retargeting between characters because uh the the facial uh rigging has the same template for every

characters. But uh when uh you do the facial mockup, you need to have the same acting as the body session. So it's quite a huge issue for the actors. So uh let's have a look to a kind of making off [music] are there. He

Oh yeah. [screaming]

Oh no. [sighs] [screaming] Hey, hey, no.

Uh so I was supposed to show you some uh video of u facial animation uh taken in real time but we've got some issue uh between video and sound. So nothing is synchronized. It's not u it's an epic fail. So let's move uh let's move on the uh on the fifth character part. I mean uh the rain. So when we scoot in Philadelphia, uh it was supposed to be a rainy season. We spend um 10 days scooting and it rains one single day. Another epic fail.

We uh were a little bit disappointed. uh so we start to build a huge collection of documentation about rain rain effect just to identify the main aspect of this natural phenomena. So uh rain can be very different can be smoky very diffuse heavy hard aggressive it's um change it change the color the light even the animation because you can see that when it's raining people always doing like that because it's very aggressive. So uh with this huge documentation

the concept art artist start to paint over a real picture of rainy day just taken from the the web and they try to improve to maximize uh some of their aspect. I mean light diffusion, reflection, saturation, uh specularity and uh it gives us uh a better understanding of what we need to recreate. So you can uh see that it has been paint over B train it's uh also uh small details

skies light and diffusion of Right. So I will show you uh in this video uh some example of uh wet environment in every vine. So, this is the the crime scene the butterfly uh power plant, some highway, a junkyard, the dock house, the highway taken from the the taxi. They are missed.

So, uh in this video you can saw the different layers uh we we use to improve uh this uh rainy effect. So a quick uh overview because uh we need to have some lack of time. Uh rain effects are not just compon layers of particles. Uh usually people say let's do particles and uh it's raining. Okay. Uh it's 2D layer and 2D sprite. We also add some atmospheric uh effect with camera shaders and post-processing effect.

We change materials uh specularity and we do uh many adjustment. We also add some procedural ripples. We've got also uh the bouncing effect by adding uh secondary uh particle and we uh also use wind effect to created by vertx wind but uh I'm not very proud of this part because it sucks. Um, also rain. It's rain on props there. that uh also on this car.

So, you can't afford to not having rain on a car if it's raining everywhere. But it's the same on the windows. It's very dark. So as I say, rain interacts with all props. We add some reflection uh effect on transparent materials. animated normal maps, ripples effect, animated UV texture map to create some flow effect. We animated a mask and video for the whipper. It's not a dynamic fluid simulation.

And uh we uh synchronize video with meshes animation. We also had to create uh over three effect not just rain. Um for people who played uh heavy rain there is a huge sequence when medicine had a shower. So um this is not medicine sorry. Uh next time we also uh have a a huge sequence uh when a car is floating. Okay.

So, we are supposed to do that in real time. It's a huge challenge. Uh this uh sequence has been has been cancelled because it was uh too paranormal u for for the story. So the icing hasn't been finished. But uh in fact we we were flooding uh the Han house with um water. So MISK effect uh done by uh dynamic

particles collider setup for the shower uh secondary bounces uh also on the shower. Uh we animated crack mask u meshes swap for explosion uh for window explosion. We've got 2D decal also for crack windows, sprite, reflection materials. But uh what is uh water without fire? So let's burn. Uh we do fire meshes and with a UV uh texture slide uh smoke uh meshes, UV UV slide. Uh we fake also some lighting variation. Uh add some uh fire and sparkle sparkle particles. Uh reflection meshes for heat effect,

additional smoke particles. Um and we did we need to do that uh for a free uh 60 uh degree uh camera and uh we need to avoid uh field rate issues. So here is the final result. It's points a little bit, but

So a conclusion everyone uh required a lot of uh high detail and high quality data also many tips and tricks. because we can't afford technically to do everything with real time simulation. Rain uh is still for me difficult to to simulate in real time and uh the other uh challenge we have to to solve was uh avoiding Hankan Valley effect. It's been a a real daily fighting.

[clears throat] Uh interactive drama is a kind of juggernaut. Um many data management, many things to produce, many path, a huge testing uh uh step and also it's required a new way to work uh especially uh regarding the complexity uh and um we uh hopefully uh add many tools to help us. The communication for us is quite crucial because uh we've got um some artistic uh team inhouse but also overseas not uh on the same page uh because there is some cultural gap. We need also to communicate uh cross

department uh challenges and also we need to uh communicate with publisher. Uh every rain is not a a firsterson shooter. So you can't do a vertical slice of a story and say oh that's it. You can't do that. So in the early stage we have one scene we can test it, show it, show show many graphics but the real um the project start to be really strong when everything is built so close to the alpha because you have the big picture. you can handle the story, the complexity of the story. So, it's quite hard to communicate to publisher, but also hard to communicate to the team

because they always saw some details, some scene, some set, some characters, but they don't they can't handle the big picture. even if they had read uh the script 2,000 pages. So we had to communicate uh the art direction to do micro synopsis and always always communicate internally with the team. We also had to make people uh more productive and creative. So build a solid pipeline, provide tools and also um increase the create creation phase uh versus the data management uh time. So um in conclusion always try uh to uh

be challenging to be involved and uh to bring new concept uh to life. Thank you for attending. [applause] [applause] Thank you for patient. Uh sorry for your hears and uh thanks to the KIC dream team because uh without them we can't uh ship the