'Insurrection 1944': Bringing Interactive Storytelling to a Museum's Audience

[Music] thank you hello everybody my name is Nina sepina I am a CEO and co-founder of realcast our company develops immersive experience in extended reality using different kind of Technologies virtual reality and mixed reality augmented reality the last four years we have developed many different experiences for cultural locations to present to the visitors of the Museum of Heritage locations my company was created four years ago with my co-founder Diego Fernandez Bravo he's a technical director and I am a producer we work together for more than 15 years in video games industry not only we developed a Ubisoft very famous games like Just Dance and arriving rabbits it's a very mainstream

games using motion motion gaming with the controllers with the Kinect cameras so it was always interested you see Cloud Interactive Innovation can bring new kind of experience to The Gamers go to the visitors when we build this company we wanted to see how this new immersive technology works with the general public the public who is not technical who doesn't know how headsets works and who doesn't know even how the games works because in museums obviously you have people from all the ages all the technical and non-technical knowledge some people are even don't like technical devices so we wanted to test this technology with this kind of audience before getting back to games that we are building right now so I will show you the experience that

caused Insurrection 1944 that speaks about the Second World War uh and the end of the war in Paris let me present you the the location so this is a view of a museum of the liberation of Paris uh it's built on the place the pharaoh's Road which is a famous place for people who already visited Paris they probably visited catacombs the catacombs just across the Boulevard on the other side of the boulevard and the particular thing about this museum is that it's built upon a very historical location which is the underground shelter that was built between two Wars for the population of Paris in case of chemical attack that never happened but this place was built for that time and then the resistance during the second world war have identified this place as a ideal place to to to to go there and to

and to act organize the resurrection and the resistance in Paris region foreign you can see the the stairs that are leading to this place which is quite deep underground there are 200 steps so it's long uh descent to the to the location and and the second review the second photo you can see uh corridors today they are completely empty which means that for the Museum Direction they they really had hard time to understand to find the solution how to present what happened at this location without any physical artifacts or things to show inside they are small small um primaries to you can see some elements but very few because most of the things are were taken away so the all corridors are different the same

sorry and and they're in the rooms as well so the solution of uh virtual and mixed reality was really ideal to to populate this place to give the life to this place so the visitors can be like in the time of 2000 1944 and to meet the the resistance and the people who was working there for two weeks at the end of the war so I will show you the video without sound that I will comment that that shows the museum and the The Experience itself uh please note that the video cannot transcript the real immersive experience so you can only see these 2D images but the only and the best way to to realize is to go there and to and to have experience so I warmly invite you to to come to Paris when you can this exhibit is permanent so it will be there anytime

and and so you can have a real experience so so this is the museum and um and the number Russell place and we have been using archive video that was filmed at the end of the war where the they filmed the the resistance still in this location and this helped us to to reconstruct the the characters the objects and the different activities that happened in this location it was for example uh Central hotel from Central and of course different uh characters like this soldier who is your guide and Colin Rock was the leader who was the chief of the resistance and very famous historical character and thanks to this experience you will be able to meet me

you can see here the the picture the photo of the corner rock that was taken the same location as as as we was building the experience and we have recreated the colonial role in three dimension using their human actors and we did the same scene the phone scene where we was phoning to um to to the network of resistance to find to resolve some very urgent situation during the liberation of Paris the particularity of this place is that is 100 humidity so very uh particular and difficult constraints for for technical materials so it means that we should not leave any headset or other devices in this place for a long time obviously the the the the depth and difficulty to access there was no Wi-Fi

at all that was also the constraint from the museum so we had to build the system for hololens that use a special anchoring but only uh locally embarked in the headset with no possibility to have a communication between uh headset and the remote calculations for example that could help in a special in this picture or anchoring and as I said we had very few elements mostly photos and and this only movie that was captured at the end of the war so the choice of the technology of Collins was really I would say the the one of you that fits the needs of the museum and and their their will to to show what happened in this place without any physical objects so we we we want this uh diverse public

appeal to to win the project and then we it was a very big rush because we had only seven months to build the whole experience which is quite short regarding the all the creative elements that had to be built and all also technical hurdles that should be resolved in this short period of time it's a little bit about the about how it works so here you can see the The Notebook of the of of the visitor because visitors plays a raw of the journalists in the time of second world war and his role was to interview all the resistance and to to understand what is going on during the war and during their action and to make article for the Press uh for the Parisian population to know how to interact and how to liberate from the confidence of

the Nazis before the Allies troops arrived because Allied troops were arriving in the two weeks and and the resistance was there to to make a communication between the resistance in Paris in the Paris observes and obviously with the Allies troops that are arriving so here you can see the the initial Works uh inside of the museum before the installation use the director of the museum showing us the plans the this is a big door uh the entrance of this place and this is the famous uh indication where is the office of the corner roll that dates from this moment and that it's part of all the elements of the of the storytelling uh the challenge was how to make life in this place without any objects so as I said we built a whole set of characters and the objects and the

storyline to like to lead you through this space uh in a very natural way without need to Any Human Assistance so it's a visual it's audio audio is very important and uh we we succeeded to build this experience uh to to to lead the the visitors you know in a very simple manner that behind that was very complex technical solution so this is the plan of the location so you can see there are multiple corridors and multiple rooms with different activities so was it like entrance so entrance was the first uh I would say contact with the with the mixed reality for the visitors so uh they they just got the headset on the on the head and then they see the door that opens so they have to press the ring button with their finger

because it works with a hand tracking and then the doors open and then they see the the virtual character inviting them to lead him into the experience and then uh the character leads them to different rooms so one of the thing that was very important is that this place is quite I would say the the corridors are not very large so we wanted that visitors can go in different rooms but not all together in the same time so we built several paths through this location uh so every time one another visitor arrives he were he would take another path and do the experience in different ways so that was a solution to to use the place the whole place as much as possible without having people working working in the same direction at the same time so the best comfort and the best quality of the experience we were using for this the the mechanics

of video games that we worked for many years which means that everything is interactive into each room you can meet the character that speaks to you that asks you to do some very simple gesture it's another video game it should be more accessible because it experience is shorter than video game but still it was very important for for the immersed immersive part the visitor can do things in each of the room and feel like they were part of the story so for example here in the in the in the room of um in headquarters they were like uh building a barricade in a in a reduced model in the phone Central there was reading the Articles to the to the operator in the in the secretary room there was uh finding the movement of the troops to be shown on the map that was in the wall and in The

Colorado Room they were giving uh a pen to the corner roll so he can show them what's going on there and after they were following the the soldier through the experience so here you can see the I would say without a wall the whole different characters and the settings so it's in fact it's a high quite huge scene but obviously separated in the smaller chunks of each experience so you can have seven characters that are in this in this experience that are getting activated when you come close to them and they they talk to you it's uh the the character approach is very much uh related to the I would say the movie storytelling because they're alive they're looking at you at the eyes uh and they they are really following what you are doing and if you for example you you get lost or you wait in

a corridor but they will stop they will wait for you they will ask you do you want to continue if you do something wrong for example you don't understand what you have to do they will they will try to help you and then at some point they will just lead you to the next step so it's really intelligent and adaptive uh interaction towards the visitors so they they feel like they're really handled by the real humans even if it's only virtual characters here you can see the the the the excerpt of the movie and uh also the archives so we we use the scan of the journals from the time and put and perform them in the 3D models and then make them interactive so visitors could choose the article that the secretary will read and then make an action to show what's going on on the

real I mean in the in the field and to show them where are the troops coming we we built the scenario with the Specialists and experts from the history that was uh provided by the museum and so we did a lot of work for the research to be really very accurate in terms of history and to and to choose the the most crucial part of the history and in the same time to make a choice not to do too much because the experience is 25 minutes which is quite long for this kind of experience but still uh in the end it goes very time passed very quickly and it's important for the visitor to remember the the key elements of the history and then after he can go upstairs in the museum and we're much more in depth about each of the elements and find information about every moment of this uh of the immigration of pairs

foreign the equipment and the habits of the soldier that we used to scan in three dimensions so our approach was really to use the the actors the real actors that we we did like a casting and we choose the actors that was uh do close closest to the loops of the historical character for example Corner roll and also we work with uh with all the I would say creative crew from the movies like uh uh costumes like Monica like um for for them for the look like of the of the face to make relief make them as real as possible before to do this camp this is the session of a capture of uh

photogrammetry so each each actor went to this session and was captured in multiple times to have a best quality model of his body with the with the habits of the time and so there was a scan of the body and there was more detail scan on the face you can see our our developer working on the on the integration of different assets uh 3D assets animations ellipsync dialogues into the same character that became alive and this is also very close to the video game industry where you build the characters they they have a full set of motion capacity to walk to talk to to to interrupt their movements regarding what you are doing

here is you will see the session of motion capture which is uh also way to after this 3D scan we we did the motion capture to capture all the movement that we have been decided to be used in this experience and this is a soldier raw he's opened the first door and inviting the visitors to to come and to enter the experience so we spent several days to to capture all these movements you can see the Sound Studio with the recording of voices uh the funny thing is that the the main character Corner Rock who is um who's obviously French and when we decided to make an English version of The Experience uh we decided to keep him speaking English with his

very strong French accent and it was a really great choice and all the visitors from England for the United States they find that very normal and have no problem understanding and makes this experience even more I would say immersive and like like the last image is about facial facial uh capture where we captured all the expressions of each um actor and after we created a facial expression that was added to the model and then enable each actor to talk different languages with a very accurate lip syncs so there are two languages available French in English here is the manual to use the important part of the experience is also to have a

person who is taking care about how the experience [Music] is managed outside of the experience means receiving the visitors explaining what they will do not in terms of content in terms of just uh handling and uh start in the end of the experience the idea is that the visitor doesn't have to touch headset at all everything is installed by this mediator and uh he's taking the headset at the end of the experience and the people should just forget about technology and go through the experience themselves uh and and they'll go completely immersed into the experience so these are the indications so this is for Holiness one because we started with holding this one in August 2019 uh so the first version was foreign so there was all instructions how to handle charging uh installation and

different adjustments for for the mediators not for the users with a lot of play tests before deployment which means that we passed at least one month's testing with different uh Personnel of the museum to to find the best adjustments for the experience and then even after the deployment we use one months more to see how the experience was working with the users and thanks to the mediator who was there watching The Experience uh get the feedbacks and uh and improved experience uh in in during integration of this semester and that was very important the Microsoft was also a great partner because they were present they even sent assembled people from the Redmond uh r d to help us with the spatial anchoring which was very big challenge for this

project um we didn't have any uh loss of material and then it breaks so this is really thanks to this system of a mediator who is taking care of completely care about about headsets and not leaving the user to manipulate because it's not needed and as you will see at the end it's the the returns of the public are really very very positive and and that was really a big uh bigger Plus for this kind of experience where normally uh one part of the visitors are reluctant to this kind of technical solution foreign [Music] six months after the installation of the experience it was one of the first companies and teams to uh to deploy a Holland School experience uh in this in this Museum was like also the

first client to to help the colonists to uh installed so it was a very big project and exciting for us to to get hands on this new device since then we have more than 10 000 visitors even we are covered uh removed big part uh still this experience is permanent and it's bringing the visitors every day obviously this kind of experience brings young audience to the museums which is their purpose but it works very well with also people for any age and uh and this was also a big Discovery for us to see that even the people for plus 60 to 70 uh can do the experience and appreciate the new technologies and and to and to see and to participate in this kind of experience so uh the the team was really very much involved and did huge efforts to make this happen

and fortunately we get a lot of feedback from the Press from the media after the release so we had a huge coverage coming even from United States uh we collaborated with Microsoft and they've sent some journalists some influencers to make different articles it was really this experience still today it's a reference I would say in use of mixed reality in a in the cultural locations finally we won two Awards status 360. it was a world given by the movie makers famous movie makers uh like you're a movie maker from Taxi which was a very famous movie in France because they discovered this kind of historical and they didn't know and we also won the oriya award in Germany which is a very famous award since uh two years now and we was like a winner in 2020 in fact the the award in 2019

but it would happen in January 2020. and obviously we are very happy to participate in GDC with one year of delay but still we believe that a great opportunity to share this experience uh with uh with you so a little bit more about visitors feedbacks so as we said the museum and as we we really uh take in account you try to take in account a lot about the opinions of the visitors especially the first visitors and to see how they are using their experience what are the hurdles maybe obstacles what makes them uh what make the experience maybe uh people who don't want to finish or who don't like the technical solution but uh frankly we was very surprised but very very positive feedback from all over the museum so I can show you some

centers there is like a book of of remarks from the from the visitors and the museum Perry music who is handling 40 museums in Paris so they have really a lot of different museums and and have different kind of digital solutions for these museums uh they they say that this this is the first one the such a big uh positive feedback from the audience it's almost unanimous uh so people are really uh very very happy and a very impressed but what's possible with this kind of experience so we uh yeah we we we we are we are happy to and we believe that there is a big future for this kind of experience in a different location we also now especially with the code are also proposing to the location to build

things situ experience but also virtual experience that use the same assets so that can enable people who are at distance to participate to this uh experience together foreign presentation on this slide and please feel free to to ask any question you have um after the after the session thank you [Music]