Project Discovery: Lessons From Scientific Research in EVE Online

welcome all welcome all to this talk about Pro Discovery and uh and integrating citizen science into E online um just to kind of you know start you off please you know the drill turn off your phones and fill out the survey and all the nice things um my name is B finon people call me Burger like hamburgers it's much easier um I'm a senior development manager at CCP games in rikic and I'm AA Sunner I'm the CEO and co-founder of massively multiplayer online science and um exactly a year ago during GDC we launched a new project code name project discovery which later became one of the biggest citizen science efforts in 2016 this even more remarkable if you consider the fact that none of the parties involved had a track record in citizen science so how did we manage to make this work

well as a teaser I can tell you that video games and Gamers played a substantial role in the success but to tell you the whole story I have to start with uh citizen science citizen science is the crowdsourcing of scientific data analysis or data acquisition but I believe it's much more than that it's um it has a very strong educational aspect and also it's an invaluable tool inside Outreach it's Paramount that we understand what scientists are working on even if it may be on a superficial level and what would be a better tool for that than to actually work on the data that scientists work with so um citizen science I believe is very important now you may have heard the term citizen science uh and I'm pretty sure you uh met some of these projects in citizen science like the folded protein folding game where we try to find native structures of proteins

or uh Z universe galaxy Zoo where we classify galaxies captured on a Hubble Deep Space images actually this project became so successful so popular that a whole platform grew out of it and now they're hosting like more than 50 different projects the another good example is uh the pho which is a a project from Canada from the mill uh University and um here resarch archers are trying to solve DNA sequence alignment problems and uh finally uh a final example iire where we help to map the visual neurons in the human brain all of these projects do an amazing job in citizen science but they have a major problem to tackle and that is user engagement and retention so that's where I our story started

two years ago with a friend of mine bnard Ras we had an idea that instead of creating Standalone citizen science games or citizen science um gamified experiences why don't we take these problems and integrate them with the most engaging form of entertainment of today which is unquestionably video games integrating them with major already existing AAA titles and integrate them as a seamless gaming experience and our hyp thesis was that if we manage to make this manage to create this experience manage to uh match the visuals connect the dots with the lore uh with the reward system with the game mechanics and as the pros of the game industry to create an engaging experience then we might be able to transform a tiny percentage of all the time that we spend with gaming to do something good in real life and uh that tiny percentage would be a virtually Limitless human computation engine to

serve science so that was our initial concept and first we contacted researchers we tried to validate the the idea and um we got very enthusiastic responses and we met with Emma lanberg uh she is heading a research group in Sweden at the human protein Atlas they're building a big open database of microscope images and uh we immediately knew that this kind of material would be a perfect raw material for a first project the she had the cor million images beautiful ones uh the the task of the gamer was to find the or localize the green uh markings on the image that Mark uh the location of a certain protein so this was one part of the job this was actually the easier part of the job we had a lot of friends in research but the harder part was to to get in touch with uh with the gaming industry

with you guys and uh when we had this idea I immediately knew that evoline would be a perfect match for the project so we started to work on that and after a couple of months I got an email from Andy orren she's the executive producer of EV online that this sounds intriguing let's have a Skype so uh she iMed immediately became a huge supporter of the project and uh around 2015 member we already flew to rikic and uh we're presenting this concept to the stakeholders at CCP so integrating citizen science into a into a AAA established computer game is is no easy task um like AA was saying the E Dev team was super curious about this about this idea and about this project and and for those of you that that don't know e on line e on line is a

single sead massively multiplayer online game where hundreds of thousands of players fight over power wealth and fame um and Eve players decide their own fate and uh write their own stories but you know Pro Discovery and even on line why well first because because of science we're all huge science Buffs at CCP and you know for myself my kind of first interaction with with crowd sour science was uh this beautiful thing called C at home um I think this was like back in 1999 or something and me and my best friend used to sit and watch this for hours looking for aliens in far away galaxies um we're actually on di up modem so his mom was not happy when she got the phone bill but hey science um but the thing you know there was no um there was no interaction with this this was just

something you watched and it just kind of chucked along so we kind of you know while we were looking at this we were kind of you know we were talking about it we were kind of role playing this whole thing and you know of course playing at that time a lot of XCOM uh apocalypse I think it was um you know because we had to prepare for the invasion so it had kind of been you know stuck in my mind for for quite some time long before I started the gaming industry it's like how can you actually merge these two together so when AA called I was super stoked um this is a this is a really interesting game design Challenge and uh you know we really look at this as an opportunity to take part in something much bigger and you know who isn't up for a great Challenge and we also believe that you know we also hope that our example sets the states for many other successful collaborations between uh science and gaming to come so yeah but there were a

lot of unknowns when I kind of dove into this and we really enjoy unknowns at CCP and um it was everything from from technical unknowns to legal issues very very very interesting legal issues um and also how in the world we're going to design and build this thing but we kind of we kind of set four main design principles to kind of guide us on this journey the first one was uh we wanted this designs to be of real world purpose uh and uh alteristic nature so you know drone strike analysis bad cancer research or mapping human proteins good um it had to be had to be aesthetically fitting and thematically adoptable to fit the eve Universe um so you know we didn't want we didn't want research data with images of giraffes or rhinos um or you know we didn't really want any street signs from

planet Earth since you know new eeden is very far from planet Earth uh the task had to take consistent length of time to solve um we don't didn't want to create a moment where you know some things are like 2 seconds and then you get like a 15minute data set you had to solve um and the research needed high volume of tasks so we yeah half a million tasks was kind of no a quarter of a million task was kind of you know that fitted the first first research data quite well so as you have seen the data from Human protein Atlas is really beautiful and actually to a point where arctor thought we had tinkered with the data before showing it to him um thematically in game uh the ingame science faction systs of Eve are actually researching tissue samples they've gotten from the mysterious dri race in the game they're

trying to kind of make sense of of these thrifters um the sister ofie uh research group is led by Professor lunberg uh and calls upon capsu of new Eden to take part in solving this scientific challenge through the project Discovery interface um and this is really great because you know you have Professor Lindberg in game and then Professor Lindberg in reality so you're kind of merching kind of playing with merching realities here um on time to complete we knew that this was not the easiest task to understand you know mapping these proteins so we you know we spent a lot of time balancing and tweaking and you know working testing the thing testing the game out on our test servers and you know getting a lot of great feedback from our players um and yeah there was a there was a lot of lot of tweaks in the first uh first month of of testing

[Music] um and as I said before the subs protein Atlas had around 250,000 samples so we thought it was a great kind of start project for for prod Discovery um our preliminary calculation gave us around two to three months to reach full consensus on the data um um we were proven wrong but I won't spoil I won't Spill the Beans fortilla he will he will you know have something for him to talk about so actually how how did we build this thing um we partner up with re University uh the computer science department at ra University to work with us on this um and we thought it was a it was a great opportunity to kind of strengthen our ties with universities in in Iceland and kind of you know it it kind of goes a full circle then um uh so this is kind of this is the first version of of Eve sitz in science

uh this was a web browser version we showed at FanFest um I think it's 2015 when we showed this yeah yeah so and yeah we showed the first version at e FunFest 2015 got a lot of great uh feedback there then took it to EV Vegas and and launched it on CC which is our test over um and then launched it uh exactly ago so kind of the core idea of of pro Discovery is that CCP provides the the framework and then MMOs provides the content it's kind of the link between us and the scientist and the scientists or the science Partners provide the the uh content for this Frame um this makes the whole kind of dependency chain really interesting because due to legal issues we are not allowed to host a lot of the data that's that's being shown here um

so there's a really interesting challenge of of kind of giving away or kind of having to depend on a third party for your content which is a which is always a super scary thing when you're a game developer because yeah if anyone has tried it you know there is time to penis is very short um so how does it actually work in game um while you play EV online you can launch the prois discovery window and play um and you know this is this is because of the the you know the time to complete is relatively short it's a it's a great thing you can do while kind of doing things that take longer time such as mining or uh when you traveling between places or just hanging around in station chatting to people so it's a it's kind of a good side project to play to work on um so when you open the interface you get a mid of a cell sample and your task

is to identify where the green Mark proteins uh yeah where where the green where the green proteins sit um for example here the between in the nucleus I think yeah nucleus probably now someone will call me out um so yeah you pick the categories on your right and uh once you kind of feel confident you submit uh your answer and it's and the answer is analyzed against all the other and all the other uh analysis submitted for that sample um that then gives you feedback on your submission so in this case here you know we picked the correct answer you can have multiple answers which makes this like exponentially more difficult um but to actually calculate uh accuracy and like you know

to reward you there are around 15% of all tasks in prodct Discovery are what we call Pre pre- solved or gold standard um tasks um that we use to yeah judge your accuracy and and reward you accordingly um when you reach a new level so you basically level up slowly uh you gain experience point to level up slowly and when you reach a new level you get something called analysis credits which um which then you can spend to to uh buy some awesome rewards that you can can't get in RS but you know while you play this thing so actually on the gold standard what's quite interesting is because we can't toast any of the images um the gold standard tasks are balanced by

the scientists so there is a that creates a lot of really interesting back and forth between between uh game us as game developers and and the scientists and kind of talking about game best practices and I think the HP guys are you know they're becoming like super game developers slowly um so once aample reaches a significant level of of agreement um we can say with some certainty that it has reached a consensus um yeah so the sample actually lives in a database until that consensus has been reached and this can mean anything between 15 to 50 submissions of each sample so the data set you know even if the data set is 250,000 images it's actually 250,000 times 15 till 50 so it's a it's a relatively big big task

and I thought to get our hands a bit more dirty with technical details uh we created a platform that on one side standardizing the data and uh the problem types that comes from research and the other side it provides an API for game developers uh which is important because uh you can focus on creating the user interface and an engaging experience uh for your players and you don't have to be swamped by the details of of the citizen science process itself um uh that is very uh important that this part of the architecture is performance so it can handle heavy load and also from a business point of view it's interesting that we're a neutral party so I think that way it's easier for other game developers to hop on board and be part of this um now I really hope that at this point you already see how how cool and important and and fun this project is uh

but I really want to show how easy it is to to integrate so this is basically there are two things that that you have to do is uh this is roughly the code that you have to call get a task from our servers we provide you with all the information that you need to visualize this task and then then when the player is ready uh you submit the task again relatively simple and our API provides you with all the information that is needed to to to show the result screen uh we give you an accuracy score that you can use to to uh to to hand out rewards for the players so it's it's as a simple as that uh other than that you can really focus on on building being building an engaging experience and connecting the dot with dots with the lore and and uh all the all the in-game mechanics so the result uh after all this preparation you can

believe that we couldn't wait to see how players reacted and and what what's coming coming out of this and it was uh way beyond our our wildest imaginations uh this is the this is the numbers from the first day uh we had around half a million classifications and almost 15,000 images reached consensus uh this was along with about 3 or 4 hours of D time I was here at GDC I couldn't come to GDC I was in the hotel room with not the best internet trying to keep the whole thing alive so it's quite interesting uh and uh another nice metric is the sisters of Eve combat suit which is the kind of the ultimate reward in uh in Project Discovery and we thought that it would take at least a couple of days to hand out uh one of those uh we we had to hand out 45 in the first day which was pretty unbelievable

I don't remember you had the number of how many hours we estimated 16 hours and what was it 16 hours and 17 minutes to hand out the first one yeah and the second one was like 2 minutes later so that guy like if he wouldn't had you know had a toast but rather just like dry bread he would have won so tough luck yeah we could call it a draw I think yeah we could call it a draw so it was it was pretty amazing uh we thought that we we had quar million images to analyze as the first batch we we had an estimation of two or 3 months uh again we were very wrong with this uh players have uh gone gone over the whole first batch in three weeks time uh and since then actually we we put the same data set for the sixth time in the system for players to analyze which is really interesting from the cities and science point of view to see how how uh the the different

consensus uh uh what what what what is the relation between time and and the different consensus that players are creating and so on so it's it's in itself it's really interesting research in citizen science and as of today we have more than 25 million classifications in the system which is pretty pretty amazing from the scientific side we're we're reseearch Searchers are still working on to analyze the data there is one uh thing that Emma already published uh is this this is a very special protein structure called rods and rings for obvious reasons and players have identified around 100 new protein candidates that play a part in this structure so it's already really nice from a game designer point of view I think it's even more interesting how the Comm commun reacted and we can say

that the community embraced the project uh they were very active they created a lot of content uh Vick Pages uh tutorial videos these are some quotes from uh e forums which show how strong the educational aspect of the project was and how uh a great experience CCP created around the problem set our one of our favorite thing is this poetry competition which was uh about project Discovery and I love this almost classic tub are red nucleus is blue but it's only the green that matters to you which actually describes perfectly what you have to do in the project uh from the researcher side uh as B mentioned uh we created a an avatar for Emma uh but it's not just that but they were really active in the whole process so they were communicating with the players they were active on the forums they were coming to FanFest um and uh for example they were giving a

giving a a a streaming lecture uh about proteomics in Eve University using the same material that they used in at in Stockholm for University students to teach proteomics which was a really nice crossover uh also one one of my favorite stories about how committed they were is that uh two years ago when we uh at EAS where we showcased the first prototype uh Emma actually cancelled her appearance at the proteon conference where she was supposed to speak about her research and opted to come to Vegas and talk to players about this and then we had um a live citizen science tournament in Rik last week it was a lot of fun uh and actually Hilmar mentioned it in his closing speech as uh this being the next Esports which was of course course a huge exaggeration but it was

really nice to hear and um then we had a lot of uh lot of articles about the project uh both in high-profile scientific journals uh actually uh the the journalist who wrote this article told me that probably this is the first time that I published a screenshot from video game which is which is nice I think and uh also in in mainstream media and of course this was very good for CCP this was very good for us for human protein at least for the project but I think having uh that much attention from the general public from the from known Gamers uh having such uh positive messages in the media is very important for the whole gaming industry now a couple of words about the future so uh we are of course trying to work with more and more game developers one project that I can talk of is gearbox uh two years ago I met with Randy pitford here at GDC and he became

a a a very big fan in supporter of this whole project and I don't know if you've uh you've seen this presentation uh he he showed some some uh images and videos about uh art new artistic SES and techniques that they're experimenting with and we just can't wait to see how citizen science can be integrated with that titles and another thing is uh is this um we realize that from a higher perspective what we're doing we are trying to find already existing established um active online communities and try to kind of tweak their activities a bit or smuggle a little bit of Science in their everyday life and so we so we thought that why don't we approach muks which are these big open online courses with tens of thousands of very enthusiastic and motivated students and uh why don't we bring real uh life

scientific data for them to analyze so uh in that effort we're working together with the Geneva University and uh we try to bring exoplanet research data into their into their course on the subject and they are very prestigious Institute on that uh because Michelle mayor uh who is a professor of the University uh is actually uh the researcher who found the very first exoplanet so so that's very interesting and of course uh a very important part of our future plans is to have new projects together with [Applause] CCP so with the human protein Atlas uh project kind of you know it has kind of run its core so we have decided to like Sunset it and you know start focusing on other other research um and it's it's actually quite great

because now we've built this framework and it's it's it's now a matter of trying to find new ways to visualize new Ty types of data um and add think an interesting creative layer so kind of the four principles that I talked about earlier what we actually learned you know since we wrote those is that you know looking for a new project kind of the the most important thing that we found is that we don't want to devalue the project as it is today um we have set a certain expectation standard in in prois Discovery and it's not only the alteristic nature and the seamless connection that uh kind of pulls in the community but it's also the complexity and the depth of the research um kind of a caveat here the eve Community is very open and willing to take on like very difficult challenges and you know they they kind of I don't know if you've played Eve but Eve is kind of a hardcore game that's yeah it's kind of a maybe a light yeah

but of course there are plenty of simpler research projects out there so you know you don't need to worry that you know you need to teach your players to to fold proteins or something um so after go going through a few different data sets and uh narrowing down you know we managed to narrow down to something that felt really close to home and we're actually going to work with with University of Geneva to uh search for exoplanets um we were going to announce it at GTC but then NASA announced trapist one last week and we were like hey that's kind of the same thing as we're doing and they're also working with University of Geneva so let's join up um so for those of you that don't know exoplanets are uh planets orbiting um Suns outside our Sol systems um all the stars that you see on the sky are in simple terms s

um and uh yeah the data set is is extremely big I mean it's almost as big as as the Stars you see in the sky and but at the same time there are some limiting factors and issues for instance uh the gold standard set is limited to the amount of exop planets that we know about and even you know that kind of makes it that the the gold standard set down to fews and even like you know if you want to get data sets or gold standards data in the in the resolution that we're looking for it actually limited even even further down um so Professor lber will actually pass the thoughts over to uh Michel Meer um who is going to be the ambassador for for this new project so he will get an avatar in game which is pretty cool um almost as cool as him getting a wolf

price this year in physics which is also really cool um but yeah like AA was saying he's he Michelle found the first exoplanet uh back in 1995 so we're really excited of getting him on board and he will actually be coming to FanFest this year giving a talk a super scientific talk about exoplanets and so the data that uh we're receiving from in this phase of of prot is um are kind of these light curves and this is basically in simple terms this is almost like a time lapse of a star so you have a telescope and it takes snapshots of the star over a period of time and um when you kind of plot these up you get these beautiful light curves um and what the players are looking for are actually dips in this light C or and that's basically when something goes in front of the star star

and blocking uh the light from it um in science terms this is called a Transit so players will be looking at transits um and here you see a b representation of of transits these are like multiple transits and um of course you know this thing you know by now this is the trapis one uh light curve and yeah this is this is this is these are the seven exoplanets that were announced last week by uh University of of leash in collaboration with NASA and and uh all sorts of cool um un yeah scientific universities and stuff yeah well um but what's actually nice is that University of Geneva holds the prestigious last position um in the nature article that announced the find so we're super happy that we're working

with them um so why can't we use algorithm to find these exoplanets well we somewhat can use algorithms to find these exoplanets um there is a box fitting algorithm used uh to search for periodic uh transits in the light curves this method is called uh BLS or box fitting least squares I hope that's correct uh it's more or less like a periodogram looking for periodic events in the light C and Computing the power as a function uh of the frequency and then the algorithm folds uh the light C to find uh to the highest identified peak in the BLS [Applause] but there are some issues with it um it's great when you have like a single planet this works great when you have a single Planet but this starts to break really badly when you have multiplet

systems like uh trapist one and when you have circumbinary planets that you see here basically systems with two suns uh monot Transit events and uh transits that are M uh masquerad by strong Stellar activity so if you have lot of activities around so humans are actually outperforming the algorithm so there is still some hope for humans this is great um and there also a lot of interesting challenges here um that we're working through with the with the students from ra University they are both really interesting technical issues with uh you know how in the world to rent a ridiculous amount of data points in in a graph in E Line um how do you fold these light graphs and how do you actually like Mark them um and then how do you

mark multiple uh transits but there's also a lot of really interesting game design uh questions and I mean since we since we only want to host one project at a time um one really interesting challenge is how do you Sunset the human protein Atlas project and how do you do it gracefully and how do you kind of store the data and and you know Store The Prestige that that people have have uh manage to create for themselves um another interesting challenge is that exoplanets are really really really really really rare um so how do you make an engaging experience searching for a needle in a haast stack so that's that's definitely a fun problem to solve and and then you know we always you know we also want to create opportunities for Mastery and and that's super important so kind of couple of takeaways from this

lecture um this was actually a super fun project to work on um even if it's considered a serious game it's still entertaining and it is entertainment um and I mean the game I think the game is fun but what's actually also really great is that we managed to you know connect people that usually don't talk you know it's uh I've probably never Googled as many words from emails in my life it's I've learned a lot of words that I can never use but again but it's uh yeah my Google searches are very interesting no um I mean we were kind of one of the reasons why why this project I think worked really well with us is that we have a really engaged community and uh they really engaged with this project and they actually made their own subreddit about Pro Discovery uh that that kind of blew up there was amazing

and there was a lot of people like non-scientists that or non Eve players that that participate in this in this community so that's that's really really great um like I said before I mean this was a huge leap of faith you know trusting a third party uh to provide content for your game is a it's a very very very scary thought and it's something we talked a lot about in CP if it's like this is like one of the big blockers for the project in the beginning is like how can you make sure that like how can you make sure that that we can trust this person and uh but it you know it was unnecessary scale I mean the the project has you know not yeah it's it's been up yeah it's very little downtime and it's really good content um and we managed to we managed to kind of emerge the the content fairly

seamlessly with the game and the the whole game experience um and kind of apart apart from you know this being fun we found that there are two M main reasons why that kind of drove people back to Pro Discovery the first one was that you know there was like a big group of people that that looked at Pro Discovery as a almost as mining so you know there was ex exclusive rewards that you couldn't get anywhere else and you know you could sell them for a good profit and then there was also a very big uh very big group of players that are just doing it for the science and you know are doing it for the greater good and it's actually amazing the amount of people that have ridiculous amount of analysis credits and are not spending it because the reward is doing science and uh so that's that's really great um and also human protein Atlas did a fantastic job uh embracing the community everything from like communicating with people on

Twitter quite actively um to opening up their W and along people into the vickies and and you know giving University lectures in in the ingame University e University so we definitely want to you know this is one of the things that kind of drove us and AA to investigate Ms and like allow players to to dive as deeply as they want into the subject at hand uh um yeah like I said and also like what I said before that MMOs is uh what MOS is providing is not limited to sell images or exoplanets it's really easy to kind of imagine how this could fit into other games I mean here is something that's super straightforward I mean these are not actual projects this is just like sketches like bad Photoshop producer Photoshop like I call it uh I mean here you can see like you know ASU Sue Universe kind of analyzing uh

images yeah animal images from webcams um or like automated images or you know even cooler like if you could you know use your pip in Fallout 4 to to uh cure cancer I I don't know that would be awesome anyone from bethesa bling bling but all in all I mean now we have the infrastructure in place um and we can really start to kind of weave the the core story line uh of the game with Pro Discovery uh giving us more tools to deepen the experience and blur the lines between the reality of E online and the reality that we live in um and by doing this we hope to set an example and we hope that our our example sets a States for many more successful collaborations between science and gaming to come so yeah citizen science begins with you any

[Applause] questions I have a question actually yeah yep there's a mic in the center it's sure um it's been a little while since I played Eve but you guys do have a lot of uh sort of for the Bartles type players that don't want to do combat but want to do exploration and research there's a lot of options for that in the game there's like the probe ships that you can get that are sort of more specialized towards exploration MH how much did project Discovery bleed into that sort of exploration Play Type like were there ships that were better at pulling in data that were like research ships or anything like that and so we wanted we wanted the project to be kind of we wanted it to be something for everyone and we wanted the barrier of Entry to be as low as we could because analyzing proteins is actually a fairly difficult task um so we didn't you know for this initial project we did not kind

of tie it into any of the core systems um but this actually I mean they served us as great PVE you know exactly and it's actually it's actually great exploration content in a in a in odd odd way um it's really cool project I just want to congratulate you guys on the success so far I think I heard you said something you had like 250,000 protein samples I was just wondering if you could elaborate where those came from and how they were collected uh they they were coming from the human protein Atlas and it's uh it's a I think it's a 10 15 year uh research project and they are building an open database of of microscopic images so it was them who acquired these images they they have a state-of-the-art technology they acquiring like I don't know how many images per week but it's growing growing continuously cool

hi um I think it's super cool that your players actually were the ones that created that uh subreddit and I'm super hoping that there uh that the scientists on the team did an AMA that we all can read uh but my question is how did you initially communicate the uh the real world quality of this uh this project or mini game to your players like they clearly know what's going on and why it's important and what did you how did you tell them um that's a good question uh we had no idea how to communicate it like we're really good at communicating like kind of ingame inl type of um communication but like kind of this tie in between you know real science and and and uh kind of in-game content there was a lot of debate like hey how much should we you know downplay or up playay the real science PA because in a way you don't want to break the immersion of you know you're playing Eve and you know when you're playing pro Discovery in Eve

you're actually doing science for the science faction so there was a it was a very delicate balance but then it just spun out and you know people kind of started writing whatever they wanted I mean you guys have such a good community that I figured you did something on the forums or something like that first you didn't do that no we did like I mean we we had the we had like a web page and we yeah forums and video blogs and and on our own kind of kind of ingame uh News Network and yeah so multiple channels awesome uh Hey guys uh amazing talk and project and I was wondering if you can shed some light about how will you turn the exop planet stuff into a game play experience um yeah so I mean we are kind of in the in the initial steps in making it but there is there is a really uh so you're basically looking for these

transits and when you're looking for these transits um they should happen in a certain Rhythm so you're what you're looking for you're both looking for the dips and then you're also looking for um The rhythms of those stps so there is there is some really cool tools that you have to use where you're kind of overlapping the uh light curves on top of it top of itself and then aligning kind of using power the power of the of the dip you're you try to align the the transits um there's actually a second states that we're kind of look like the curves kind of you know they they have usually have somewhat of a sign wave so you know you need to also normalize that so there is a you know multiple kind of tools that you use um but it's I mean it's one of the things that we are debating a lot right now um and what extra things we will add into it um but yeah you are looking for a needle and a Hast stack so there is a lot of data that has nothing in it so

there is yeah there is an interesting Mastery element there okay okay thank you thank you hey uh during the when you pick the game you mention that you want a consistent play time so it then sometimes finish too fast or too slow is that because of the game play or more for rewarding because for City at home you mentioned right they had no problem dealing with different kind of CPUs because we can always weigh all the outcomes then reward fairly and even if the rewards are not different I mean people apparently buying crates trying to open them in CS go so I I suppose even random chance of different rewards may not be a problem so that more from game play perspective to make it more interesting or yeah it was definitely from a from a gameplay perspective and kind of from a you know we want to when you open up this thing we kind of want

want you to have the feeling that you know what you're getting yourself into like you don't want to we don't want to create a feeling of you like not knowing if you know you will do like two and that will be like 5 minutes or if you will do two and you will do like you know 30 minutes so we just the the consistent time was more to um yeah just kind of create familiarity and and not not scare people away okay thank you hi there uh my name's Brandon I'm a grad student at the University of Utah uh I make games that heal people so hearing this talk from you two is really inspiring um yeah what you're doing is making a huge impact uh really really exciting to see um so one of the things that kind of stood out to me as really important for making this work uh is the partnership between the university and N

CCP um so I was wondering if you could maybe elaborate a bit more on that like what value did um did the University see in collaborating with CCP and and vice versa too I'd love if you guys could elaborate on that a bit oh sorry what was the last thing uh just elaborating on the the partnership between the two and what value you each saw on each other yeah um I mean it's it's working with uh universities I mean you the good thing about working with universities is you get just a much more of academic approach to the issue rather than you know us game developers and you know we've been groomed in thinking in Productions and production cycles and like oh this won't work but you know so kind of research can you know is sometimes organized to the bottom when you when you're in production sadly um so we've learned a lot from from working with the university and just kind of their approaches to to the issue and how

they kind of you know deal with and dissect the the the issues at hand um kind of a bonus from that is that we've actually found awesome developers through these projects that are now working with us and uh so that's also just kind of helping I mean that's helping the tiny island of Iceland a lot because you know we have it's it's sometimes tricky to to low people to move to Iceland to work H want to why so no it's been it's been really good and you know there's a very uh we have a very healthy relationship with universities in Iceland MH yeah and um I don't know if you care to to share at any any about that relationship too well I think uh one thing which was uh really interesting and probably bero should talk about that uh we had a presentation with Andy and one big obstacle uh in in the in the very early stage of the project is how to how to

justify it both Insight is even both for the gamer Community you know this this is something that that needs a bit of focus so it means that other things will lose a bit of focus and uh players always have you know things that they want to improve and so she had to justify all this and this was a very lucky setup that we can say that a lot of the work was done by uh external students or interns in the company so we we could we could communicate that there is not uh huge loss of focus uh in CCP so that was I think that was really interesting approach okay thank [Music] you hi uh thanks for the talk first of all and I just want to ask like um I think it's great what you're doing because well the human brain is great at pattern recognition uh even to the point where

humans can recognize patterns where there are none did you take any steps to like avoid uh kind of fake data from people seeing patterns where they're aren know yes we did um so one of the issues with crowdsourcing and crowdsourcing science is are like Trends and you know if if if things go viral and we had the thing go viral very very early on in the project where people like to I mean they started to spam one of the protein categories called cytoplasm mhm um and we had to react really quickly and luckily it was it was you know really early on and we managed to kind of deal with it right away so we had to we kind of changed uh our reward structure a little bit um but this is always a this is like one of the top things you know you have to worry about in any uh crowdsourcing absolutely thing it's like if everyone decides to you know you know press a then a must be the right answer it's kind of a the the

issue it's one of the kind of the downsides of of crowdsourcing but I don't know I think there's we're getting really good data and it's it's actually um the accuracy is is relatively good when you compare to to um gold standard and and Analysis done by scientists so so just to adjust a little bit more on that uh in the beginning we we used uh two ways to Benchmark players performance one was gold standard that was uh that was pretty uh straightforward and the other way way was when we reach consensus on a certain image and we retroactively uh adjusted the scores of players and that was the that was a a small point where players could actually exploit the system if they wanted to plus cytoplasm is a location in the cell which appears uh more often and it's kind of a judgment call in many cases whether you mark it as a class or it's there but it's not

really a a signal so so all putting all this together we we had this thing which we called later cyop pism scam and then we quickly quickly really just uh removed that that kind of um uh thing in the in the system where we where we retroactively adjusted this course and we just we now just use uh uh gold standard which is actually it's interesting because in many classical citiz and science projects they use they actually use this approach to Benchmark uh users performance but you know games are a bit different yeah great thanks say keep up to good work thank you hi uh a you might have just answered this question partially with that response um but with project Discovery you uh talked about your accuracy being measured against all the other submissions uh from crowdsourcing you guys blew up pretty quickly within the first 24 hours so maybe this wasn't at

the end of the day a problem but did you have any concerns or considerations within the first few hours or the first you know few days on not having the database like to to base your samples against um actually the first few days the only consideration was for to keep the system running so they didn't have time to analyze the the data quality uh but I think already at that point we knew that it's uh it's going to be very successful uh from the perspective of activity so it gives us uh more space to maneuver uh so that's that's why we focused on first to find the glitches in the system and the bottlenecks and then it took like two weeks and then after that we we could we could focus on the quality but this this cym scan was so obvious and so everywhere in the forums as far as I remember so that actually

that we corrected in the in the first week okay but I mean this is also where the consensus of the consensus was quite interesting so we could see after after the first 3 weeks when we reach consensus when we got the second consensus we could basically look at the two data sets together and see hey you know how much offset was in the initial Spike and you know how much uh Badness or goodness you know was the initial Spike but they were relatively consistent the at least the first three consensuses it's really interesting because in In classical citizen science platforms they're try to optimize more or less the the the the usage of Citizen science resources so they try to find the try to use algorithms uh so that you can reach the consensus for relatively low number of of uh of votes uh whereas here now we are improving a plat form that we can kind of automatically put on the Queue the same

task that has been solved because it's an interesting thing of course we try to optimize the the the the the solution of one certain task but then you know it's not possible to change uh like research project in a game from one day to the other whereas on the classical platforms you just close the project and you choose another here here we have to keep it running so here here even if they solved it in 3 weeks we have to keep it running for half a year so that's that's also an interesting specialty of this approach yeah thank you guys thank you thank you all right well thank you very much thanks for coming