Intelligent Music For Eve Online

[Music] well I think we should get started so hello everyone first I would like to ask you to silence your phones because we have some music to play here and also if you have questions at the end then ask you to walk up to the microphone in the middle and make a line if there are many have questions so thank you all for being here and thank you to TTC for accepting us here with this talk about intelligent music my name is pal - pal - zone and I work as an audio director at CCP games mainly working on EVE Online if in case you

don't know what even line is it's a sandbox MMO with some twenty to sixty thousand players playing at one time on one server and with me here is dr. Catherine olesen composer and professor who's been working on procedural music for many years more than 20 years and we're going to be talking about something we like to call intelligent music now let's find out what we mean by this so first I'm gonna do a short introduction on EVE Online and why we seek better ways of making music in this game then my colleague dr. captain will

introduce his system he calls Kalmus and finally we'll do some demonstrations how the system works together with the game also before I start or four we start I think it's important to stress the fact that this is very much work in progress so please bear that in mind we start with the EVE Online so the music has been part of the game from the beginning obviously like for most games there is music and from 2003 to 2009 we had a thing in the game we called the jukebox which was really just a playlist that played a number of old tracks specifically made for the game this actually we still use this music and we have added some and we still play whole

tracks for part of the game this is really really simple it worked well especially since the game was very very atmospheric and it worked really well just to have some music playing that was very atmospheric and didn't change very much so before 2009 we had a bunch of solutions handling the water in the game most of it was home prune so proprietary solutions hunting audio this was obviously very expensive and time consuming and in 2009 we implemented twice which changed a lot and for music we were able to add precomposed elements or music we can stack up it and every way we wanted and play in any number of ways we want it and so on so it was so great to have wise as part of the whole system so in the

game there are many different systems the lack of a better word so we have Empire space which is pretty safe for plays play start out there and they can go out and they can mine and to whatever they do when they start the game it's pretty safe the police is that you're not attacked the music can be really easy and you can play health tracks like you have done and the the system the new eating as we call the space in EVE it's called New Eden it has more than 77 thousand solar systems with I don't know how many moons and planets just countless almost and

most often very safe to start in Empire space but if you could down to low sack and down to zero sack and wormhole space it changes a lot so usually it's not recommended for players to wander around and go to these places like low second zero in the beginning at least so they stay in Empire space a lot so the music in the game is based on these systems so an empire we have it easy going in low sack we have the similar to Empire space but it has some more you know difficult intervals in the music maybe and you know the temporal temper is different in the instruments and so on in Siro sack

we have vertical really almost sound mass music with wise place in any way we want we can stack up whatever we want there and it just plays sound masts kind of thing with classical instruments and the wormhole space is similar it's even more sinister so it's it gets worse as it gets a lower so this is this is fine I mean it this works up to a point it sets the state in most cases unless some players or decide to change things because it's a sandbox we don't know what's going to happen we haven't set the story so the players are really the ones that change the things there so we don't know what they're going to do we have no idea so for example in my space you think it's safe so sometimes we have

kamikaze pilots flying into you and you you're dead or some corporation declares war on you and you're just trying to mind and mind your own business so that should change the music as well but we still have the same kind of music that it's easygoing and so on and likewise in lowsec or or Siro if there's a big corporation there that has a station there and a lot of players it's not very difficult for them to stay there so maybe the music is too much in that case for them to be like this so let's hear what this sounds like just so you get an idea this is my space where everything is

final and the mine fly-around hope your friends have fun and so on all the music that is like easygoing you listen to for hours nice then lowsec similar but little more sinister maybe gives an idea of well maybe I shouldn't be here it's not very frightening a little bit at least so it tells you something then if you go to Cyril a sack it's like this this is like

clusters of things that you hear that are stacked up randomly in advice it's very nice feature you can do any number of a slant only and you shouldn't hear much of our repeated segments but it still comes up but it's it works really well up to a certain point of course and then we have wormhole space which is if you call the New Year this you you probably have just feel just frightened do you want to go back back so that's what it tells you you know flying around and having fun here in the senior years like shouldn't be along this

so we basically want an intelligent music system because we can't or maybe shouldn't really precompose all of the elements in that game that or a story that hasn't taken place is it really fair to a composer to compose everything in a game that hasn't does never store it hasn't happened I think we should find ways of making this better also hearing the same piece of music over and over again diminishes causes diminishing aesthetic return and sometimes just makes us ignore it completely completely so even if you had a budget of composing thousands of minutes of music which is like a million dollars or something or

two million I don't know it would still happen we would still hear the same thing over and over if we have a king that plays for more than ten years that's just going to happen we will end up with the repeated segments because of that so that's why we want an intelligent music system we want it to be able to create musical material by itself so with some help from us of us of course as composers but it should be able to create the material according to the atmosphere of the game though it should evolve according to the actual progression in the game in real time it should be possible to control various musical parameters in music simultaneously this is our race list but we made this suit this is what we want

so the first steps in this direction we took in 2009 this wasn't like we have to have this tomorrow we have to have us now not gonna happen and we knew that that was not gonna happen so it was a side project we started looking at what others had done such as EA with Spore and some other nice projects some really interesting things we took a look at max/msp and PT how if we would use that and so but we want to have a system that can create real music not just atmosphere so it made it a little bit more complicated we started researching with dr. Catherine Allison use a professor at the Atlantic Academy of the Arts at the time to find better ways of playing music in a sandbox game we

started with the beginnings of music with the basics of music intervals tempo rhythm harmony etc and then onto methods systems such as set theory probability distributions and other methods in order to realize and identify what kind of system could work then we took a look at Catherine's own system which he called Cal MERS that system had he had been working on for more than 20 years he which he wrote to be able to make contemporary orchestral and instrumental music for himself really and now I'm gonna hand the mic over to captain and he's gonna tell its part about calvess hello everyone I'm going to be a little

bit about this system and and why this system became a composing system when I was studying at the CBL is a khadeem in finland in in the 80s we got the opportunity to work with both professional composers and programmers and at that time it was possible for for a normal people to learn how to program with personal computers and scenes in music composition the mathematics has been used for at least thousand years as we know but most of it was from the last century we decided to consider the program as my thesis in my doctoral studies there and first of all the idea was to compose - to make a composing program that could use musical elements

and materials as we are doing in with pencils and papers and use this these new tools to make experiment with music try to find new ways to compose music and in the end we had this program that could compose music or for Orchestra for chamber music electronic music and now today we are focusing on real-time composing which is my interest actually but when the CCP people came to me and we started to talk about this it was incredible that we were thinking about the same things a composition for an orchestra is sometimes called three level composition in computer games music it's also three levels and because

this system is working with artificial intelligence it was obvious that this could be used in because artificial intelligence which I will talk about little later more is something that can handle new situation with new preferences every time every step in the process so a composition with columns is based on musical object and we are talking about melodies harmony and development polyphony and so on which is a common concept in composition without computers as well the parameters we are working with are themes which we create melodies from harmony that will decide what kind

of connections are between the melodies scales which we knew know mostly today we are using major and minor texture polyphony etc and the musical material is quite traditional cells which creates the melody and sorry and harmony that will connect them together and polyphony strokes is how they actually are talking to each other the melodies the algorithms we are using they take care of putting together tones for harmony and for melodies and we are using in a very traditional way different scales and the scales in our time they are reflecting a different geographical position like we have Japanese scales European scales and so on and and the

artificial intelligence system makes it possible for the system to go on to compose music by itself but as we are doing it the composer using the system is defined in the framework creating the material and deciding kind of color or texture it will contain so a one simple example about artificial artificial intelligence when creating a new pitch in melody construction and as you can see on this picture there are graphic and notation there are four nodes that are called last pizzas plus test pitch and this these pizzas they are compared to the input cell from the

composer and the program is deciding what are the relations between this new bits in this new melodic line according to the actual input cell from the composer and what happens when it finally is accepted after having being compared to what kind of scales what kind of interval are preferred then the new bits is accepted and there's a new situation in the male construction and that means that the program has to continue to make another selection according to this new situation and this is called artificial intelligence in music creation the same things or quite similar things are happening in the computer game like samples as they are working with working on it is something

happens in the game and a new situation have occurred and then the program like this can actually decide we have to change the musical texture and prepare for another development of the music and to do this we made the several project just to connect this computer composing system to the actual game and the first one is called canvas way services we using dances to control the program and this column's program can be controlled its parameter can be changed in real time which means that we can take music from the 20th century change the harmony and the polyphonic structure and we can go back in time to the 15th century by

doing this without changing the material and to do this we created constructed a sensors connected to the dances and the dances are sending message to the program that will compose according to their movements this is what we can use in the computer games taking message from the games and use the message to compose music according to what is happening the second thing we did is called canvas play with this kind of app for fun to create very simple music and what this is doing it's this simple app is communicating with the program a special version of the program which is on a server in Europe in Iceland and this small app can compose music so if I

press this button it will send the simple message to the server and compose the music and send it back this is composed in Iceland and this is how we are just trying to communicate with the server just through TCP to through the satellites and telephone system and then a small example from the actual use of the program with this program I have compost and other composes different kind of music and one of the music is called viola concepto which is for a full orchestra and and the solo viola and I

just play a small example from this piece it's the end of the piece is a live recording from the Icelandic Symphony open [Music] [Applause] whoo [Music] [Applause] [Music] [Applause] Oh [Music] yeah

and this is a screenshot of the Excel program and I'm trying to change now so I will go into the program and show you a little bit bit about how it's possible to use individual parameters so I have to leave this and so it comes here so this blue box is actually a musical object and it's containing a couple of melodies and it can be played very simple things from it and it's possible to enter the material from the composers through MIDI files to play play directly into the program and to export as well

as a MIDI files but this this interface is made in Xcode it's communicating with the actual program the same program as is used for Thomas play and for this computer game is only communicated with the numbers so when I press the button play it sends the number 50 to the actual brain which starts to play and all these parameters we can see here very traditional deciding what kind of intervals we are using in the Harmony what kind of polyphony we are using and polyphony is kind of if it's canon or if everybody are playing together and this is very traditional for a composer to work with and then we can decide in the melody what kind of scales we are using and whole tone scale is very well-known fantasy and other composers and here it

can be changed the scales we can do this one when it's red it's composing it's within a second it has changed now we have a different situation and of course all the instrument can be selected here or any electronic instruments and [Applause] this is the editor for the material can that can be used in the computer game for the composers the game can then afterwards use this material and use it in the game and that's about it from me so part of you will continue thank you

yeah so it's nice to have a fancy system like calibers to work with to help you create the music in the game but we also need to know the game that we're working with we have to know what to look for in the game since we don't have a story we have to find other ways other means of creating music so we don't have a single character or a story to build a music on but we have other things in EVE we have different races of people we have different environments these are things that we can use we have wormholes we have cosmic signatures and other anomalies and we have other players as well as some NPC players as well but there are no scripted stories at all

so we have to work with these things the game has territories so it's when you start that it's race has a certain territory but they start in so four territories that you get it you from so instead of connecting to a single character which is usually a pretty straightforward for composer to connect to a type of a character and composer music foreign we connect to the whole race so we take the character and connect this karate to the whole race and we connect this what we created to this whole territory in the space so it works in a similar way so we have four basic races that we get to choose from when we start in order to understand how the

music can be defined for the races it's vital to really understand and study its race on its own we have a more gallant a cult re and Minmatar these are what the races are called and all of these have a different kind of backstory or or history which is written doesn't really mean a lot but it's there and we can use it as composers and as well they look different so we can use that as well and as you see we've found some ways of defining the characters a modern noble recal a little bit evil they were slave all this I want talent here smooth and curvy the French cult re are hard and edgy cold very technical and Minmatar arrestee and a bit dirty you know rock

and roll almost only also has men pc races Jove there are mysterious and they're hugely they're advanced to any other race we don't really know what they can do there that is like gods almost there there we can use them to create some music for situations or have our system create music for situations with a minute and we have some pirate faction's as well many different palaces factions actually so we can actually have many different kinds of things for the pirate faction's because they are very different they also have a backstory was a lot of material tool to be able to be able to pace the music on so to speak so six to ten races that we can define for music and we can start creating

something we would like to call a character cell for its days and for the for the game it is just make a specific library let's look at the amar the first one is regular noble a petty will perhaps some would disagree but I don't know I feel a little pity you know the pittypat the principal melodies I mean this this could be anything it's up to us to define it as composers so 36 scale to orient with the raised sixth on the minor scale around see to the pits latest e and the speedrun tatata or something like that and polyphonic structure is like this or it could become non horizontal vertical at this distribution and the instrument text it

also is important so we haven't really made set any of this in stone we just we took a look at all of them and it also really matters how they sound it's not enough to think this well yes this is this guy is really sinister it's a minor second definitely well it has to work in the game with other things as well so we have to try it out so within the process of doing this now and yeah the instrument actually mixed meaning classical and some electronic instruments together and now we have to find things in the game that can change things in the music so what can for example intervals changes in harmony can be changed by various tensions in the game I mean columns can be chained calvo's contains interval t2 interval

tenses in the Harmony depending on various tensions in the game in real time this is important also type of polyphony complexity and conflicts this could be anything again we are just in the process of defining it so it's really exciting to be able to do that change in tempo this tension as well and matching threats more tempo it's a threat coming there's more tempo it's really it's really straightforward if you think about it it's similar to if you just compose with with a story so we have an example I'm going to play an example at last for I think a lot six minutes but it's it's nice I put some explanations and I'll talk over it maybe it explains how the music changes while you play and also let you hear how the

program composes in real time it's a video it was supposed to be plan B because we weren't sure if we're going to be able to play live here but it sounds or looks like we are able to play life so we're just gonna play both I'm gonna do Plan B first and then we're gonna listen to plan a in the end so we'll have it real time over the server so this is Ammar again that we defined you'll hear the definition I made melodies thirds and sixths I have four what colors can do it can forbid some interval so I have forbidden intervals just for normal - ekans and major sevenths which is basically the same thing but the torii end and the Tadzio and the mixed intrumental selection then at interior occurs and the interval changes changed really because it allows

every interval to be played and it gets a little bit lower and the instrument get a little bit changed as well but the same melody again then we'll jump into another system and there you will hear I'm a different thing but still being composed in real time so let's hear what this sounds like [Music]

[Music] this is all singleplayer so there's not much happening I will find some parrots attack me but that's about it [Music]

[Music] [Music]

so then we have the intervals that are no interval span any longer and the the instrumentation is taste

so so now it finishes the what it is playing and finishes the this is our segment and starts the next one when it works out but it finishes this one first it doesn't happen right away could they let it happen right away but that's not that

again what we started [Music]

now we're in the calendar system during different single together you [Music] of course everything is done in real time so there's no editing done here will tell me that now we're gonna play some

real-time plan a [Music] so this is also on a local server suggest one player maybe with some NPC pirates attacking or something but this is in rhythm now / began maybe let's do mr. pasetta No I can accomplish this written in Lisp it runs in ECL which is a an open source interpreter for Common Lisp and we worked with OTO kinetic to open its API so we could send MIDI events

into the sound engine was really important but now it's open for everyone there just including it in there in there wise I think so commerce feeds the meteor vents into ice which hosts the instruments we have the sampler in and wise play the instrument [Music] it's probably would probably have some percussion and stuff like that there it's all work in progress but it's all in real-time and the potential you can hear the Ponte pant ends I was there and

it changes while you play the game [Music] [Music] [Applause] [Music] [Music]

[Music] [Applause] as in traffic since it's a local server if that's [Music] [Music] so there we are I don't think we have to if you want to if you want to hear this take a better look at this we will be at the audiokinetic booth later today after this set this up and demo it you can listen for your own with atoms maybe but for now I think that's that's it

so in summary for fantasy MMO where the story hasn't happened we'd create leap refer non precomposed music some come to some kind of a system is needed to think one such system is Kalmus obviously but no system can work without first really knowing what you're dealing with no in the game you have to know the game that you're dealing with what to look for and currently the system runs outside twice but ideally we're gonna have it on a plugin so we can use it with with other games that we're making so thank you and some questions hi thank you so much that was fascinating to to hear an excellent explanation about and to watch in action thank you so much I have two quick questions first of all with the

percussion that you mentioned just briefly maybe you could talk some about your vision for how that could play out with the different territories or especially with the different races within the game I think with per question and you're talking about percussion yes I think mainly with when there is danger in your attacked that's what we're starting with we're not going to tie it to to erase to start with at least so like I said it's a it's a game it's in an in-progress thing we were still trying out things and to start with I think we're gonna type percussion with when you're attacked and stuff like that yeah my second question is about forbidden intervals I was maybe a little bit confused and I was wondering does that work horizontally or vertically yeah it can work for harmony so what I was

showing you was for the harmony mm-hmm so it was not the the melody so you can for pit in tools and melody as well but I didn't do that so this was all for the for the harmony thank you so much all right that sounds like a fantastic system so great work on that and I've got two very quick questions so it sounds really fantastic to be able to have this great sprawling soundscape that never repeats and kind of generates itself on its own but as an audio designer and a composer sometimes there can be a really great value in having a repetition of some sounds in terms of feeding back to the player when they've done something in the right way or feeding back to the player and this is absolute game over situation or this is you're absolutely fine and first something that shifts so much and can be interpreted in until it's settled on

what it is do you think there's still space on a project like your own for a composer to still have I kind of finite this is what we require for certain times of the game are you intending on having the whole thing to be open up free yeah so I thought I understand your question also hoping for this question actually okay so I think the the system can be used for as a hybrid system as well you can tell it to play a certain thing and then you can compose your thing in the same scale and have it play over it also about repetition we can have it repeat but it will always change the tiny bit so you can tell it there's a certain slightly left that you can you can move which called the rate ratio of truth so if you move it to a hundred percent it will play what you ask it so if you have it like 50 percent it will

kind of play what you asked yeah you know so it'll play something else as well so we plan to be able to use it you know with if we use it as a hybrid system you should be able to yes definitely fantastic one last little question just something I was wondering and sometimes when I'm using MIDI instruments I find obviously they have their limitations it's like a violin can only go so high and so low and what have you have you had any problems with that with the MIDI instruments where this tried to compose something outside of the range of the actual an instrument so can see annotation now so maybe Catherine you can maybe answer that that's really that the system really knows how to deal this yeah actually the program is made for a traditional instrument and it has 16 MIDI channels and the MIDI channels assigned to the traditional instrument of an orchestra I mean Symphony Orchestra so number one

is the flute and obvious 16 is the double bass so it's there is a kind of sections in the classical Symphony it's one to forest woodwinds and five to eight is the brass and then there's a percussion and the last five are the strings two violins viola viola cello and double bass but the thing that we are doing in this is to construct sounds that are that are similar to for example for a flute and the program can take care of the woodwinds and take care of their harmonies the different harmony that you use on the woodwinds and on the on the lower register and instrument so that's the that's the thing that is we are creating artificial were not artificial burn electronic symphonic orchestra so we can apply all the functions we're actually made for that for the traditional symphonic Ossa on electronic sounds as they are creating

in ccp so it's taken care of all the ranges for the instruments and playing with with ranges of each since instruments together awesome great job guys hi there I was wondering so it's obviously very early of what you're working on but are you planning on expanding the palette of instruments that you're working on I know you just mentioned that you're using all just like a regular Orchestra and stuff like that have you run into any problems with the program itself like as far as mixing like the instruments together or having like playback issues with that and having it mess with the game audio itself what is playing back yeah we I don't think we're there quite yet I mean we are playing around with instruments we're creating instruments ourselves and hearing what works I mean we've had some problems but we haven't really we're not really there to to tell you this is how

we fix this and this is how we fix that no we're not quite right there but as captain stated before if you are a composer you know and you know how to use the system I think you kind of need to be a composer if you want to use the system not just anyone can just step in and just use it it's just it's not maybe captain disagrees but I think but yeah he wants to add to it but yeah we're not quite there but you know the instruments know where they should be yeah I can maybe add to this all the functions in the system are made for musical purpose it's like controlling harmony controlling the tension and color and melodic development and this is very important even users are not educated on the highest level and composition if you

have a good ears and you have a good sense for music you can learn from the system because it's always react to what you're doing so it's a kind of new way of composing music you are dealing and communicating with a composing system and experimented with this thank you yes I also think that it's a great way to at least for me to get ideas just put something in and it spits something out in and like our second to something and you like amazed I didn't think of that that's that's a great idea that's one thing about the system that I love yes hello thank you for the talk I was interested you know a bit more about the technical side of how you you two eyes as a sampler because sample sampled symphonic instruments are quite complicated right you have different articulations and different timbres depending on for example the the

intensity of the breath and then you should have different samples to avoid repetition so I guess this is a lot of work so I wanted to know how deep did you go into that sampling aspect in wise we did not go deep it's very shortened and to it we just took some samples and yes we did not go deep we are thinking about this okay and we are gonna probably start the project just about this because it's really important i SEL think that's almost impossible to play classical instrument really fast by Machine it's it's at least it's very it's not it doesn't sound right to my ears at least so I want to really work on that and we know we have we haven't really worked on it maybe you'll work with someone that is already doing that we don't know yet like I said it's all work in progress these are the examples that we put in but remarkably you know I think it sounded remarkably well just using the

basic sampler in wise and just playback okay and if you want to go deeper into that would you need to redefine a bidi algorithm to integrate more parameters like for example dynamics and and stuff like that or do you think it already has everything to handle that kind of deeper sampling technology you mean the samples yeah yeah I mean if you really go deeper into the like having more samples and more parameters we do need to rework a bit on the algorithm to integrate those new products so yeah we're not there yet okay yeah we're not definitely not okay okay so like I said that's a that's a material for a whole new project I think cause it's so important to to have them sound right thank you so much yeah hi thank you so what the plan for the evolution of different classical styles so I you gave a really

great example and this is working really well for Eve and I'm wondering how deep do you think the technology can go in the future for can you can you make it sound Baroque or can you make it sound like John Williams or like how how far do you think we can go with this AI composition here so if I start maybe maybe captain can continue when when we started with this it was able to play back for us when what we put in it played back really complicated 20th century music that's not really what we're looking for at least and in mostly with you looking for something more simple so we're quite I think we have quite made a lot of success and making it make create simpler positives of

music as far as you're talking about maybe you can compose John Williams or something like that I don't see why not if we create the right cells for it and the input has to be right the and it has to be has to have the right tempo and so on then I don't see why not maybe a Cattleman's to add to it yeah maybe the methods and the alcoholism in this program they can handle different type of music and you cannot do our kind of Bach fugue in this or you cannot it's not meant to imitate styles but you can certainly get the color of the styles like you can control the harmony you can count us very consonant based on triads using melodic cells that are very connected to the Classical period in the

music history so you can get kind of color of it but it's not math to do too too imitate and I can tell you something because I have used this for teaching contemporary music at the academy for many many years and obviously the students they can after some time they can actually control the system in in the personal way which is very important and I was once very surprised when one of the students could actually make a minimalistic music which I didn't think was possible but adjusting the parameters and knowing they're the symbol and the character of the music you can get there me hi thank you for the presentation how many the scientists so I was pretty like fascinated about aspect about this so I was wondering if you guys ever consider using player in Co to train the

system for example if you play something doing a combat and you'll have a button to click if this is if the player thinks a match is moved I actually feedback to the system of the AI and a trainer system that way have you guys thought about it before no we have not but I don't see why we shouldn't be able to do something like that we're not there yet but I don't see why not I don't see why the so we create a cell of maybe 8 notes for Kalmus to compose music - or it reads his eighth notes and creates music I don't see why not the system or the game would be able to create these eighth notes somehow I don't see why not but we haven't tried it captain wants to say so very sort the system is listening to the

user so whatever the uses is - it's giving to the system the system is and this is the perfect thing to work with but the artificial intelligence is is used in that way it will use the material you put in and continue to to compose with it for the memory for the learning it's we are using kind of library which is a memory and the system can save it and call back these memories and this is mainly for for the composers to create the personal the personal style with the library as we all have in our summer in our mind and yeah this is this is the functions in the system can actually if we put it on and let it compose it can kind of continue forever it can select from the library by itself but also the composers can come in and change it in real time if if needed okay

thank you so thank you very much for coming it's great to see here thank you