Real-Time Data Processing for Multiplayer Online Games

[Music] hi everyone we are very glad to have you at our speech today our topic is real time that processing for multiplayer online games i'm leicester a senior dead mining engineer from sound fire ux team of netherlands games this is my colleague certain shoe a senior project manager also from sunfire before entering today's topic let's think about such a scenario a new game is launched today your boss is eager to know today's in-time new players and cash flow can you give him this data in real time or a batch of new items have been launched can you let the boss knows which are the

five best sellers now finally these new items were copied illegally by some players can you detect it in the first time before the boss goes crazy and yells at you why didn't we find them in time to solve these problems we need a real-time debt processing system therefore we were many discussed three questions in today's presentation first how to build a real-time debt processing system for online games second how do we process real-time data in netease games and third how to optimize real-time data processing most of you probably have had little

knowledge on big data so i'd like to introduce some big data tools that are frequently used in practice before digging into today's topic firstly kafka kafka is a distributed streaming platform which can be used for lock transferring and temporary storage storm and flink are systems for real-time data computations and druid is a real-time analytics database the specific functions of these tools will be explained later in addition there's a commonly used eok system in order to search and query the logs and eok is actually the abbreviation of the three words elastic search

logstash and kibana there are tools of log search log collection and visualization this picture presented here shows how elk works logstash collects passes and filters the data and then sends them to elasticsearch for indexing elasticsearch is used to index store and extract your data kibana allows you to visualize your data stored in elastic search helping with data searching and data analysis by far we have had an overview of the big data tools so now let's begin our first topic how to build a real-time data processing system

for online games corresponding to the real-time data processing system is the offline data processing system so you may ask what is the difference between these two systems a main difference is timeliness offline data processing system is not immediately and usually causes delay for hours or a day as we often call it t plus one while real-time processing system reduces the delay to minute level second level or even millisecond level secondly the processing modes are different offline system uses batch calculation while real-time system uses micro batch calculation or streaming calculation

in terms of results consumption offline data processing has a higher demand for disk and network io due to the relatively large amount of debt processed real-time data processing requires higher cpu and memory usage for processing real-time data finally they each have different representative technology for offline data processing these hadoop hdfs and mapreduce hive wire for real-time data processing their spark streaming storm flink and so on now let's look at a typical real-time data processing system as

shown in the slide the source of it is log collections and the transmission then transferring and a storage stands at the middle of the process while computation and application is a final step the corresponding tools for this step are lockstash kafka and flink this is a brief overview of the data processing pipeline and we are going to have a more detailed analysis of it players in the game will have a variety of behaviors these behaviors will be recorded by the log system on a local game server for example some typical behaviors such as login and logout duplicating a character

paying for items and the use of items etc in addition to server-side behavior some client-side click behavior can also be recorded such as the operation of starting the app in order to achieve real time processing these logs needs to be transmitted in real time we can use log collection tools such as logstash or flume to transfer logs from local to a stream processing platform like kafka kafka is a transfer station for the entire real-time data processing system it plays a key role in the system on the right side of the slide

we can learn that kafka works as a transfer station for their game data of all kinds that are written into kafka from producers after which various applications start consume data from kafka through this the whole data processing pipeline is built and the real-time data analysis is achieved kafka's advantage lies in its higher throughput and low latency it can possess millions of messages at the same time and has local storage and a data backup so it has a certain degree of fault tolerance node failure is allowed as a distributed system kafka is very easy to scale so

you can add new nodes without stopping the server the last step of the real-time data processing system is computing and application earlier in our presentation we have introduced the elk system which is a typical log search application however in daily bi applications we will more often deal with database systems and encounter the following two processing modes one is to process the data through the engine like flink or storm and then write into the database system the other is to directly transfer the data from kafka to real-time

data analysis database like a druid for online analytical processing so called olap what is flink flink is a framework and distributed processing engine for stateful computations over unbounded and bounded data streams flink has been designed to run in all common cluster environments perform computations at in memory speed and at any scale for the advantages of flink first of all it has the flexible windows such as tumbling and sliding window this will be explained to you in detail in the following cases with my

colleague secondly there's another important feature which is the exact ones the exactly ones means that all data flows over and the data will be processed only once this is very important for in streaming systems to avoid debt loss and repeated debt calculation finally in order to deal with possible service failures we may need recover to specific time or data points the checkpoint and step point mechanism provided by splinc can solve such problems very well the flink programming models as presented in a slide can be divided into three modules

the source module receives various kinds of data source the same model is to output all kinds of storage the transfer mission module in the middle refers to various operations of data transformation such as map reduce filter and so on now i will introduce you my colleague sergeant shi a senior project manager in sunderfire for the next part he will show you a lot of used cases in netherlands games thankfully let's start with the accomplishing case as an example how to calculate the key measures such as new accounts dlu and revenue following what we have discussed before the first step

is to build a pipeline we use log dash to collect log kafka to transfer it and link it to calculate and analyze it the result of computation is written into my circle database then we can get a desired result through carrying the bad database in order to calculate these indicators in real time the first things we have to do is to record sound log behaviors and to give you an intuitive feeling we have listed some log examples for logging and logging out and paying the very first step is to transfer logs to kafka here we take logstash as an example a standard logstash configurations include input and output on the slide the input is a local file and the output is kafka with such straightforward configuration we can achieve our goal

finally we use flink to consume the data in kafka and write the result in 3 my circle data tables new accounts clu and revenue respectively based on which we calculate the data we need the example hopefully have has clearly presented you with the way a real-time system is built up along with how it serves for our application in the next part i will introduce some more complicated technology analysis based on flink programming models we can achieve real-time etl very conveniently the so-called etl here refers to a conditions that extract the transform load is in the exact match with the source transform sync in flink through syncing the data into so creating external tables in hive we

can create the tables for analysis in a very short time the example we gave before is a typical real-time bi application combined with fleeing tumbling time window function we can easily know some key elements and monitor some in-game items and currency follows for instance if we want to monitor the turtle coin acquisition of each account every minute we can use a fling programming mode to achieve our growth first group the data by account then use tumbling time window and set the time window at 1 minute and finally we calculate the sum of the coins however if the time window changes to the latest hour by now then we should choose sliding time window rather than tumbling time window and then we calculate the total number of each account and make a top k so far we have clearly

understanding of how flinx programming models works however we wonder whether we can achieve a real-time data analysis and query without flink here we will introduce another tool druid to give you another way to solve problems the data can be transformed directly from kafka to three and then query based on trade so what is drip drip is a real-time analytics database that can store much larger data volume than my circle while still maintains very fast query performance with read we can no longer need to write flink codes over and over again to implement the previous case but only to use sql to query after the data is loaded into it the analysis process becomes more flexible

in this screenshot if we want to check all payment data in the latest day by far all we need to do is just to query the database directly and we don't need to worry about issues such as performance for this more complex top k accounts problem here we put the corresponding derivative query which is fast and direct and has no need for programming ability it is suitable for data analytics to analyze the data quickly finally let's look at some more special case we need to monitor some special keywords in the game let's call them backwards here firstly we can monitor the real-time chat data then with the water segmentation techniques and backward dictionary reset we can quickly monitor the bug message

in the game thanks for certain wonderful speech by far we have solved two problems i mentioned in the very beginning how to build a real-time data processing system for online games and the real-time data processing in native games in the section to follow i will present a more technical problem how to optimize the current system how to tackle some detailed technical issues an important indicator of distributed services is high availability which means how to automatically restore services even after some kind of node failure such as configuring flink on h.a young and setting up the system

checkpoint etc also flink can guarantee processing exactly once but if the upstream data is duplicated blink itself cannot avoid dealing with wk data we need to pay attention to the logical deduplication which is the so-called idiom potential calculation in addition there are some more detailed questions for example what if flink's practicing exceeds the number of partitions in kafka well we should use rebalance option also how to ensure the usability if we restart or update flink then we should use save point

the future or real-time data processing system might from my perspective focus on the following three points firstly real-time olap is a very promising direction as it can reduce the complexity of programming secondly we would continue to improve the performance and efficiency of the real-time processing system last but not the least we could work on monitoring the real-time processing system so that abnormal alarming is issued in time thanks to you all for listening our talk