Face-to-Parameter Translation via Neural Network Renderer

[Music] Hi everybody. I'm Tianyang Shi from Fuxi AI Lab. In this talk, I would like to share our new technology named the face-to-parameter translation, which can help players create their character with single photo. And this technology can be used in the games with character customization system. Fuxi AI Lab was established in September 2017. We are the first game AI Lab in China, and our vision is to ignite games with AI. Now, we are serving our games based on various AI abilities, and this and this talk focus on the topic of computer vision. We also have a series of talks in the in this year, and we welcome you all very much. Back to this talk, it has three parts.

An an introduction of character customization system, the details of our algorithm, and some some discussions. Character customization system is a common system in recent RPGs. It allows players to edit the profiles of their in-game characters. For example, in the game Justice, players not only can can adjust the facial shape by controllers, but also can choose various hairstyles and makeups for their characters. However, it is very consuming to create a perfect character with hundreds of parameters, especially when players hope to put themselves into the game. In this talk, we focus on the facial part and explain how to automatically

create characters for our players. Obviously, we need achieve the following goals. Firstly, we need to generate a set of parameters based on the input photo, and then these parameters can be recognized by the game to create corresponding character. Most importantly, this character should look like the input photo, and we also need to meet the requirement of further editing. Now, let me explain how we design and implement the character auto creation. There are two challenges in character creation. The first challenge is how to solve the problem that the game engine is non-differentiable. If the game engine is differentiable, we can use a powerful tool, the gradient descent method, to solve this appearance matching problem.

On the other hand, it's a second challenge to find a suitable metric that can measure the similarity between the input photo and the created character. Generally speaking, we adopt three convolutional neural networks in our method, and you can see them in the pink and green trapezoids. In phase one, we use the generating neural network to imitate the behavior of the game engine, which means it can generate the same facial images based on the input parameters as what the game engine do. In phase two, we use two discriminative neural networks to build our feature extractor, which defines the facial similarity between input photo and the game character on identity level and shape level. Because the neural networks are

naturally differentiable, we can combine these three networks together and use gradient descent method to iteratively optimize the facial parameters of a character according to the input photo. Now, let me Now, let me introduce how to make the game engine differentiable. Facial parameters are the bridge between players and characters. In our method, we define tens of parameters, which are continuous parameters and discrete parameters. The former ones are the combination of bone parameters, which correspond to the player available controllers. For example, the game Justice provides 208 controllers for players to adjust the facial shape. The latter ones represent hairstyles and makeups.

We only focus on some important ones and encode them them with one-hot coding for easily processing. Based on the definition of facial parameters, we build our generative neural network named the imitator in order to imitate the behavior of the game engine. In details, we use eight transpose convolutional layers in the imitator with corresponding ReLU and BN layers. We are the input and output of this network as the facial parameters and the facial images of a game character. For training this neural network, we randomly generate 20,000 facial parameters obeying uniform distribution, and render them by the game engine of Justice. The following picture shows a part of our training dataset. Furthermore, we use L1 norm to define

the distance between the training images and the network outputs. And then we use stochastic gradient descent method with momentum to optimize the weights in our imitator. Based on our experiments, it's quite easy to train such network because the texture of game characters is much simpler than our skin. Here are some pairs rendered by the imitator and the game engine with the same facial parameters. Can you guess which column is the ground truth? Let me reveal the answer. The right one is ground truth. Indeed, it's quite difficult to tell the differences, and this means that we can imitate the game engine very well and replace it,

even if the training data is generated randomly. After we can generate differentiable game characters, the next problem is how to define the face similarity between game character and the human face. In our face In our feature extractor, we use two discriminative neural networks to model human perception. The face recognition network is used to judge whether the persons in two images are the same one or not. And they And the face segmentation network is used to compare the detailed facial texture in two images. The face recognition network takes in the facial images and can generate facial embeddings to to represent identity information. Apparently, photos from the same person can always share similar embeddings.

In our method, we adopt famous pre-trained neural network Light-CNN to extract global face information, which is identity information, for input photo and generated character. This network can take in 128 + 128 gray gray images and generate 256-dimensional facial embeddings with good speed and accuracy. Based on this model, we can define the identity loss function by computing the cosine distance between facial embeddings. Obviously, this loss function will achieve a small value when we generate a character that looks like the input photo. Gen- Generally speaking, game characters and real faces have different distributions,

which may affect the performance of the face recognition network, because this network is trained on the real face. So, we further adopt a face semantic segmentation network in in the extractor to solve such domain gap problem, because the local shape information like edges is domain irrelevant. Detailedly, we adopt ResNet-50 as backbone to design our segmentation network. We replace the last fully connected layer by 1 + 1 convolution layer and increase the the resolution of this network from 1 32 to 128. As a fully convolutional network, it can generate a 2D semantic segmentation map. Each pixel in this map represents a class of human face, such as nose, eyes, and mouth.

And its low-level features contain local details of the input image. We collect uh thousands of human photos with fine labels, which which has 11 classes, including eyes, nose, mouth, and so on. We also use data augmentation technique to further enhance the performance of the semantic segmentation network. We use normal pixel-wise cross-entropy loss in the training procedure. And uh it can converge well, as shown in the following figure. Because we expect that the created uh character shares similar local information with input photo. We use pixel-wise L1 norm to achieve this global To achieve this goal, when we define the content loss.

Considering that facial facial features are more important, we further use semantic probability maps to weight these parts to improve the shape similarity of the facial features. After we introduce two kinds of loss functions representing identity and shape information res- respectively, we combine these loss functions to construct an optimization problem related to the facial parameters. And then we can use gradient descent method to solve this problem and obtain the optimized facial parameters. Since each sub-module in our method is differentiable, we can easily use the chain rule to calculate derivative of the loss function LS with respect to the facial parameters X and

update X accordingly. Finally, the updated parameters can be read by the game, and the corresponding character will be created. Geometrically, the imitator GX is a low-dimensional manifold in the high-dimensional image space. So, the gradient descent procedure of solving the facial parameter can be viewed as searching for the nearest point to the input image measured by defined uh facial similarity. In summary, we create a character through three steps. Firstly, we align the user-uploaded photo based on the default game face, which will help which will be helpful for comparing the features pixel by pixel.

Secondly, we freeze all three networks in our method and solve the optimization problem that we defined previously. This will maximize the facial similarity between the input photo and the created character. The The iterative process can be seen in the GIF on the right. Finally, we can write the optimized parameters in the game, and then we get a character that looks like the input photo. Most importantly, our method retains the original interactivity of the game, since it can generate the facial parameters that are open to players. Players now can further edit the characters we provided and uh even add a darker makeup on them.

In prac- In practical, some discrete variables, such as hairstyle and eyebrow, have a great impact on characters. But they are difficult to optimize them directly due to the one-hot coding. To solve this problem, we consider one-hot coding as a probability vector and then use soft- max function to convert an extra unnormalized vector SD into the proba- probability vector X hat D before feeding the facial parameters into the imitator. Finally, we can construct a a new optimization problem, which is LS HX, and it can be optimized uh smoothly. When the iteration stops and the optimized facial parameters are obtained, we only need to use argmax function to calculate the index

corresponding to the maximum value in the vector of each discrete variable for game characters on the hairstyle and makeup configuration. This table shows the performance of our method, which has a short development period and a high execution efficiency in the running time. Here we show some results. Our method has been verified on our game Justice. It can generate vivid characters based on a single photo. Our method is not limited to different classes and the different genders. Our method can also Our method can also be applied on mobile games. Here are some results on our upcoming game.

Our method has three obvious advantages. Firstly, it does not require any labeled data, since it is under a self-supervised learning paradigm. Secondly, our proposed imitator can be easily integrated with the game engine, regardless of the render type and 3D model structure. In the end, we found that deep learning-based features can represent human perception very well on face similarity evaluation. Unfortunately, our method still has some shortcomings, such as being sensitive to pose and obstacle, solving discrete parameters not very well, and requiring human-like 3D model. Here we show some robustness experimental results. Our method is robust to common image degradation deg-

degradation, but uh not robust enough to the pose, because our imitator can only generate frontal images. Fortunately, we could also ask our players to upload a frontal image as to meet our requirements as much as possible. And this will improve the created characters. That's all. Thank you very much for your watching.