Session: The Future Is Agentic: Unlocking Creative Research With AI - Robert Huynh, Reforged Labs.

[Music] The next talk is by uh Robert from Reforge Labs who's here to talk about the uh the future being agentic, unlocking creative research with AI. So, Robert, can I bring you to the stage? Thanks for the great introduction, David. Hi, friends. I'm Robert. Nice to meet everyone. Hi. I love the warm welcome. I'm very excited. I discovered a sauna yesterday and so a little bit more hung over than I would want to be. Uh do we have the slides up? Awesome. So, I'm here to talk about how the future is agentic and what exactly does that mean? For me, as someone that's been in AI for

like the past decade, I feel like it's a bit of a buzzword right now. Like what is this AI thing? What does it mean for our industry? And I really want to unpack like what are the capabilities of AI, but also the limitations of AI today. Let's go next slide. Or do I have a clicker?
I think you've probably got a green button there which will take you.
I do.
Oh, no. Listen, here it is. Here it is. Okay, great. I'm not looking in the shadows. There is that one.
Awesome. Thanks, David. Cool. Cool. Awesome. So, I'm here to talk about mobile games and creative research. One of the things that I've realized is that the process of coming up with new concepts, new ideas is super manual. People are telling me how they spend hours going out there to go and scan their competitors, find the best creatives out there, watch all them, tag them, understand what's happening. And when I think about what's possible with AI technology today, it's that deep level of understanding that can take all that manual work so you can actually

focus on what's actually productive and creative for the next stage of evolutions. So our vision at Reforge is to build a system that can essentially understand all the creatives in the market, analyze them and give you recommendations, the key concepts that are working that are not working for your competitors. A little bit about me. I started with uh AI about 12 years ago at Microsoft. went on to Google to build their self-driving car team in Google's moonshot labs. So, if you've been to SF, you've probably seen the Whimos driving around. A lot of that is powered by large scale data annotations. And what that means is that we had people manually going and tagging what exactly was a stop sign, what is the crossroad? And if you ever logged into your PayPal account, you've probably helped us do that labeling

yourself. you've clicked in and tagged what was a stop sign as part of the verification login process recapture. I didn't come from mobile games, but as a lifelong gamer, it was really exciting to move into the space. And so I built a team around me that was able to do it. And about a year and a half ago, we decided to launch the company. Two weeks after meeting the founders of King, we raised 4 million and started running at this problem. And so to build out a creative research tool, we needed to get really deep into this workflow, this process. And so we started partnering with like top mobile game studios around the world in order to go out there and actually like reimagine what that process could look like instead of just trying to force like AI into someone's workflow. It doesn't work like that. And I quickly learned that the large scale data annotation I was doing is actually called creative tagging.

And so that's how we came up with our AI creative research partner. I'll show you what that looks like in a second. So do I need to click to play the video right here?
Will the video just play?
Maybe on that side. [Music] So, what you're going to see is a couple of the key uh use cases that we've discovered. Searching for video ads. We built out a large scale labeling team to go out there and tag those creatives. So, you can do semantic search. You can analyze your competitors. Instead of like just trying to scan through, tag and understand what books are doing. We're able to give you deep level analysis by network, by geography into what concepts are working. So looking at top performers, mid- performers, low performers, you're going to see a really

big difference. Generate, we don't do any sort of creative production. We just stop in making creative brief. And just like how Chat GPT helps you figure out what kind of email draft works best, you're going to go in there and customize it a little bit. This is what that process could look like. This is an early build and so we've been building this product in stealth for like the past year and we just launched like two weeks ago. So I'm really excited for that. I'll talk to you like what these like actual points mean cuz I know that video was super fast. Um but that first feature that we've seen these are just a couple of the key use cases that a AI that's watched every single video creative out there and has tagged them for the emotional understanding chapterized the video so that you understand the hooks. you can really do an in-depth analysis of like, hey, actually my competitors are looking at these specific concepts and it's not just an ASMR concept. There's a little twist to it and they're testing it in the Philippines. They're

testing this concept in Brazil and actually two of those concepts, they didn't work. They didn't start to scale them in tier 2 markets, but one of them did and you can get ahead of those learnings. Two, answering things in depth. You know, one of um the popular use cases I saw was like instead of looking for a creative by name or by the publisher or the studio that made that game, sometimes you just want to see examples of what gold coins look like in other games. And by typing in, hey, I want to see examples of gold coins. Because our AI has watched through all those videos and tagged them, you can now surface up all those videos very quickly. You can surface up things that are scary or hey show me funny ads or show me things with this specific thing in it and how they might have been used for this particular market for this particular network and we correlate that with some sort of signal in the case of competitors third party signals probably impression or estimates of impressions. Um but with

internal data we can start to correlate that with um your own metrics of success. So day seven um you can look at your monetization metrics, your retention metrics and start to see like okay what is a good ad and that analysis um at depth is really interesting because for example I've been getting a ton of these like royal match ads like everywhere. Some of them actually all of them have like a king in peril. This king he's about to get eaten by like this giant snake. But what differentiates that from like the very top 10% and the bottom 10%. I was curious. I asked our AI and it was able to tell me that, you know, the top performers, yes, they're all king in peril, but the very top ones had a reaction on the like king's face that you knew like, oh he's in trouble. Like, and I can do something about it. There's a possibility for me to action on that. And the lower performance is

not as immediate, right? there isn't that tension that's built up. And so when you have an AI that's been able to deeply understand what is happening, not just from like a surface like this is the color, this is like the shape, but actually the emotions and the motivation behind it. I think that's where you get to get some really interesting insights. And then finally, we just wrap it up all together with creative briefs. We stop at creative briefs because I think that creative producers, UA teams, they know what's best for their game. But as a tool, we want to be able to give you a great starting point. And so I'll give you an example. This feature isn't fully built out yet. But not everyone has the budget of like Dream of Celebrities for their ads and not everyone has the budget to do all these different 3D animations. So if you can have a creative brief that understands your context, your constraints, how many animators you have, and what are the different skill

sets of those animators, they're all probably using Nano Banana today. They're already using Flux Cling in order to generate their AI stuff, but they're also using their traditional After Effects and all the tools that they know and love. All that context is super powerful to have a brief in your format for the agencies that you already work with to triple that process from idea to actual production. And so that's where I see the limitations of where AI is today. AI can't solve all of our problems and it's not going to be able to do that for a really long time. But it is a really great tool to ideulate, analyze, synthesize information, and give you that starting point. So I'll just recap. We're building the future of creative research. And what that looks like is agentic. You have these underneath the surface, these agents that will go out there and do research

for like 10 minutes, an hour at a time, and come back and say like, "Hey, Robert, I'm done with my research. This is what I found are the top concepts that will get people to stop doom scrolling on Tik Tok. And these are a couple things that you might want to try this week. Helping creative teams to figure out what's working and what to make next. That's the goal that we have. And we want you to not just understand the trends, but set their trends. Thank you and would love to take your questions.
Thank you very much. Great stuff. Great. We can get a mic to you. Does anyone have any questions in the audience? Oh, we got one on the front row right here. Thank you.
Thank you for the talk. Al from Alto University. Uh do you think the tool the AI you made can also help designers for design based research or quantitative research not just qual qualitative research not just

quantity based research or help them design not by looking at the data
but also trying to find out why some I don't know human behaviors happens.
Yeah it's a really interesting question. It's actually all about what sort of data is available. So what's interesting about like ads and video ads in particular is that it's unlike like full feature films where what what is a good movie, right? Like I really love the latest Marvel movie. My friends don't. And when I see that it's like it's very subjective. An ad I can optimize for some sort of objective function like impressions or clickthroughs. So for a design it's the same problem, right? Like what is a good design? How do I know it was a good design? And unless you have that quantitative measure of success, it's hard to like chase after that. If you have some sort of proxy signal, and I know designers often do, um, we can go and optimize towards that things. But for us right now, we're very focused on this narrow use case of, hey,

let's take in a ton of video ad creatives by network, by country in order to optimize against, hey, how do I get high impressions for my particular game, and how do I understand what kind of compets are working today and what could work next?
Thank you.
Yeah, appreciate the question.
Great. Um, do we have any other questions? We got a little time. So, I've got a I've got a question for you, actually. And by the way, I I didn't realize you were involved in Whimo. I I went in one of those things at San Francisco. That's amazing, right? That's the future. Thank you. Yeah. My mom was terrified when I took her one in 2016.
And quickly, she was just like, "Actually, this is so boring. The car is moving so slow." You know, I I I know I thought it was I thought it was incredible. And I don't if anyone has a chance to go and go in a self-driving car, particularly that that process in in San Francisco, I I was while I was driving around it, I was going in 10 years time, we won't believe that we let humans drive their own cars. This is what this is what I thought.
I'll make a plug. It's like 15 times

safer than human.
Yeah.
But the, you know, threshold to go out there and go worldwide, like it's very very high.
Yeah. Sure. Anyway, sorry that was just I didn't realize you were involved. That was just on the side. I've got I've got a question for you. I guess this is about um the process. So, um, the the the insights that you can get from from what you're talking about is is great. Analyzing, um, kind of competitors and other, uh, successful creatives online, if someone is working on a game right now, at what process, what at what time in the process is this perfect to get involved in? Is it something that comes down the line when you're ready to spend on marketing or should you be thinking about it right from the start so it can help you form your basic kind of the basic game itself kind of thing? What what's your insight into that? Yeah, it's really interesting like you know nextgen UA like I talked a lot about our AI tool and what we've been building
where in the process you think about UA you think about it from the very beginning you think about it throughout live ops
and what I've seen in particular when I think about nextg UA is like I spend about half my year in Asia where we have a large scale data labeling team and

particular in Vietnam where a lot of the studios there have been really able to scale super fast because they've been thinking about creatives from the get- go. There's a lot of things you can do to optimize UA, but the creative side is actually where you get your advantage. And in a world where like you have Nano Banano or Sora 2 that helps you generate stuff, the thing that's going to differentiate you is like the ideas and the concepts. And so what a lot of these studios in Asia have started to do is that they've been testing games quote unquote because they built two of these games a week. They have half levels and if the UA seems to work, then they start to build out the rest of the game. And so they go from, hey, let's copy this puzzle screw the screw puzzle concept and I'm going to have a variation that is like horror. I'm going to have a cozy type of game. I'm going to have a 3D game and if it works, I'm going to

triple down on that. And that's worked out really well for a lot of the studios in Vietnam.
That's interesting.
So you think about it from the very beginning. uh you think about it when you're trying to scale. When you're trying to scale a lot of your creatives or often iterations on your winners, how do you replace those winners? That takes a lot of effort. But the way I think about it is like kind of throwing a dart onto a board. Once you throw that dart and it lands, your iterations are only going to move it up and down. You're going to find a local uh local min or local max, but you can't reach that global maximum. To do that, you really need to get a new dart and throw it at the wall. And finding those concepts are winners. That's the really tricky part. So when you start to understand what's happening in the market, you yourself will come up with new ideas.
Perfect. Thanks for the question. I see there's a question at the back. I had another one of my own, but let's get this uh
Hello Robert.
Hey yo.
Hey. Uh so massive data labeling exercise. uh you've done that before, but uh what did you learn this time

around? Um what have been the most powerful signals that you've captured through that exercise?
Yeah, I think like when you talk about powerful signals in data labeling, what I've learned now in mobile games is creative tagging is that you have two ways to think about tagging. You either have like a fixed taxonomy that isn't flexible enough to capture new trends and detect things that you weren't specifically looking for or do you have something that's too rigid and anything could be clustered up into a new quotequote trend. So what we've really spent a lot of time thinking about is in a world where one detecting new trends is harder than ever because the amount of content there is impossible for someone to watch AI has to play a bigger role. Second is labeling is a combination of not just like doing human tagging like that's a lot of manual work. AI can do the starting point for

that nowadays as well. And so being able to understand like how to build a very robust labeling tagging process is kind of a secret sauce. And so you think about that in terms of like levels of uh risk with these labels. Who is labeling your creatives and what are they looking for? Because your insights are only as good as your data. And if you're labeling data, you also need to have really great labelers, processes, and tools around that in order to express the things that you're looking for. Thanks for the question, you all.
I've got another question for you. Actually, we've got a couple of minutes left. So, if you don't mind, um, what's on next on your road map for you? You're, you've obviously here today, you told us it gave us some great insight in there. If you were back in a year's time, what is what is Reforge Labs going to have done in that year?
In one year's time. Oo, that's a great question. So, we launched two weeks ago. We had been building in stealth for about a year with three of the largest

game studios and it's been super cool to work with brands that I love. I would love to do more of that. And so, being able to not just work with large mobile game studios, but go work down towards like smaller midsize studios and we're already beginning to do that. One, two, is constantly learning about mobile games. I come from an AI background and be able to work with our partners that's been super rewarding for me and to understand like okay these are super painful things like I just spent um a month in Brazil working with the largest studio in Latam and I just sat by with two of the creative producers five PMMs and we were looking at like okay you do that like you go into sensor tower you watch the videos you try to understand is this a trend you look at other markets I was like we can actually build a a workflow that automates a lot of this. So if you're using chatbt to try and come up with analysis, it's not reliable. It's going to make stuff

up. And they didn't understand that until we try to check that against the insights that like you get when you look at a sensor tower and that manual process or hey we have a monthly creative like roundup. What are the creatives that are working? And so being able to expand into all those area that's in the one-year roadmap as well as really focusing on like how are my campaigns doing? Do I have a really tight loop between my UA team and my creative team? How does my creative production evolve, especially with Gen AI tools? And so all those different things are really key for, you know, our industry and what we've been building.
Perfect. Thank you. Well, we're we're kind of out of time. So a big round of applause, please, for uh for Robert and that great insight.
Hey, everyone. Appreciate the time. [Music]