1. Could you tell us what led you to start this research?
At first, I never expected the project to turn into what it is today. In fact, I began with a rather vague idea inspired by mystery investigation stories like Sherlock Holmes or the FBI. I imagined an AI that could infer the culprit based on traces or clues left at a crime scene. At the time, I was not affiliated with KRAFTON but was a graduate student in a university research lab. I tried designing crime-investigation scenarios myself and even attempted to build 2D maps and environments. The problem was that, as a student, it was nearly impossible to design detailed scenarios while also creating maps and simulation environments. For context, this was in 2024, when the idea of “vibe coding” was still unfamiliar, so I was trying to do everything manually.
Eventually, I thought it would be better to use environments that already existed, and from that point on, I began adopting various video games as simulation environments. While I was conducting the research in that direction, I unexpectedly joined KRAFTON, which allowed me to continue the project together with KRAFTON and eventually bring it to where it is now.
2. Among the many types of games, why did you choose flash games?
“Why flash games?” is a very important question in this research. There were already many video game benchmarks, but what they were missing was that they did not require AI to solve an in-game story arc from beginning to end.
Then the natural question becomes, “Why is it important to solve a game from beginning to end?” From a research perspective, I thought it was important to examine whether AI could perform long-horizon tasks that require very long-term and meticulous planning, rather than simply carrying out short and simple plans. From a game-design perspective, I also thought that completing the given story from beginning to end would align with the game designer’s original intent.
Another question might be, “Was there a reason existing game benchmarks did not cover entire story arcs?” Some benchmarks used games that were too simple—artificially designed research games where a mission could be completed in just 10 to 30 clicks—so it was ambiguous to say they involved solving a full story arc. Other benchmarks used highly complex AAA games such as Red Dead Redemption 2, where AI was only asked to play a tiny portion of the story, making it impossible to solve the entire arc. If the full story had been required, it would likely have taken tens or even hundreds of hours. On top of that, AAA games also involve the additional issue of purchase costs.
Flash games, on the other hand, solved all of these problems. Among the roughly 100,000 flash games available on the internet, there were many with solid story arcs. The selected adventure games also covered a wide range of subgenres, including mystery investigations, escape rooms, visual novels, and life or management simulations. When we had real people play these flash games, we found that they could clear them within about 20 to 30 minutes using around 1,000 clicks. From a human perspective, this was a very interesting sweet spot: the games were not too demanding to complete, yet they still required a non-trivial number of clicks and actions. As an additional advantage, flash games are free and lightweight enough to run on the web without hardware constraints. For these reasons, we selected flash games for the benchmark.
3. What was the most difficult moment during the research process?
There must have been a huge variety of games, and setting criteria for each one probably required you to play many of them yourself. Please feel free to share the difficulties you faced while setting up the research environment. These are exactly the kinds of behind-the-scenes stories that only the author can tell.
In fact, FlashAdventure was the research project with the most practical constraints among all the studies I had worked on. The most memorable constraint was that we had to physically hand over control of the laptop to the AI so it could play the game. In other words, because we assumed a GUI-based agent that observes the actual computer screen and solves the game through actions such as clicking, we could not simply run the experiments on a server. So we physically arranged around 20 laptops in rows in the lab and had the AI play the games directly on them. This turned out to be more difficult than expected.
We did not have enough laptops, so I borrowed some from acquaintances and rented others from the university. The rental cost was not insignificant, and transporting 30 to 40 kilograms of equipment on foot was also difficult. Since many laptops were running at the same time, heat became an issue as well; even though it was winter, the room became noticeably warm.
Beyond that, recruiting participants to measure human gameplay performance, reviewing and analyzing their playthroughs, and paying participation fees were also not easy. In addition, because we covered more than twice as many games as previous game benchmarks—a total of 34 games—it took a great deal of effort to define evaluation metrics. Apart from overall success or failure, we also had to manually set milestones or checkpoints to evaluate intermediate progress and achievement.
For most games, there were walkthroughs available on YouTube or elsewhere online, so we set milestones by watching those videos. For some games, however, I remember having to play them myself to define the milestones.
Surprisingly, during the experiments, spending money became a bigger bottleneck than using GPUs. The main models I used included paid models such as OpenAI Computer Use Agent and Claude Computer-Use. Moreover, to evaluate the performance of eight models across 34 games, we had to make up to 1,000 API calls per game for each model, which meant the experiments were impossible without sufficient funding. Fortunately, in the early stages, the lab supported the cost, and in the later stages, KRAFTON provided substantial financial support, allowing us to complete the experiments successfully.
4. Were there any moments when the AI made you think, “A human would not have done it that way…”?
In the paper, you mentioned weak planning ability, visual perception limitations, and a lack of lateral thinking as reasons why the AI failed in games. If you could share specific examples, readers would likely find them very interesting.
When I actually watched the AI play games on the monitor, it was often so frustrating that it was hard to stay in my seat. Haha. On a small scale, the AI would repeatedly fail to move the mouse to the desired location or click properly. On a larger scale, even when there seemed to be multiple possible plans, it would endlessly move back and forth between the same places during actual gameplay.
In some games, there are situations where the player must creatively solve a problem in the later stages based on clues obtained early in the game. Compared with humans, the AI sometimes had the clues but failed to remember or combine them. Also, although it was not included in the paper, because we gave the AI control of the laptop, there were also occasional situations where it left the game screen and started wandering around the computer by itself—almost like a kind of strike.
What was interesting was that the AI clearly performed well or poorly depending on the type of game. In mystery investigation games or escape rooms, clues from the early part of the story often become decisive later on, so if the AI failed to remember clues or think creatively, it could hardly solve anything. In contrast, in visual novels or simulation games, clues were not as important; the AI simply had to make appropriate decisions that matched the story progression, so it tended to perform better.
5. What are the advantages or unique aspects of the Seoul National University–KRAFTON collaboration environment?
Some readers may find it unfamiliar for a company to publish academic papers. I am also curious about your impressions from experiencing both institutions. For example, how did the company’s infrastructure and the university’s knowledge create synergy?
This research happened to be conducted as a collaboration between Seoul National University and KRAFTON, and personally, I believe the company’s infrastructure and the university’s knowledge created meaningful synergy. At the university, we could conceptually analyze what AI agents still lack, or what capabilities they will need in the future, in order to achieve long-term goals. At the company, KRAFTON contributed significantly to concrete implementation and the integration of AI with games based on its game-related expertise. And as I mentioned earlier, without the company’s financial resources, we would not have been able to obtain all of the experimental results in this research. In that sense, I would like to emphasize once again how important the company’s support was.
6. What was the most memorable moment at EMNLP?
Please share any insights you gained from fellow researchers you met on site, or any episodes that may have influenced your research philosophy.
I attend NLP conferences every year, but at this EMNLP, I was especially struck by how many Korean authors there were. I met a wide range of people, from undergraduate students to professors. Among them, I had a fairly long conversation with a Korean author of a benchmark paper based on escape-room games, which had a major influence on the FlashAdventure project. We talked a lot about how AI, especially LLMs, can achieve a certain level of performance in text-based outputs or high-level planning as agents—for example, in a text-based plan such as “move the apple”—but its abilities remain seriously limited in situations involving multimodal information such as vision, or in low-level actions such as continuously generating agent-specific commands for setting and moving to particular coordinates. We also discussed whether approaches beyond LLMs might be necessary. Looking back now, I think this connects with the way physical AI is developing beyond LLM/VLM foundations and moving toward video world models and similar approaches.
I also had many opportunities to meet researchers from other countries. For example, in an EMNLP keynote talk, Professor Heng Ji repeatedly emphasized that AI4Science would become a very important challenge in the NLP field. Professor Jana Diesner discussed the intersection of NLP and Computational Social Science, and Professor Hannaneh Hajishirzi introduced OLMo 3, developed at AI2. My experience giving an invited talk at an EMNLP workshop was also extremely interesting, and I will talk about that next.
7. What did you most want to convey in your invited talk?
I heard that invited talks are relatively flexible in terms of topic and atmosphere. I am curious about what you presented in that setting and what your intention was in preparing it.
While working on the FlashAdventure research as well as KRAFTON-led research such as Orak, I increasingly felt that game benchmarks could become an important tool for developing LLM agents. Reinforcement learning is currently a mainstream direction for LLMs, and environments used in reinforcement learning need to define states, actions, and rewards. Commercially available games naturally satisfy these conditions, so I felt it was important to emphasize their value. Therefore, in the invited talk, I gave a 45-minute presentation titled “LLM Agents for In-the-Wild Video Game-Playing.” The presentation covered Orak, FlashAdventure, and a brief introduction to various AI projects currently being conducted at KRAFTON.
What I felt during the presentation and from the many questions afterward was that more people than I expected agreed with the direction of combining AI with games. Even for those who did not focus specifically on games, there was broad agreement that simulation environments like games are essential for reinforcement learning. I also introduced KRAFTON services such as PUB Ally and Smart Joy during the talk. Afterward, during the QA session, it was very memorable to have in-depth conversations with researchers from organizations such as Microsoft, Sony, and Midjourney, who already knew KRAFTON and spoke highly of the company’s AI integration and AI expertise.
8. How did you feel when the paper received the Spotlight Paper recognition?
I am curious whether you expected the recognition, haha, or whether it came as a surprise.
Of course, I was happy. Haha. In fact, I was informed about the recognition before attending EMNLP. Since Orak also received an Outstanding Paper Award, rather than being happy only about the Spotlight Paper itself, I felt that the overall outcome was very meaningful. Additionally, FlashAdventure was accepted to EMNLP 2025 Main, and I was also happy that it gave me the opportunity to deliver an invited talk at the workshop.
Beyond the conference, newspaper articles (https://news.nate.com/view/20251110n38418) were published, and the news was shared several times through department announcements (https://gsai.snu.ac.kr/category/board-89-GN-XLpzHDEo-20220216190236/ , https://snuti.snu.ac.kr/category/board-215-gn-y4hutp8u-20240523112917/?var_page=2&search_field=ALL&search_keyword=). I think all of that was possible thanks to the recognition.
9. Did this research change anything for you personally, whether in your thinking or your perspective on research?
My research direction changed quite significantly. Previously, I had focused on natural language processing and dialogue systems, but afterward, I became much more interested in agents, reinforcement learning, and embodied AI. Since my research background is not in robotics or reinforcement learning, there is still a lot I do not know. However, considering how quickly physical AI is developing and how the research landscape is changing, I felt that it was a timely and important research area.
I also found it fascinating that while the latest AI systems are rapidly improving at creating games or building worlds through world models, their ability to play games remains relatively weak. That made me think it is important to narrow the gap between AI’s generative ability and its ability to perform tasks. At the same time, I believe that in the near future, it will become important for agents not only to act in the world but also to talk or communicate with other people and agents. For that reason, I am seriously considering the social intelligence of embodied agents as a future research direction.
10. Personally, this research made me feel that AI still has a long way to go. What do you think?
Yes, I agree. Although there are areas where AI clearly performs well, I also think there are areas where it clearly struggles. Even in game playing, FlashAdventure clearly showed that AI performs much better in text-based or CLI-based decision-making, while it finds long-horizon decision-making in multimodal information environments such as GUIs extremely difficult. In fact, even outside multimodal settings, several papers have already shown that AI and LLMs struggle with dialogue or task execution in multi-turn environments.
Of course, reinforcement learning and related methods can improve performance in specific environments. However, I expect the biggest challenge will be making AI perform well in the kind of general-purpose environments humans have in mind. To address this, rapid progress is being made across many research areas, including agents, physical AI, and world models. Personally, although LLMs have weaknesses such as hallucination, I think they have evolved into powerful coding agents such as Claude Code and Codex through sophisticated reinforcement learning and agent engineering. In a similar way, I hope that embodied agents may one day, through effective learning and related engineering, be able to operate in the real 3D world as well as humans do.
11. Final message
Personally, I am truly grateful that I was able to work on FlashAdventure at KRAFTON and achieve such rewarding results. Since there are few places as genuinely committed to integrating AI with games as KRAFTON, I was able to carry out the research with great satisfaction and even fulfill one of my research dreams. I am still continuing to pursue and expand my research aspirations at KRAFTON, and I hope to have an opportunity to introduce more good results in the near future.
