Jim Fan, a senior researcher at Nvidia, believes that artificial intelligence has the potential to radically reshape video games. He believes that rapid advances in artificial intelligence are opening a new frontier for video games: infinite open worlds, unique content, dynamic dialogue with non-player characters (NPCs), and the ability to accelerate game development. He added that artificial intelligence and games were born for each other and their happy marriage has just begun. Nvidia introduced an AI platform called “Avatar Cloud Engine for Games” earlier this year that allows players to converse with non-playable characters.
Jim Fan is an artificial intelligence researcher whose research focuses on basic multimodal models, reinforcement learning, computer vision, and large-scale systems. He is one of Nvidia’s principal researchers and responsible for the company’s AI agents initiative. He stated that his mission is to create skilled and competent agents in the physical (robotics) and virtual (games, simulation) world. He earned a doctorate at the Stanford Vision Lab and completed an internship at OpenAI (with Ilya Sutskever and Andrej Karpathy), but also at Baidu AI Labs and at the Institut québécois d’IA (MILA) founded by Yoshua Bengio.
Before we have a million robots in the physical world, we will first see a billion agents embodied in virtual worlds. Gaming is the second major area that I will devote myself to in 2024. Artificial intelligence and gaming were born for each other, and their happy marriage has just begun.
Fan believes that recent advances in artificial intelligence, including large language models (LLMs) and generative AI, will usher gaming into a new era. He led Voyager (the first AI agent that plays Minecraft efficiently and continuously develops its skills), MineDojo (an open agent that learns by watching hundreds of thousands of Minecraft videos on YouTube), Eureka (a five-fingered robotic hand which performs extremely dexterous operations) tasks such as turning a pen) and VIMA (one of the first basic multimodal models for robot manipulation).
In a post on X on Thursday, Fan said 2024 will be a turning point for the video game industry. He claims that ongoing efforts to integrate MLLs into video games will transform the industry once and for all and make non-playable characters “alive” and interact much more fluidly and cohesively with humans. Below are Fan’s predictions for 2024 and the future of video games:
Posted by Jim Fan
Before we have a million robots in the physical world, we will see a billion agents embodied in virtual worlds. Gaming is the second major area I will focus on in 2024. Artificial intelligence and games were born for each other, and their happy marriage has just begun.
On the one hand, open games constitute a “primordial soup” for the emergence of general AI. An agent’s capabilities are limited by the complexity of the world in which he lives. Minecraft is a great example of this. In 2023 we have seen an explosion of new algorithms made possible by Minecraft. To name just a few:
– MineCLIP (from our team’s MineDojo open source framework): Learn about reward features by watching 100,000 gaming videos on YouTube.
– VPT (OpenAI): Mimic behavior via pseudo-tagging actions on YouTube.
– Travel: Lifelong learning in the context of an explicit library of coding skills.
– STEVE-1: guides the actor to follow commands in the latent space of MineCLIP, inspired by DALLE-2.
– And many others: DEPS, Jarvis-1, DreamerV3…
Besides Minecraft, there are many other games that require extremely advanced perception, agility, exploration, reasoning, and planning. We’re just scratching the surface.
I believe that games (and simulation in general) will provide the next trillions of high-quality tokens to train our baseline models. The nice thing is that these tokens are actively selected by the agent itself through exploration. He can choose to experience things that minimize his internal uncertainties, just like human curiosity.
On the other hand, artificial intelligence will bring a paradigm shift in the gaming industry. This year we’re seeing renewed community interest in Stanford Smallville, where 25 AI agents inhabit a digital city. They go to work, gossip, organize social activities, make new friends and even fall in love.
It’s an exciting experience, but we haven’t felt any impact on actual games yet. This is because our LLMs are too boring and too expensive. If you take a look at the Smallville chat log, you’ll see that the conversations aren’t funny at all. No parent speaks to their children so politely. The unit economics of employing such a large number of agents also make no sense on a large scale.
That said, I think 2024 is a turning point. The digital world of Westworld is coming and will transform the industry once and for all. The games will truly be alive. Characters will interact with humans and each other, form relationships, take consistent actions throughout their lives, and respond in human ways. Each game will have infinite replayability and each player will have a unique and personalized experience.
NPCs generally have a fixed pattern of behavior, and their mannerisms and speech are stilted and unnatural. However, Fan believes that generative AI will help game developers create a greater variety of original dialogue for NPCs. Some publishers are already using generative AI in NPCs to make conversations more realistic. Chinese gaming company NetEase uses ChatGPT to generate NPC dialogue in its mobile game “Justice,” while Replica Studios, an Australian AI company, recently introduced “AI-driven intelligent NPCs” for giant Unreal game engines Engine.
This allows game developers to use AI to read NPC dialogue, rather than hiring a human voice actor. However, Julian Togelius, an associate professor at New York University, said that using AI-generated NPCs comes with a risk: Game publishers could lose control of the game’s narrative. Togelius is conducting research on artificial intelligence and gaming at New York University. He states, “NPCs may say things that interrupt gameplay, are rude, or otherwise break immersion.” He also said that creating AI-driven NPCs might prove more complicated than you think.
According to him, creating “good” AI-generated NPCs who help the player is also much more difficult than creating enemies who fight against you. “We haven’t seen a lot of progress in the AI that animates other characters in games, that tries to shape the player, or that tries to generate the world. So we’ll see a lot of progress in those directions,” he said. Additionally, some critics say that the he integration of generative artificial intelligence into video games could have a negative impact on most multiplayer games, making levels unplayable or deviating from the game’s plot.
Open-world games, such as “Grand Theft Auto,” “Skyrim” and “Elden Ring,” approach gameplay with nonlinear missions and stories. Generative AI therefore has the potential to reshape the gaming experience. Togelius imagines a “huge open game” with endless possibilities, new cities, new people and new landscapes, each with its own story and interactive elements. According to him, using data collected about the player from previous games, generative AI may be able to create unique scenarios and missions tailored for players of the right level.
However, if developers maintain control over procedurally generated content, generative AI has the potential to develop unplayable levels or unintentionally deviate from the game’s narrative. “Games have functional constraints: levels must be completeable, and NPCs must not be in the game world,” says Togelius. According to other critics, the mind-blowing AI models could also pose a real headache for video game developers. This may cause NPCs to deviate from the game’s plot and thus contribute to a worsening of the player’s experience.
What is your opinion on the topic?
What do you think about integrating generative AI into video games?
In your opinion, what are the risks associated with integrating generative AI into video games?
See also
Nvidia Introduces an AI Platform Called “Avatar Cloud Engine for Games” That Allows Players to Converse with Non-Playable Characters
Artificial intelligence now allows you to have real conversations with NPCs in video games: the case of Matrix Awakens. This development could give rise to a multitude of possible scenarios
The generative AI revolution in gaming: a path to limitless creativity, with 20% of professionals considering cost reduction
