AI for GAMES+Direct answer
How are LLMs used in games?
LLMs in games can interpret observations, reason about goals, plan actions, communicate, and help control characters. They may receive text, pixels, structured state, APIs, or multimodal input. In turn, they can produce dialogue, high-level plans, tool calls, or commands that a learned policy or control module converts into timely game actions.
LLM game agents range from systems specialized for one title to generalist agents that work across games and interfaces. Their core challenges include grounding language in the current game state, handling partial information, remembering events across long episodes, learning unfamiliar rules, adapting at test time, and coordinating with human or AI partners. Explaining a good strategy is not the same as executing it under real-time constraints.
AI NPCs are a related but distinct use case. They must keep dialogue, memory, personality, and actions consistent with the shared world rather than merely produce fluent conversation. Evaluation should therefore match the role: task completion, progress, efficiency, robustness to changed visuals or controls, long-horizon memory, and cooperative behavior for agents; grounded, useful, and coherent interaction for NPCs and teammates.
This topic is surveyed in AI for Games in the Foundation Model Era.
Research scope
What this topic covers
Generalist game agents
Policies and multimodal agents that act across games, interfaces, tasks, or visual settings.
Planning, memory, and control
Hierarchical decisions, tool use, long-horizon memory, test-time adaptation, and action execution.
NPCs and cooperation
Teammates, companions, opponents, and characters that coordinate through actions and language.
Evidence
What should evaluation establish?
A convincing agent evaluation separates reasoning from execution and names what changed between training and testing. It should show whether competence survives new visuals, controls, rules, goals, partners, or interaction budgets—not only whether the model can describe the right move.
- Completion, progress, and action efficiency
- Transfer across games and interfaces
- Memory and adaptation over long episodes
- Coordination with people or other agents
Starting points