Researchers at the University of California San Diego’s Hao AI Lab deployed artificial intelligence models into live Super Mario Bros. gameplay on Friday to test real-time decision-making, revealing that Anthropic’s Claude 3.7 outperformed rival systems while advanced reasoning models failed due to processing delays.
Anthropic’s Claude 3.7 secured the top spot in the evaluation, followed closely by Claude 3.5. In contrast, Google’s Gemini 1.5 Pro and OpenAI’s GPT-4o struggled to navigate the classic platformer effectively.
The experiment did not use original 1985 Nintendo hardware. Instead, the game operated through an emulator connected to GamingAgent, a custom framework designed to give AI models direct control over Mario’s actions.
How GamingAgent Forces AI to Plan and Adapt
Developed internally by Hao AI Lab, GamingAgent provided each AI model with baseline operational rules—such as instructing the character to move or jump left when encountering obstacles or enemies—alongside live visual screenshots of the game. The AI models processed these inputs and generated executable Python code to execute controller commands.
Despite relying on simple preliminary instructions, the lab reported that the setup forced each system to learn complex maneuver execution and formulate real-time strategic gameplay on the fly.
The Latency Problem Facing AI Reasoning Models
The benchmark delivered surprising results regarding reasoning-focused models like OpenAI’s o1. Although designed to work through problems step-by-step and typically dominating standard industry evaluations, these models performed worse than standard non-reasoning architectures.
The downfall of reasoning models in real-time gaming environments stems directly from speed. Models like o1 require several seconds to evaluate scenarios before committing to an action. In Super Mario Bros., where precision timing dictates survival, a single second of processing delay results in missed jumps and immediate failure.
Debating Video Games as Valid AI Benchmarks
While computer scientists have utilized games to measure machine intelligence for decades, industry experts frequently question whether gaming success translates into real-world capabilities. Unlike physical environments, digital games operate on abstract, simplified logic and offer essentially unlimited training data.
The wave of public gaming demonstrations highlights what former OpenAI research scientist Andrej Karpathy described as an “evaluation crisis” within the artificial intelligence sector.
In a recent statement on X, Karpathy noted that determining true model capability has become increasingly difficult, highlighting a lack of clarity regarding which evaluation metrics actually matter for frontier AI development.
Leave a Reply