Skip to main content

Google shakes up Android Bench ranking: Claude5 takes the crown, Gemini lags behind

Google just gave its Android Bench code ranking a major makeover, and the results are turning heads. On July 9, the company announced a complete overhaul of the developer benchmark, introducing a standardized Harbor sandbox framework. This new setup moves all testing into a secure, isolated environment, making it easier for developers worldwide to run independent evaluations, customize their development environments, and share data. To top it off, Google has open-sourced the benchmark on GitHub, inviting the community to submit custom Android development tasks and model evaluations.

But here's the kicker: in the freshly measured rankings, Google's own models didn't exactly shine. Image Anthropic's Claude5, the flagship model, claimed the top spot with an impressive 84.5% accuracy. Close behind was OpenAI's GPT-5.5 at 80.2%. Meanwhile, Google's Gemini 3.1 Pro could only manage fifth place. Ouch.

Now, Gemini does have a leg up on cost—each iteration runs about $87, while top models exceed $130. But the lightweight Gemini 3.5 Flash exposed some serious efficiency issues. When parsing the hundred-question evaluation dataset, it took a whopping 28 hours per run and cost $165. That's not exactly a selling point for developers looking for speed and affordability.

The industry is increasingly moving toward autonomous intelligent development workflows, and Google's lag in local mobile development benchmarks poses a real challenge to its AI strategy. But Android Bench, with its objective and transparent evaluation mechanism and the openness of the Harbor framework, is quickly becoming an authoritative AI code evaluation platform—one that's free from marketing hype.

Key Points

  • New Framework: Google's Harbor sandbox provides a secure, isolated testing environment for fairer benchmarks.
  • Top Performers: Claude5 leads with 84.5% accuracy, followed by GPT-5.5 at 80.2%.
  • Gemini's Struggles: Gemini 3.1 Pro ranks fifth; Gemini 3.5 Flash takes 28 hours and $165 per run.
  • Industry Shift: Core engineering projects are moving toward autonomous AI workflows, making these benchmarks critical.
  • Open Source: Android Bench is now on GitHub, inviting community contributions.