Skip to main content

Alibaba's Qwen-UI-Agent: Bridging the Gap Between Simulation and Reality

Alibaba has officially released Qwen-UI-Agent, a GUI agent foundation model that's all about the real world. This isn't just another simulation toy—it's built to handle mobile, desktop, web, and deep search environments, aiming to break free from the limitations of traditional testing. The goal? To let the model seamlessly operate on complex, real devices.

Outperforming the Big Names

In head-to-head tests, Qwen-UI-Agent has shown some serious muscle. On the MobileWorld benchmark, it scored an impressive 82.1%, beating several major international flagship models. But here's the kicker: on its own real-device benchmark, MobileWorld-Real, which involves over 100 actual phones, it hit a success rate of 92.2%. And on Android Daily, it was nearly flawless.

Desktop performance is equally strong, with a 79.5% score on OSWorld-Verified, and it even cut down the number of execution steps needed for various tasks. On the web front, it ranked first among all comparison models in WebArena. Its general and agentic capabilities have fully surpassed its training base model, handling long-tail demands with ease.

From Simulation to Reality

One of the biggest hurdles in AI development is the gap between simulated environments and the messy, unpredictable real world. Qwen-UI-Agent tackles this head-on by building a real mobile environment with over 100 real phones and 150+ applications for task creation, trajectory collection, and training. They also introduced MobileWorld-Real, a benchmark with over 400 tasks, allowing developers to fine-tune based on real-device success rates.

Image

Smart, Safe, and Efficient

What sets this model apart isn't just its performance—it's also how it handles instructions and safety. Beyond conventional GUI operations, it can directly execute command-line commands. It outputs batch actions in a single decision, which shortens the execution trajectory and boosts efficiency. But it doesn't stop there.

Safety is a big deal. The model has a comprehensive security judgment mechanism throughout the process. If it encounters illegal or high-risk requests, it refuses and terminates the task. For sensitive scenarios like payments, data deletion, or privacy authorization, it pauses at key steps and waits for user confirmation before proceeding. That's a thoughtful touch.

Continuous Learning

Qwen-UI-Agent also supports online reinforcement learning on trajectories longer than 100 steps, combined with adaptive curriculum learning. This means it's constantly tackling challenging long-term tasks, getting better over time.

Image

Key Points

  • Real-World Focus: Qwen-UI-Agent is designed for real devices, not just simulations, with a dedicated real mobile environment.
  • Top Performance: It leads in benchmarks like MobileWorld (82.1%), MobileWorld-Real (92.2%), and OSWorld-Verified (79.5%).
  • Safety First: It includes robust security checks, refusing risky tasks and pausing for user confirmation in sensitive operations.
  • Efficient Operations: Supports batch actions and command-line execution, reducing steps and improving efficiency.
  • Continuous Improvement: Uses online reinforcement learning and adaptive curriculum learning for long-term task handling.

For more details, check out the project homepage: https://tongyi-mai.github.io/Qwen-UI-Agent/