Skip to main content

Alibaba's Qwen-UI-Agent: Bridging the Gap Between AI and Real-World Tasks

Alibaba has officially open-sourced Qwen-UI-Agent, a GUI agent foundation model designed to operate in the real world. This model covers mobile, desktop, web, and deep search environments, aiming to break free from the limitations of traditional simulation testing and enable seamless operation on complex real devices.

Image

Performance That Exceeds Industry Standards

Qwen-UI-Agent has demonstrated impressive capabilities across various authoritative tests and real-world scenarios:

  • Mobile Performance: It achieved a remarkable 82.1% score on the MobileWorld test, surpassing several major international flagship models. In its self-built benchmark, MobileWorld-Real, which involves over 100 real devices, the success rate reached as high as 92.2%. It also performed nearly perfectly on the Android Daily benchmark.
  • Desktop Performance: On OSWorld-Verified, it scored 79.5%, and it significantly reduced the number of execution steps compared to baseline models in multiple tasks.
  • Web and General Capabilities: It ranked first among all comparison models in the WebArena web test. Its general and agentic capabilities fully surpassed the training base model, handling long-tail demands with ease.

From Simulation to Real Devices

To bridge the final gap between simulation and reality, Qwen-UI-Agent has built a real mobile environment that includes more than 100 real phones and over 150 applications for task building, trajectory collection, and model training. The team also introduced the MobileWorld-Real benchmark, which contains over 400 tasks, allowing developers to make precise selections based on real-device success rates.

Image

Efficient Instructions and Security Control

In terms of functionality, this model not only supports conventional GUI interface operations but can also directly execute command-line operations. It outputs batch actions in a single decision, significantly shortening the execution trajectory and improving efficiency. At the same time, it has a comprehensive security judgment mechanism throughout the process: when facing illegal or high-risk requests, it will directly refuse and terminate the task; when involving sensitive scenarios such as payments, data deletion, and privacy authorization, it will stop at key steps and wait for user confirmation before continuing.

Additionally, this model supports online reinforcement learning training on trajectories longer than 100 steps, combined with adaptive curriculum learning, to continuously tackle challenging long-term tasks.

Key Points

  • Real-World Focus: Qwen-UI-Agent is built for real devices, not just simulations, with a dedicated environment of over 100 phones and 150+ apps.
  • Top-Tier Performance: It leads in mobile, desktop, and web benchmarks, with a 92.2% success rate on real-device tests.
  • Security First: The model includes robust safeguards, refusing risky requests and pausing for user confirmation in sensitive operations.
  • Efficiency Boost: It can execute command-line operations and batch actions, reducing execution steps and improving speed.

Project Homepage: https://tongyi-mai.github.io/Qwen-UI-Agent/