Alibaba unveils qwen-ui-agent, a real-world multimodal foundation model for gui interaction

Alibaba has officially launched Qwen-UI-Agent, a foundation model designed to operate within real-world graphical user interfaces. The model is capable of functioning across mobile, desktop, web, and deep search environments, marking a significant advancement over traditional simulation-based testing methods. It enables seamless interaction with complex real devices, bridging the gap between simulated and actual environments. The Qwen-UI-Agent has been tested in various authoritative benchmarks and real-world scenarios, showcasing its robust capabilities. It includes a real mobile environment featuring over 100 actual phones and more than 150 applications, supporting task building, trajectory collection, and model training. Additionally, the MobileWorld-Real benchmark, which contains over 400 tasks, has been introduced to allow precise evaluation based on real-device success rates. Functionally, the model supports both conventional GUI operations and direct command-line execution. It can generate batch actions in a single decision, reducing execution steps and enhancing efficiency. A comprehensive security mechanism is integrated throughout the process, ensuring that the model can identify and reject illegal or high-risk requests. In sensitive scenarios such as payments, data deletion, and privacy authorization, the model halts at critical steps and awaits user confirmation before proceeding. The Qwen-UI-Agent also supports online reinforcement learning training on trajectories exceeding 100 steps, combined with adaptive curriculum learning to tackle long-term and complex tasks. The project is available at https://tongyi-mai.github.io/Qwen-UI-Agent/.