Alibaba Tongyi Lab have launched MAI-UI—a household of basis GUI brokers. It natively integrates MCP software use, agent consumer interplay, gadget–cloud collaboration, and on-line RL, establishing state-of-the-art outcomes typically GUI grounding and cell GUI navigation, surpassing Gemini-2.5-Professional, Seed1.8, and UI-Tars-2 on AndroidWorld. The system targets three particular gaps that early GUI brokers typically ignore, native agent consumer interplay, MCP software integration, and a tool cloud collaboration structure that retains privateness delicate work on gadget whereas nonetheless utilizing massive cloud fashions when wanted.
What’s MAI-UI?
MAI-UI is a household of multimodal GUI brokers constructed on Qwen3 VL, with mannequin sizes 2B, 8B, 32B and 235B A22B. These fashions take pure language directions and rendered UI screenshots as enter, then output structured actions for a stay Android surroundings.
The motion area covers commonplace operations comparable to clicking components, swiping, coming into textual content and urgent system buttons. On prime of that, MAI-UI introduces specific actions for answering consumer questions, asking the consumer for clarification when the objective is ambiguous, and invoking exterior instruments by MCP software calls. This makes the agent able to mixing GUI steps, direct language responses and API stage operations in a single trajectory.
From a modeling perspective, MAI UI unifies three parts, a self evolving navigation knowledge pipeline that features consumer interplay and MCP instances, a web-based RL framework that scales to a whole lot of parallel Android situations and lengthy contexts, and a local gadget cloud collaboration system that routes execution primarily based on job state and privateness constraints.
GUI grounding with instruction reasoning
A core requirement for any GUI agent is grounding, mapping free type language like ‘open month-to-month billing settings’ to the right on display screen management. MAI-UI adopts a UI grounding technique impressed by the sooner UI-Ins work on multi perspective instruction descriptions.
For every UI aspect, the coaching pipeline doesn’t depend on a single caption. As an alternative, it generates a number of views of the identical aspect, for instance look, perform, spatial location and consumer intent. These a number of directions are handled as reasoning proof for the mannequin, which should choose a degree inside the right bounding field. This reduces the influence of flawed or underspecified directions, a difficulty that UI Ins quantified in present datasets.
Floor fact bins are collected from a mixture of curated GUI datasets and enormous scale exploration of virtualized working techniques in containerized environments. Accessibility bushes or OCR primarily based parsers are used to align textual metadata with pixel areas. The coaching goal combines supervised advantageous tuning with a easy reinforcement sign that rewards right level in field predictions and legitimate output format.
On public GUI grounding benchmarks, the ensuing MAI-UI fashions attain 73.5 p.c accuracy on ScreenSpot Professional with adaptive zoom in, 91.3 p.c on MMBench GUI L2, 70.9 p.c on OSWorld G and 49.2 p.c on UI Imaginative and prescient. These numbers surpass Gemini 3 Professional and Seed1.8 on ScreenSpot Professional, and considerably outperform earlier open fashions on UI Imaginative and prescient.
Self evolving navigation knowledge and MobileWorld
Navigation is tougher than grounding as a result of the agent should keep context throughout many steps, presumably throughout purposes, whereas interacting with the consumer and instruments. To construct sturdy navigation habits, Tongyi Lab makes use of a self evolving knowledge pipeline.
Seed duties come from app manuals, hand designed eventualities and filtered public knowledge. Parameters comparable to dates, limits and filter values are perturbed to increase protection, and object stage substitutions are utilized whereas staying inside the identical use case. A number of brokers, along with human annotators, execute these duties in Android environments to provide trajectories. A decide mannequin then evaluates these trajectories, retains the longest right prefixes and filters out low high quality segments. The subsequent supervised coaching spherical makes use of the union of recent human traces and prime quality mannequin rollouts, so the information distribution regularly follows the present coverage.
MAI UI is evaluated on MobileWorld, a benchmark from the identical workforce that features 201 duties throughout 20 purposes. MobileWorld explicitly mixes three classes, pure GUI duties, agent consumer interplay duties that require pure language backwards and forwards with the consumer, and MCP augmented duties that require software calls.
On MobileWorld, MAI UI reaches 41.7 p.c general success, a achieve of about 20.8 factors over the strongest finish to finish GUI baselines, and aggressive with agentic frameworks that use bigger proprietary planners comparable to Gemini 3 Professional.
On-line RL in containerized Android environments
Static knowledge isn’t sufficient for robustness in dynamic cell apps. MAI-UI subsequently makes use of a web-based RL framework the place the agent interacts instantly with containerized Android Digital Gadgets. The surroundings stack packs rooted AVD pictures and backend providers into Docker containers, exposes commonplace reset and step operations over a service layer and helps greater than 35 self hosted apps from e commerce, social, productiveness and enterprise classes.
The RL setup makes use of an asynchronous on coverage methodology, GRPO, applied on prime of verl. It combines tensor, pipeline and context parallelism, much like Megatron type coaching, in order that the mannequin can be taught from trajectories with as much as 50 steps and really lengthy token sequences. Rewards come from rule primarily based verifiers or mannequin judges that detect job completion, together with penalties for apparent looping behaviors. Solely current profitable trajectories are stored in job particular buffers to stabilize studying.
Scaling this RL surroundings issues in apply. The analysis workforce exhibits that growing the variety of parallel GUI environments from 32 to 512 yields about 5.2 share factors enchancment on navigation success, and growing the allowed surroundings steps from 15 to 50 provides about 4.3 factors.
On the AndroidWorld benchmark, which evaluates on-line navigation in a normal Android app suite, the most important MAI UI variant reaches 76.7 p.c success, surpassing UI-Tars-2, Gemini 2.5 Professional and Seed1.8.
Key Takeaways
- Unified GUI agent household for cell: MAI-UI is a Qwen3 VL primarily based household of GUI brokers from 2B to 235B A22B, designed particularly for actual world cell deployment with native agent consumer interplay, MCP software calls and gadget cloud routing, moderately than solely static benchmarks.
- Cutting-edge GUI grounding and navigation: The fashions attain 73.5 p.c on ScreenSpot Professional, 91.3 p.c on MMBench GUI L2, 70.9 p.c on OSWorld G and 49.2 p.c on UI Imaginative and prescient, and set a brand new 76.7 p.c SOTA on AndroidWorld cell navigation, surpassing UI Tars 2, Gemini 2.5 Professional and Seed1.8.
- Real looking MobileWorld efficiency with interplay and instruments: On the MobileWorld benchmark with 201 duties throughout 20 apps, MAI UI 235B A22B reaches 41.7 p.c general success, with 39.7 p.c on pure GUI duties, 51.1 p.c on agent consumer interplay duties and 37.5 p.c on MCP augmented duties, beating the perfect finish to finish GUI baseline Doubao 1.5 UI TARS at 20.9 p.c.
- Scalable on-line RL in containerized Android: MAI-UI makes use of a web-based GRPO primarily based RL framework over containerized Android environments, the place scaling from 32 to 512 parallel environments provides about plus 5.2 factors in navigation success and growing the surroundings step funds from 15 to 50 provides one other plus 4.3 factors.
Try the Paper and GitHub Repo. Additionally, be at liberty to observe us on Twitter and don’t neglect to hitch our 100k+ ML SubReddit and Subscribe to our Newsletter. Wait! are you on telegram? now you can join us on telegram as well.
Asif Razzaq is the CEO of Marktechpost Media Inc.. As a visionary entrepreneur and engineer, Asif is dedicated to harnessing the potential of Synthetic Intelligence for social good. His most up-to-date endeavor is the launch of an Synthetic Intelligence Media Platform, Marktechpost, which stands out for its in-depth protection of machine studying and deep studying information that’s each technically sound and simply comprehensible by a large viewers. The platform boasts of over 2 million month-to-month views, illustrating its reputation amongst audiences.
