ExpActivator

Relevance Does Not Imply Applicability: Experience Activation for Personal GUI Agents

Fuyao Zhang1Xuan Wang1Zherui Li1Jiaming Zhang1Longtao Huang2Wei Yang Bryan Lim1

1 Nanyang Technological University2 Alibaba Group

ExpActivator activates the experience that applies to the current screen, time, and scenario. A training-free framework for personalized GUI execution and proactive suggestions.

Paper Code
+28%

Within-trajectory step success

Relative to no history, averaged across four backbones
~1/5

History tokens

Compared with task-level retrieval
2.3×

Proactive suggestion MCC

Approximately, vs. the strongest proactive baseline
ExpActivator framework: task-relevant histories are matched to current visual states for action references, while routine neighborhoods are scored by time and scenario for proactive suggestions.
Figure 01 The ExpActivator framework. All encoders and policy models remain frozen. Historical actions guide the policy; they are not prescribed operations.

Why relevance does not imply applicability

“Order my usual coffee” leaves a lot unsaid. Personal history can identify the app and the drink. But a relevant past journey doesn’t tell the agent which action fits this screen—or whether coffee is even due now.

Task-level retrieval strongly improves the opening decision, yet offers little sustained benefit over the remaining 90% of steps. ExpActivator turns personalization into a decision-time selection problem: use the current situation to activate the experience that applies.

Where does history help?Qwen3-VL-8B · Step success rate (%)
Opening stepFollowing steps
No history
28.26
18.43
Retrieve
62.06
18.35
ExpActivator
70.06
26.67

Retrieval helps most at the opening. Activation continues to help across the 7,140 actions that follow. Source: Table 2.

Read the full abstract

Personal Graphical User Interface (GUI) agents rely on interaction history to infer what a user wants from ambiguous instructions and to anticipate recurring routines. Existing approaches retrieve task-relevant history and append it to the policy’s context, implicitly assuming that experience relevant to a task remains useful for each decision within it. We find that this help is largely spent at the first decision: retrieved history strongly improves the opening step of an episode, yet provides little sustained benefit over the remaining 90% of steps, and offers weak guidance on whether a proactive suggestion is warranted. A relevant record may tell the agent where to begin, but not which past action applies to the current screen or whether a routine is due now. The underlying issue is that relevance does not imply applicability: relevance is determined at the task level, whereas applicability depends on the situation at decision time.

We therefore recast personalization as experience activation and introduce ExpActivator, a training-free framework that activates only the experience applicable to the current situation. During execution, ExpActivator matches each new screen to historical states in the frozen GUI backbone’s latent space and supplies the corresponding action as a reference. Before execution, it activates a recurring intent only when the current time and scenario provide sufficient support, and otherwise abstains. Across four GUI backbones, ExpActivator improves within-trajectory step success by 28% on average, achieves the best personalized execution on every backbone while using about one-fifth as many history tokens, and reaches approximately 2.3× the Matthews correlation coefficient of the strongest proactive baseline. Experience pays where it is activated, not where it is appended.

How ExpActivator works

A frozen backbone, with experience selected at the moment it matters.

During executionStep-level

Let the screen select the step.

Match the current screen to recorded states in task-relevant trajectories, using the GUI backbone’s latent representations. Supply the matching action as a reference, and repeat after every observation.

Current screen→State match→Action reference
Before executionTask-level

Let the context activate the routine.

Evaluate recurring intents against the current time and scenario. Suggest a representative intent only when the supporting history is sufficient. Otherwise, abstain.

Time + scenario→Support→Suggest / abstain

Interactive demonstration

Two ways to personalize a GUI agent.

    Applicable experienceStep-level activation

    Illustrative simulation of the method. App screens, preferences, prices, and weather are example data; no live agent or real transaction.

    Instruction-driveniPhone simulation

    01 / 06

    Results on AndroidIntent

    The best personalized execution on all four evaluated backbones. No fine-tuning required.

    Evaluated on AndroidIntent: 82 users, 135 applications, 775 execution episodes, and 315 proactive states. Within each backbone, every method shares the same history and frozen policy.

    Results at a glance
    View all Table 1 results
    AndroidIntent results (%) · Table 1 in the paper
    Backbone / methodType ↑SSR ↑CSR ↑Sem. ↑P ↑BA ↑MCC ↑
    A closer look · Qwen3-VL-8B

    730.6 → 144.6History tokens per step, Retrieve → ExpActivator.

    18.35% → 26.67%Within-trajectory step success with the same comparison.

    Source: Table 2. History tokens are the prompt tokens added over the no-history condition.

    Follow the screen, not a fixed script.

    “Check the next bus from PGPR to Kent Ridge MRT.”

    Three current NUS NextBus screens paired with activated historical screens. Similarities are 0.96, 0.97 and 0.93 as the task advances from direction search to route selection and the map.
    Figure 02 Each screen activates a matching recorded state. References can move between past trips as the interface changes, rather than following one historical trajectory from start to finish.

    Citation

    @misc{zhang2026expactivator,
  title  = {Relevance Does Not Imply Applicability:
            Experience Activation for Personal GUI Agents},
  author = {Fuyao Zhang and Xuan Wang and Zherui Li and
            Jiaming Zhang and Longtao Huang and Wei Yang Bryan Lim},
  year   = {2026},
  eprint = {2609.33304},
  archivePrefix = {arXiv},
  primaryClass = {cs.CV},
  url    = {https://arxiv.org/abs/2609.33304}
}