ExpActivator

Relevance Does Not Imply Applicability: Experience Activation for Personal GUI Agents

Fuyao Zhang1Xuan Wang1Zherui Li1Jiaming Zhang1Longtao Huang2Wei Yang Bryan Lim1

1 Nanyang Technological University2 Alibaba Group

Paper Code

Two ways to make an agent personal

Your habits.
The next right action.

    Applicable experienceStep-level activation

    Illustrative simulation of the method. App screens, preferences, prices, and weather are example data; no live agent or real transaction.

    Instruction-driveniPhone simulation

    01 / 06
    +28%

    Within-trajectory step success

    Relative to no history, averaged across four backbones
    ~1/5

    History tokens

    Compared with task-level retrieval
    2.3×

    Proactive suggestion MCC

    Approximately, vs. the strongest proactive baseline
    01 / Perspective

    Retrieving is only the beginning.

    “Order my usual coffee” leaves a lot unsaid. Personal history can identify the app and the drink. But a relevant past journey doesn’t tell the agent which action fits this screen—or whether coffee is even due now.

    Task-level retrieval strongly improves the opening decision, yet offers little sustained benefit over the remaining 90% of steps. ExpActivator turns personalization into a decision-time selection problem: use the current situation to activate the experience that applies.

    Where does history help?Qwen3-VL-8B · Step success rate (%)
    Opening stepFollowing steps
    No history
    28.26
    18.43
    Retrieve
    62.06
    18.35
    ExpActivator
    70.06
    26.67

    Retrieval helps most at the opening. Activation continues to help across the 7,140 actions that follow. Source: Table 2.

    Read the full abstract

    Personal Graphical User Interface (GUI) agents rely on interaction history to infer what a user wants from ambiguous instructions and to anticipate recurring routines. Existing approaches retrieve task-relevant history and append it to the policy’s context, implicitly assuming that experience relevant to a task remains useful for each decision within it. We find that this help is largely spent at the first decision: retrieved history strongly improves the opening step of an episode, yet provides little sustained benefit over the remaining 90% of steps, and offers weak guidance on whether a proactive suggestion is warranted. A relevant record may tell the agent where to begin, but not which past action applies to the current screen or whether a routine is due now. The underlying issue is that relevance does not imply applicability: relevance is determined at the task level, whereas applicability depends on the situation at decision time.

    We therefore recast personalization as experience activation and introduce ExpActivator, a training-free framework that activates only the experience applicable to the current situation. During execution, ExpActivator matches each new screen to historical states in the frozen GUI backbone’s latent space and supplies the corresponding action as a reference. Before execution, it activates a recurring intent only when the current time and scenario provide sufficient support, and otherwise abstains. Across four GUI backbones, ExpActivator improves within-trajectory step success by 28% on average, achieves the best personalized execution on every backbone while using about one-fifth as many history tokens, and reaches approximately 2.3× the Matthews correlation coefficient of the strongest proactive baseline. Experience pays where it is activated, not where it is appended.

    02 / Method

    One history. Two levels of activation.

    A frozen backbone, with experience selected at the moment it matters.

    During executionStep-level

    Let the screen select the step.

    Match the current screen to recorded states in task-relevant trajectories, using the GUI backbone’s latent representations. Supply the matching action as a reference, and repeat after every observation.

    Current screen→State match→Action reference
    Before executionTask-level

    Let the context activate the routine.

    Evaluate recurring intents against the current time and scenario. Suggest a representative intent only when the supporting history is sufficient. Otherwise, abstain.

    Time + scenario→Support→Suggest / abstain
    ExpActivator framework: task-relevant histories are matched to current visual states for action references, while routine neighborhoods are scored by time and scenario for proactive suggestions.
    Figure 01 The ExpActivator framework. All encoders and policy models remain frozen. Historical actions guide the policy; they are not prescribed operations.
    03 / Evidence

    Better decisions, less history in context.

    The best personalized execution on all four evaluated backbones. No fine-tuning required.

    Evaluated on AndroidIntent: 82 users, 135 applications, 775 execution episodes, and 315 proactive states. Within each backbone, every method shares the same history and frozen policy.

    Results at a glance
    View all Table 1 results
    AndroidIntent results (%) · Table 1 in the paper
    Backbone / methodType ↑SSR ↑CSR ↑Sem. ↑P ↑BA ↑MCC ↑
    A closer look · Qwen3-VL-8B

    730.6 → 144.6History tokens per step, Retrieve → ExpActivator.

    18.35% → 26.67%Within-trajectory step success with the same comparison.

    Source: Table 2. History tokens are the prompt tokens added over the no-history condition.
    04 / In practice

    Follow the screen, not a fixed script.

    “Check the next bus from PGPR to Kent Ridge MRT.”

    Three current NUS NextBus screens paired with activated historical screens. Similarities are 0.96, 0.97 and 0.93 as the task advances from direction search to route selection and the map.
    Figure 02 Each screen activates a matching recorded state. References can move between past trips as the interface changes, rather than following one historical trajectory from start to finish.

    Experience pays where it is activated,
    not where it is appended.

    Read the full paper
    05 / Reference

    Citation

    @misc{zhang2026expactivator,
  title  = {Relevance Does Not Imply Applicability:
            Experience Activation for Personal GUI Agents},
  author = {Fuyao Zhang and Xuan Wang and Zherui Li and
            Jiaming Zhang and Longtao Huang and Wei Yang Bryan Lim},
  year   = {2026},
  url    = {https://fyzhang1.github.io/ExpActivator}
}