Multimodal Agent MemorySubmit a paper

MemoPhishAgent

MemoPhishAgent: Memory-Augmented Multi-Modal LLM Agent for Phishing URL Detection

Xuan Chen · 2026-04-21

Read the paperPDF

Management
T2 Dynamic Memory
Memory changes during deployment through a stable management policy.
Primary modality
Image
Also: text
Representation
Cross-represented explicit

How experience becomes memory

Each path is one memory component, traced from what is stored to how it is found again.

  1. episodic memory of past investigations: LLM-distilled keyword set (from page text plus rendered screenshot) as key; full tool-calling trajectory and final verdict as value

    1. Represented asPerception-to-text records & summariesCross-represented explicit
    2. Stored inVector-indexed collection
    3. Retrieved bySimilarity search
    4. Source evidenceSource not retained
    For each investigated URL the agent 'prompt[s] the LLM to distill compact keywords {k_i} ... given the page text and screenshot', embeds the keyword set with a pre-trained sentence encoder, and stores it 'in a vector index along with its value: the full tool-calling sequence <t_1,...,t_m> and the final verdict y_hat'. The persisted artifact is therefore a textualised record of a multimodal observation (rendered page screenshot plus visual-tool outputs); the screenshot itself is not retained, so the perceptual source is not recoverable from memory.Full paper: Sec. 3.3.1 (episodic memory construction) · HTML v3, Sec. 3.3.1 · framework figure (Fig. 1/2); Sec. 3.3 memory design
    At query time the incoming URL's keyword embedding is matched by cosine similarity against the index under a fixed threshold tau; the number k' of matches routes the agent to majority vote over stored verdicts (k' >= k), to reuse of retrieved trajectories as in-context exemplars (0 < k' < k), or to a full ReAct investigation (k' = 0).Full paper: Sec. 3.3.2 (hierarchical retrieval policy) · HTML v3, Sec. 3.3.2 · Fig. 3 (sensitivity to k and tau)

How memory changes (T2 operations)

  • Update · Rule-based update

    Every URL processed in deployment yields a new episodic entry: 'Each URL yields a tool-calling trajectory and decision, which we treat as reusable supervision: when similar URLs recur, the agent can retrieve prior investigations to accelerate and stabilize its reasoning.' The store therefore grows and changes during operation (the system runs in production over ~60K URLs weekly) under a fixed write rule, and later URLs are decided from entries written earlier at inference time.

  • Forgetting · Discrete eviction

    A time-window pruning policy discards stored trajectories during operation: 'after every 50 processed URLs, we prune the least recently used 20%, 40%, and 60% of stored trajectories separately', with detection performance reported as stable across the settings. Eviction is by a fixed LRU rule, so the content shrinks while the policy does not change.

Related systems