Multimodal Agent MemorySubmit a paper

VISTA

VISTA: A Visual Harness for Reasoning in an Interactive World

Qiushi Han, Keya Hu, Linlu Qiu, Cathy Wu, Kaiming He · 2026-10-01

Read the paperPDF

Management
T2 Dynamic Memory
Memory changes during deployment through a stable management policy.
Primary modality
Image
Also: text
Representation
Hybrid
Original visual evidence and revisable explicit notes are separately addressable and jointly used for subsequent reasoning and action.

How experience becomes memory

Each path is one memory component, traced from what is stored to how it is found again.

  1. lossless indexed visual interaction history

    1. Represented asRetained perceptual source evidenceSource- and modality-specific
    2. Stored inHierarchical multi-tier store
    3. Retrieved byPlanner / tool-assisted retrieval
    4. Source evidenceSource directly available
    Every environment-returned frame is retained losslessly with turn and frame indices. inspect and read_pixels revisit original frames, regions, and pixels.Lossless visual memory · official project, accessed 2026-10-09

  2. durable game understanding and current-level notes

    1. Represented asPerception-to-text records & summariesCross-represented explicit
    2. Stored inHierarchical multi-tier store
    3. Retrieved byPlanner / tool-assisted retrieval
    4. Source evidenceSource not retained
    GUIDE.md retains durable revisable game understanding; WORKING.md retains current-level state. Notes, visual memory, and action history survive context restarts.The Complete Pipeline: Reason, then act · official project, accessed 2026-10-09

How memory changes (T2 operations)

  • Update · Prompted LLM/VLM record update

    The same reusable prompt asks the agent to update its compact game model when new evidence changes what is supported; the notes change, without evidence that the memory-management policy itself is persistently revised.

Related systems