Multimodal Agent MemorySubmit a paper

Vesta

Vesta: A Generalist Embodied Reasoning Model

Johan Bjorck, Zhiqi Li, Yunze Man, Jing Wang, An-Chieh Cheng, Sifei Liu, et al. · 2026-06-18

Read the paperPDF

Management
T2 Dynamic Memory
Memory changes during deployment through a stable management policy.
Primary modality
Embodied & 3D
Also: Image, text
Representation
Hybrid
Sampled source images and the timestamped textual subtask history are jointly re-injected into the planner prompt to select later embodied subtasks.

How experience becomes memory

Each path is one memory component, traced from what is stored to how it is found again.

  1. sampled history of egocentric observations retained by the planner memory harness

    1. Represented asRetained perceptual source evidenceSource- and modality-specific
    2. Stored inFlat sequential records
    3. Retrieved byMetadata / spatiotemporal retrieval
    4. Source evidenceSource directly available
    At each planning step, the explicit memory harness retains timestamped egocentric observations from prior steps. It re-injects at most K historical images into the planner using either uniform or recency-biased temporal sampling, while always retaining the first frame to preserve the initial state.Full paper: Sec. 2.4 Action Planning with Memory · PDF p. 4 · Figure 3; memory-harness definition

  2. timestamped running cache of prior text subtasks and goal-conditioned progress

    1. Represented asSpatial, temporal & cross-modal structured memoryCross-represented explicit
    2. Stored inFlat sequential records
    3. Retrieved byMetadata / spatiotemporal retrieval
    4. Source evidenceSource not retained
    The history is an ordered set of tuples containing step index, timestamp, observation, predicted subtask and goal. The planner writes each predicted text subtask to this running memory and conditions later non-Markovian planning decisions on the resulting compressed context.Full paper: Sec. 2.4 Action Planning with Memory · PDF p. 4 · Memory-harness tuple definition

How memory changes (T2 operations)

  • Update · Rule-based update

    During deployment, each planner call appends the latest observation/subtask step to the planner-owned memory, and subsequent calls use the updated history. The write and temporal-sampling rules are fixed, so the persistent state evolves without policy evolution.

Related systems