Axis I · Formation and Representation
How raw experience becomes memory
Formation decides what an agent keeps from what it sees and hears, how that stream is segmented into memory units, and in what form those units persist. The survey classifies each memory by the form finally handed back to the model, not by how it is indexed. Three families, six forms, and many hybrids.
- image or region
- video frame or clip
- audiovisual event
- object track
- interface state
- action–observation transition
- trajectory
- episode
From modality to retrieval, across the corpus
486 memory paths from 297 systems, traced from the input modality to the stored form, the store that holds it, and how it is found again. Band width counts paths; a path using several stores or retrieval methods is split evenly between them. Hover to trace a flow, or pick a modality.
One observation, six possible memories
The same moment can be kept as pixels, features, text, a graph, a hidden state, or a change to the weights. Pick a form to see how it works, what it keeps, and which systems use it.
The same observation, every time
Frame 1042, kitchen camera, 08:10. A mug and keys sit on the counter. The user says, “I’ll be back at six.”
Source- and modality-specific
Retained perceptual source evidence
Observations are kept in their native modality as addressable entries and handed back to the model as evidence, not only through a derived caption or embedding.
1/4 The robot’s camera sees the kitchen at 08:10.
What gets stored
frame_1042.jpg (1920×1080)
crop_mug_keys.png
audio_0810.wav “I’ll be back at six.”
index: { time: 08:10, camera: kitchen } Typical units: images, crops, document screenshots, video frames, audiovisual segments, RGB-D observations
Three families, three trade-offs
165 systems use it
Source- and modality-specific
Keep experience close to how it was perceived: raw observations, or features that preserve their structure.
Trade-off. Rich detail and re-encodable, but costly to store and process; encodings are lossy and tied to one model.
202 systems use it
Cross-represented explicit
Translate perception into explicit text, records, or graphs a language model can read and edit.
Trade-off. Interpretable and editable, but perceptual detail is discarded.
62 systems use it
Internalized latent & parametric
Carry experience as hidden state or write it into weights and adapters.
Trade-off. Tightly integrated with inference, but harder to access, trace, and control.
Hybrid memory
139 systems (47%) combine forms. The most common pattern, used by 97 systems, combines explicit records with source- or modality-specific representations. In source-linked designs, a record points back to the evidence it came from.
Named in the paper: VimRAG, FOLIO, MementoGUI, ObsGraph, HiMe
1/4 A compact text record and the original frame are both kept.
Which forms each modality uses
Systems using each form. A system with several forms counts once in each. Click a bar to list those systems.
- Retained perceptual source evidence 100
- Modality-specific encodings & geometric state 78
- Perception-to-text records & summaries 78
- Spatial, temporal & cross-modal structured memory 159
- Persistent multimodal latent state 39
- Experience-conditioned parametric memory 25
| Category | Image | Video & streaming | Audio & speech | GUI & computer use | Embodied & 3D |
|---|---|---|---|---|---|
| Retained perceptual source evidence | 27 | 39 | 2 | 11 | 21 |
| Modality-specific encodings & geometric state | 7 | 26 | 3 | 2 | 40 |
| Perception-to-text records & summaries | 24 | 21 | 3 | 15 | 15 |
| Spatial, temporal & cross-modal structured memory | 35 | 29 | 4 | 29 | 62 |
| Persistent multimodal latent state | 5 | 14 | 4 | 3 | 13 |
| Experience-conditioned parametric memory | 3 | 7 | 0 | 3 | 12 |
How memory is found again
Retrieval and activation methods by modality. Planner- and tool-driven retrieval is as common as similarity search.
- Similarity search 145
- Metadata / spatiotemporal retrieval 102
- Structure traversal 112
- Planner / tool-assisted retrieval 145
- Latent / parametric activation 71
| Category | Image | Video & streaming | Audio & speech | GUI & computer use | Embodied & 3D |
|---|---|---|---|---|---|
| Similarity search | 41 | 38 | 4 | 17 | 45 |
| Metadata / spatiotemporal retrieval | 14 | 37 | 3 | 11 | 37 |
| Structure traversal | 18 | 18 | 2 | 15 | 59 |
| Planner / tool-assisted retrieval | 34 | 42 | 3 | 28 | 38 |
| Latent / parametric activation | 7 | 24 | 4 | 7 | 29 |
Can the agent get back to the source?
Abstraction saves space but can cut the link to the original evidence. Only some memory paths keep the raw observation available, or link back to it, so that a summary can be checked against what was actually seen.
- Source directly available102
- Source linked & recoverable144
- Retained but not accessible to the agent46
- Source not retained193