Multimodal Agent MemorySubmit a paper

SKILL.nb

SKILL.nb: Selective Formalization and Gated Execution for Durable Agent Workflows

Amine El Hattami, Nicolas Chapados, Christopher Pal · 2026-06-06

Read the paperPDF

Management
T3 Self-Evolving Memory
Feedback during deployment changes the memory-management policy itself.
Primary modality
GUI & computer use
Also: Image, text, code
Representation
Hybrid
Source-preserving screenshots and outputs remain linked to the executable versioned workflow that they validate.

How experience becomes memory

Each path is one memory component, traced from what is stored to how it is found again.

  1. versioned notebook screenshots, outputs, and error traces

    1. Represented asRetained perceptual source evidenceSource- and modality-specific
    2. Stored inHierarchical multi-tier store
    3. Retrieved byMetadata / spatiotemporal retrieval
    4. Source evidenceSource directly available
    Versioned workflow notebooks preserve code, outputs, screenshots, and error traces that support later maintenance and validation.Full paper: §3 and Appendix A · pp. 5, 17–27 · Workflow and loader examples

  2. released versioned workflow notebooks

    1. Represented asSpatial, temporal & cross-modal structured memoryCross-represented explicit
    2. Stored inHierarchical multi-tier store
    3. Retrieved byPlanner / tool-assisted retrieval
    4. Source evidenceSource linked & recoverable
    Natural-language instructions, executable code, visual outputs, validation evidence, and lifecycle metadata are organized as versioned durable workflows and loaded for repeated execution.Full paper: §3 and Appendix A · pp. 5, 17–27 · Lifecycle and loader examples

How memory changes (T2 operations)

  • Update · Prompted LLM/VLM record update

    Runtime failures and local repairs generate proposals that maintenance can refactor, validate, and promote as new workflow versions.

  • Consolidation · Merging & deduplication

    Repository maintenance can consolidate duplicate workflow variants before promotion.

  • Reorganization · Memory-level migration (short- to long-term)

    Evidence-calibrated lifecycle gates promote, demote, release, or retire workflow versions.

How the policy evolves (T3)

  • Retrieval policy

    Execution failures and validated repairs persistently revise versioned workflow procedures; promoted versions replace earlier procedures for later repeated multimodal workflow executions.

Related systems