Multimodal Agent MemorySubmit a paper

HiKEY

HiKEY: Hierarchical Multimodal Retrieval for Open-Domain Document Question Answering

Joongmin Shin, Gyuho Shim, Jeongbae Park, Jaehyung Seo, Heuiseok Lim · 2026-05-28

Read the paperPDF

Management
T1 Static Memory
Memory is built once and reused without change during deployment.
Primary modality
Image
Also: text
Representation
Hybrid
The retained multimodal unit store and the hierarchy tree are deliberately linked: tree traversal supplies Upper Context that enriches the stored visual units, and the two-stage scorer consumes the hierarchical routing signal and the unit-level multimodal signal jointly.

How experience becomes memory

Each path is one memory component, traced from what is stored to how it is found again.

  1. Section-level cards holding text, table and image units as first-class retrieval units over the document corpus

    1. Represented asRetained perceptual source evidenceSource- and modality-specific
    2. Stored inHierarchical multi-tier store, Vector-indexed collection
    3. Retrieved bySimilarity search
    4. Source evidenceSource directly available
    Each document is decomposed into Doc_cards (title plus section paths, used for global routing) and Sec_cards, whose entries are multimodal units typed as Text, Table or Image. Tables and figures are retained as first-class retrieval units rather than being replaced by captions, and the retrieved visual units are delivered to the reader, so the perceptual source is directly available downstream. Stage-1 routing ranks Doc_cards with a hybrid lexical-plus-dense score; Stage-2 globally ranks constituent Sec_cards and their units with a type-specific unit score combining text-semantic and dense visual-semantic signals.Full paper (arXiv HTML v1): Sec. 3.1 Offline hierarchy-aware graph and index construction; Sec. 3.2 Type-Specific Unit Score · HTML Sec. 3.1-3.2 · Table 1 (document-level retrieval), Table 2 (end-to-end QA)

  2. Document Hierarchical Parsing tree linking blocks as parent-child nodes with ancestor section paths, traversed to attach Upper Context to caption-less visual units

    1. Represented asSpatial, temporal & cross-modal structured memoryCross-represented explicit
    2. Stored inGraph / relational store, Hierarchical multi-tier store
    3. Retrieved byStructure traversal
    4. Source evidenceSource linked & recoverable
    Documents are reconstructed into a hierarchical tree T(d) whose blocks are nodes connected by parent-child tree edges, with each unit assigned an ancestor section path such as Title > Section 5 > 5.3. For visual units lacking captions, HiKEY traverses the tree upward to the logically preceding textual node and attaches it as Upper Context, so the tree structure is itself operative at retrieval time and links every enriched unit back to its originating document block.Full paper (arXiv HTML v1): Sec. 3.1 Document Hierarchical Parsing and Contextual Enrichment via Graph Traversal · HTML Sec. 3.1 · Sec. 3.1 index-construction description

Related systems