Jacob Thompson

Embodied Semantic Memory

Jacob Thompson, Haochen Zhang, Quanting Xie, Fan Yang, Yonatan Bisk

Neither humans nor Visual Language Models (VLMs) can effectively reason over increasingly vast scene contexts; attempting to process hours of raw video is both computationally prohibitive and prone to attention collapse. To achieve genuine large-scale scene understanding, embodied agents must instead continuously summarize and abstract their experiences into a compact, structured representation. We propose Embodied Semantic Memory (ESM), an agentic framework enabling VLMs to actively decide what spatial, semantic, and temporal information to store in a hierarchical graph. We first validate ESM on the standard Open-EQA benchmark, where it achieves parity with human baselines on episodic queries. To stress-test the paradigm on longer horizons, we introduce Campus-Bench, a continuous 2-hour real-world egocentric walkthrough containing over 3,000 queries that require reasoning across district-scale environments. On this benchmark, ESM demonstrates superior reasoning capabilities compared to naive memory baselines, while matching or exceeding the accuracy of brute-force long-context approaches with drastically reduced computational overhead. By shifting the focus from ever-longer context windows toward agentic, structured memory, we outline a path toward more generally useful embodied agents.