I wrote a posts recently on the need for agentic systems to have semantic-associative memory capabilities. Someone asked whether something like LeCun’s Joint Embedding Predictive Architecture (JEPA) gives us semantic memory.
The answer: not really, but it does potentially do many other good things.
JEPA, as I understand it, trains a model to predict latent representations of future or missing input. That pushes the system to learn abstract features, not just reconstruct pixels or tokens. It's a meaningful step away from shallow pattern matching and toward conceptual understanding.
But semantic-associative memory — the kind humans use to form, refine, and reuse concepts — needs more. To mention just a few items:
Persistent structure. Human memory stores and stabilizes concepts over time; JEPA’s representations are ephemeral unless paired with external memory.
Explicit associations. Human concepts are richly linked (by function, causality, hierarchy). JEPA relies on proximity in embedding space, which is statistically shallow by comparison.
Learning dynamics. Humans consolidate and reinforce categories through use and salience. JEPA lacks this selective strengthening over time.
JEPA gives us abstraction without accumulation. It could be foundational for building systems with real semantic-associative memory, but it's not really sufficient to do so in its current formulation.
