H-JEPA: End-to-End Learning of Hierarchical World Models for Visual Planning

Latent world models promise to plan in imagination, but most plan at one clock speed. That is a poor match for real tasks, whose structure lives at several timescales at once — slow task progress on top of fast, noisy motion underneath. H-JEPA’s argument is that the hierarchy should be learned, and learned as a whole.

Background

Planning with learned latent world models requires reasoning across timescales and levels of abstraction: a goal is pursued over a long horizon while the details that matter change on a much shorter one. Existing task-agnostic JEPA world models are built for a single timescale, or they carry several prediction horizons inside one shared latent space. The second design keeps everything in one representation, which forces the model to spend capacity on fast, unpredictable detail that is irrelevant to long-horizon progress, and it does not expose a clean interface between a coarse plan and the fine behavior that carries it out.

Core Innovation

H-JEPA is an end-to-end recipe for training a hierarchy of action-conditioned JEPAs in which each level predicts farther ahead in its own learned latent space. Because the levels have separate latent spaces and separate prediction horizons, the representation at each level can specialize. Planning then proceeds top-down: the top level optimizes progress toward the goal, and each level’s predictions serve as subgoals for the planner at the level below it. The mechanism the authors highlight is that when the factors in the data evolve at separated timescales, higher levels naturally discard the fast, unpredictable detail and retain the slower, task-relevant state — the hierarchy is not merely a scheduling convenience but a way of allocating representational capacity to the right timescale. The whole stack is trained end-to-end rather than assembled from separately trained components.

Results

• Across four simulated navigation and manipulation environments, hierarchical planning improves over a flat JEPA.
• On Visual AntMaze, a three-level hierarchy raises success from 18% to 73% while using less planner compute.
• Ablations attribute the gains to both temporal decomposition and higher-level goal representations, rather than to the extra parameters alone.
• With inverse-dynamics supervision, the approach extends to diverse real-robot videos from DROID, where the hierarchy improves offline planning fidelity at lower planner compute.

Limitations

The headline result is measured in simulation, and the real-robot evidence is offline planning fidelity on DROID video rather than closed-loop control on hardware, so the gains have not been shown to survive contact with a live robot. The mechanism depends on the data actually containing factors that evolve at separated timescales; environments without that structure would give the hierarchy little to exploit, and the abstract does not characterize when the approach fails. The ablations are reported at a summary level, so the relative contribution of the number of levels, their horizons, and the goal representations cannot be read off directly.

Industry Implications

Planner compute and long-horizon coherence are two of the persistent obstacles to putting world models in real control loops, and H-JEPA offers a way to improve the second while reducing the first. For teams building on JEPA-style latent models, the transferable idea is that temporal decomposition belongs inside the learned representation, where it can free capacity from short-timescale noise, rather than in an external scheduler. The recipe is also appealing because it is trained end-to-end from action-conditioned data, which fits the large, heterogeneous video corpora teams already collect. As always with world-model results, the practical question is whether the improvement holds under closed-loop control on real hardware, which this work leaves to future study.