Temporal Policy: History-Initialized Action Generation for Robotic Learning from Demonstration

Dylan Miller
Supervisor: Martin Jagersand
Computing Science
University of Alberta
Paper Thesis Slides Code

Standard generative models for robotic LfD initialize with random noise. This creates a complex mapping between an uninformative prior and the action space. Instead, Temporal Policy initializes the generative process with the recent robot state. This formulation reduces transport costs and vector field curvature, enabling faster sampling.

Abstract

Generative modeling has emerged as a powerful paradigm for robotic imitation learning, enabling policies to capture complex, multimodal action distributions from expert demonstrations. However, predominant approaches such as Diffusion Policy typically rely on iterative denoising from uninformative Gaussian noise. This approach forces the network to bridge a significant geometric gap between a high-entropy prior and the physical configuration space, resulting in long and potentially high-curvature transport paths. This transport complexity acts as a fundamental bottleneck, imposing computational burdens and inference latency that limit deployment on real-time, resource-constrained robotic systems.
This thesis introduces Temporal Policy, an action generation framework formulated using stochastic interpolants. Our approach replaces the uninformative noise source distribution with the robot's recent state history. Mapping recent state history to future actions provides a structurally simpler regression objective, allowing the model to learn a direct, low transport cost vector field. This enables fast sampling in the robot's configuration space rather than an abstract noise space. Furthermore, the stochastic interpolant framework allows a single trained network to be deployed using either low-latency deterministic sampling or robust stochastic sampling.
Empirical evaluations on visuomotor simulation benchmarks and a physical 7-DoF WAM teleoperation platform validate this geometric efficiency. On the Robomimic benchmark suite, Temporal Policy matches state of the art success rates, while reducing transport costs by nearly an order of magnitude compared to standard baselines. This is further demonstrated through single step inference, where our method outperforms noise initialized baselines. These attributes allow low-order ODE solvers to achieve high-fidelity action generation with a minimal number of integration steps, enabling higher frequency, real-time control. Furthermore, unlike methods operating in abstract noise spaces, the resulting vector fields act directly as physical kinematic velocities.

Method Overview

Method Comparisson

Overview of Temporal Policy. Unlike standard generative models that initialize from independent Gaussian noise ($\mathbf{x}_0 \sim \mathcal{N}(0, \mathbf{I})$), our method explicitly initializes the generative process with the robots recent state history. The architecture consists of: (1) A Observation Encoder that processes sensory observations into a conditioning vector $c$; (2) A U-Net Backbone that predicts the drift $b_\theta(\mathbf{x}_\lambda, \mathbf{x}_0, \lambda, c)$; and (3) An ODE/SDE Solver that integrates this vector field from $\lambda=0$ to $\lambda=1$ to generate the target action sequence.


Temporal Policy takes advantage of the fact that in robotics, state and action usually share the same representation space (e.g., joint positions or end-effector pose). This allows us to define a generative process that maps from past state to future actions.

Method Comparisson

Temporal policy maps a historical state sequence $\mathbf{x}_0$ of horizon $H$ to target sequence $\mathbf{x}_1$, shifted forward in time by an offset of $d$ control steps.

Temporal Policy provides several benefits over independent, noise-initialized approaches:
  • Reduces transport costs by nearly an order of magnitude.
  • Straighter vector fields that allow ODE/SDE solvers to take fewer sampling steps, resulting in faster inference.
  • Vector fields represent the robot's physical velocity and intermediate states remain close to the robots configuration space.
Overall Temporal Policy provides a method capable of representing complex multimodal action distributions, but reduces the computational complexity of standard noise initialized generative models.

Simulation Tasks

Our Temporal Policy matches state-of-the-art performance on the Robomimic Benchmark Suite. It achieves an inference latency of 19ms by combining a 15x smaller network with straighter vector fields that require fewer sampling steps.

Real World

We validate our Temporal Policy on a real world mug hanging task. We achieved 95% (19/20) success during the grasp phase and 50% (10/20) success on the hang phase.