Diffusion Policy

Human demonstrations are multimodal: faced with an obstacle, one operator goes left, another goes right, and both are correct. A policy trained with mean-squared error on that data learns the average of the modes, and the average of "left" and "right" is "straight into the obstacle." Diffusion Policydiffusion policyA policy that produces actions through a conditional denoising diffusion process over a whole action sequence instead of regressing a single action directly. Introduced by Chi et al. in 2023, it represents multimodal demonstrations faithfully rather than averaging their modes, and it executes only the first few steps of each predicted sequence before observing again and replanning., introduced by Chi et al. in 2023, sidesteps the averaging problem by changing what the policy outputs: not an action, but a conditional denoising process over a whole action sequence Chi 2023↓Diffusion Policy: Visuomotor Policy Learning via Action DiffusionCheng Chi, Zhenjia Xu, Siyuan Feng et al., 2023. The extended journal version describes a 15-task evaluation. It reports a 46.9% average relative improvement for its simulation comparisons; Appendix B.2 selects the strongest baseline and diffusion backbone per reported column and excludes multi-human results Chi 2023↓Diffusion Policy: Visuomotor Policy Learning via Action DiffusionCheng Chi, Zhenjia Xu, Siyuan Feng et al., 2023.

Drag the denoising step slider below from k=0 to k=10. The cloud starts as pure Gaussian noise and collapses onto two separate action modes rather than the average of them, which is the property MSE regression destroys and this whole architecture exists to keep.

Denoising in action space

The policy is a denoising diffusion probabilistic model (DDPM) conditioned on recent observations. Where a regression head computes $a_t = f_\theta(o_t)$, Diffusion Policy models the distribution $p(A_t \mid O_t)$ over the next $T_p$ actions. Inference starts from pure Gaussian noise and iteratively refines it: each step applies the learned denoiser $\epsilon_\theta$, which predicts the noise component of the current sample, and subtracts a fraction of it,

repeated until the sample lands on a valid action chunk Chi 2023↓Diffusion Policy: Visuomotor Policy Learning via Action DiffusionCheng Chi, Zhenjia Xu, Siyuan Feng et al., 2023. Training uses the standard noise-prediction MSE loss with a square-cosine schedule; the paper's main text reports DDIM with 100 training diffusion timesteps, 10 inference iterations and 0.1-second inference latency on a Nvidia 3080 GPU for real-world experiments. Its hyperparameter appendix and real-world table rows instead specify 16 inference iterations. Both the original v1 and the extended paper contain these settings; the cited passages do not identify a configuration or revision change that reconciles them Chi 2023↓Diffusion Policy: Visuomotor Policy Learning via Action DiffusionCheng Chi, Zhenjia Xu, Siyuan Feng et al., 2023.

Because sampling is stochastic, the policy can commit to a mode instead of averaging modes. That is what the figure at the top of this module shows on a bimodal demonstration set: each sample lands on one strategy or the other, and the distribution of samples reproduces the distribution of demonstrations.

Receding-horizon control

Diffusion Policy executes $T_a$ actions from a $T_p$-action prediction before observing again and replanning. The CNN configuration uses eight of sixteen actions for many of the reported tasks; the formulation also permits warm-starting the next inference from a previous prediction Chi 2023↓Diffusion Policy: Visuomotor Policy Learning via Action DiffusionCheng Chi, Zhenjia Xu, Siyuan Feng et al., 2023. The tail of each chunk exists to make the committed part coherent; it is revised away before it ever runs.

$T_a$ is the same dial as ACT's chunk size, named differently: large $T_a$ gives smooth, committed motion but a slow reaction to anything changing mid-chunk; small $T_a$ reacts quickly but re-samples often, and each new sample can pick a different mode mid-motion Zhao 2023↓Learning Fine-Grained Bimanual Manipulation with Low-Cost HardwareTony Z. Zhao, Vikash Kumar, Sergey Levine et al., RSS 2023. The policy issues commands at 10 Hz, so $T_a = 8$ means the robot commits to each plan for 0.8 seconds.

CNN and transformer variants

The paper ships two denoiser backbones Chi 2023↓Diffusion Policy: Visuomotor Policy Learning via Action DiffusionCheng Chi, Zhenjia Xu, Siyuan Feng et al., 2023. Its CNN adopts a temporal backbone from Diffuser with modifications: it denoises action sequences conditioned on observations instead of a concatenated state-action trajectory Chi 2023↓Diffusion Policy: Visuomotor Policy Learning via Action DiffusionCheng Chi, Zhenjia Xu, Siyuan Feng et al., 2023. Diffuser's backbone is a one-dimensional temporal U-Net over state-action trajectories Janner 2022↓Planning with Diffusion for Flexible Behavior SynthesisMichael Janner, Yilun Du, Joshua B. Tenenbaum et al., ICML 2022:

The CNN variant is the recommended default: it trains stably and is robust to hyperparameter choices. The authors report strong transformer results particularly in state-based experiments with rapid action changes. They suggest that the CNN’s preference for low-frequency signals likely contributes to its difficulty on such tasks, and recommend trying the transformer as a potential improvement at the cost of more tuning; the paper reports that simply adding depth sometimes made results worse Chi 2023↓Diffusion Policy: Visuomotor Policy Learning via Action DiffusionCheng Chi, Zhenjia Xu, Siyuan Feng et al., 2023.

The successors: fewer steps, then flow matching

Iterative denoising buys multimodality at the cost of sampling time. Ten network evaluations per chunk is fine on a workstation GPU; it is a problem on a mobile manipulator or a quadrotor. Two successor lines attack the cost differently.

Distillation: fewer steps

Consistency Policy distills an EDM-based Diffusion Policy teacher into a consistency model with one- or three-step action-sequence generation, targeting compute-constrained robots. In its Robomimic Square ablation, teachers with success rates of 0.92, 0.88 and 0.84 produced students with success rates of 0.92, 0.92 and 0.88, respectively; the authors report robustness over this tested range Prasad 2024↓Consistency Policy: Accelerated Visuomotor Policies via Consistency DistillationAaditya Prasad, Kevin Lin, Jimmy Wu et al., 2024. One-Step Diffusion Policy goes further, distilling the chain into a single-step generator that lifts action prediction from 1.5 Hz to 62 Hz on real Franka tasks Wang 2024↓One-Step Diffusion Policy: Fast Visuomotor Policies via Diffusion DistillationZhendong Wang, Zhaoshuo Li, Ajay Mandlekar et al., 2024.

Flow matching: a different formulation

pi0 swapped the DDPM for conditional flow matchingflow matchingA generative modeling recipe that learns a time-dependent vector field carrying samples from a noise distribution to the data distribution, trained by regressing the model's field against the conditional flow rather than by estimating a score. pi0 brought it to robot control: a dedicated action expert is trained with the flow-matching objective to denoise continuous action chunks, and at inference the learned field is integrated as an ODE for a handful of steps, which keeps the policy fast enough for high-frequency control while keeping the multimodality that made diffusion policies attractive. on a straight transport path: sample noise $\varepsilon$, form the interpolant

and train the network to predict the target vector field $u$, with the timestep $\tau$ drawn from a beta distribution weighted toward the noisy end Black 2024↓π0: A Vision-Language-Action Flow Model for General Robot ControlKevin Black, Noah Brown, Danny Driess et al., RSS 2025, 2024. Sampling integrates the field from $\tau = 0$ to $\tau = 1$ with forward Euler in 10 steps. π0 fixes a linear noise-to-action interpolant and chooses both the training-timestep distribution and the numerical integration settings.

Limitations

The original 2023 paper evaluates observation-conditioned policies on single- and multi-task benchmarks and on separate UR5 and Franka setups. Its reference visual encoder is trained end-to-end without pretraining, but the paper also tests ImageNet- and R3M-pretrained encoders Chi 2023↓Diffusion Policy: Visuomotor Policy Learning via Action DiffusionCheng Chi, Siyuan Feng, Yilun Du et al., 2023. Octo combines language or goal-image conditioning with a DDPM diffusion action head and pretraining on demonstrations from multiple tasks and robot embodiments Octo Model Team 2024↓Octo: An Open-Source Generalist Robot PolicyOcto Model Team, Dibya Ghosh, Homer Walke et al., 2024. π0 combines a pretrained vision-language backbone with a flow-matching action expert and cross-embodiment robot training Black 2024↓π0: A Vision-Language-Action Flow Model for General Robot ControlKevin Black, Noah Brown, Danny Driess et al., RSS 2025, 2024. Reactivity is bounded by $T_a$: whatever happens inside a committed segment, the policy will not respond until the next inference. And the sampling cost, though tamed by DDIM and the successor methods above, never disappears entirely.