RL for Robotics

Reinforcement learning (RL) for robotics trains robot policies from their own experience of acting, formalized as a Markov decision process of states, actions, transitions and rewards^{Source 1}1 Sutton et al. 2018, Reinforcement Learning: An Introduction. Richard S. Sutton, Andrew G. Barto, MIT Press, second edition, 2018. A simulated step is nearly free when thousands of GPU robots act at once^{Source 2}2 Rudin et al. 2021, Learning to Walk in Minutes Using Massively Parallel Deep Reinforcement Learning. Nikita Rudin, David Hoeller, Philipp Reist et al., CoRL 2021, while a real step costs machine time and a reset.

Photo: Philipp Wu et al. / arXiv. Licence: CC BY 4.0.

What a budget buys, and what it forbids

A review of robotic deep RL case studies lists real-world challenges that research on games and simulated control seldom addresses^{Source 23}23 Ibarz et al. 2021, How to Train Your Robot with Deep Reinforcement Learning; Lessons We've Learned. Julian Ibarz, Jie Tan, Chelsea Finn et al., International Journal of Robotics Research, 2021, and a 2025 survey of real-world successes calls stable, sample-efficient real-world RL a priority^{Source 24}24 Tang et al. 2025, Deep Reinforcement Learning for Robotics: A Survey of Real-World Successes. Chen Tang, Ben Abbatematteo, Jiaheng Hu et al., Annual Review of Control, Robotics, and Autonomous Systems, 2025.

Two ways to spend a transition

An on-policyon-policyAn on-policy method is a learning method whose updates are valid only for data from the current policy, so each batch of experience is used once and discarded. PPO, the robotics default, alternates between sampling interaction data and optimizing a surrogate objective that allows several minibatch epochs per batch where plain policy gradient takes one step. This is costly when samples are expensive and nearly free when a GPU simulator yields hundreds of millions. method can only update on data the current policy generated, so each batch is collected, consumed and discarded; PPO allows several minibatch epochs per batch, where plain policy gradient takes one step^{Source 7}7 Schulman et al. 2017, Proximal Policy Optimization Algorithms. John Schulman, Filip Wolski, Prafulla Dhariwal et al., 2017. An off-policyoff-policyAn off-policy method is a learning method that can improve one policy from data generated by another, so old experience stays usable. Haarnoja and colleagues' soft actor-critic, the robotics reference point, combines off-policy updates with a maximum-entropy objective against the sample complexity that keeps model-free deep RL off real hardware. They reported substantially better sample efficiency than on-policy methods on the same tasks. method improves one policy from data another produced, so experience accumulates in a replay bufferreplay bufferA replay buffer is the store of past transitions an off-policy learner samples gradient batches from. Its contents need not come from the current policy or even the learner. RLPD mixes prior offline data with fresh online experience in one buffer, and HIL-SERL seeds one with human demonstrations before autonomous collection.. Underneath is Q-learningQ-learningQ-learning is learning a function predicting the return from each state-action pair, instead of a policy directly, then picking the action its own estimate rates highest. Sutton and Barto present it as the archetypal off-policy temporal-difference control method. Its update uses the best next action, whatever the behaviour policy did, so it can learn from experience it did not generate. Many off-policy and offline robotics algorithms build on it, adding safeguards against value estimates running away on unverifiable data.: fit the value of a state-action pair and bootstrap from the best next action, even one never taken^{Source 1}1 Sutton et al. 2018, Reinforcement Learning: An Introduction. Richard S. Sutton, Andrew G. Barto, MIT Press, second edition, 2018.

TD3 reduced DDPG's overestimation bias with clipped double critics, delayed policy updates and target policy smoothing^{Source 9}9 Fujimoto et al. 2018, Addressing Function Approximation Error in Actor-Critic Methods. Scott Fujimoto, Herke van Hoof, David Meger, ICML 2018. Soft actor-critic added a maximum-entropy objective aimed at sample complexity and brittle convergence^{Source 8}8 Haarnoja et al. 2018, Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor. Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel et al., ICML 2018.

Why robotics defaulted to PPO anyway

With thousands of GPU environments, steps are nearly free and training stability is the scarce resource. SAC's authors credit on-policy training with stability at the price of "poor sample complexity", and call off-policy learning with neural function approximation "a major challenge for stability and convergence"^{Source 8}8 Haarnoja et al. 2018, Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor. Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel et al., ICML 2018.

Paper figure: Viktor Makoviychuk et al. / arXiv. Licence: CC BY 4.0.

Rudin and colleagues trained ANYmal with 4,096 parallel robots, 98,304 steps per update and 1,500 updates in under twenty minutes on one workstation GPU, more than 122,880 environment steps per second^{Source 2}2 Rudin et al. 2021, Learning to Walk in Minutes Using Massively Parallel Deep Reinforcement Learning. Nikita Rudin, David Hoeller, Philipp Reist et al., CoRL 2021. Minitaur's 160,000 control steps over about two hours average about 22 per second, below its 50 Hz rollout rate because resets and overhead count^{Source 3}3 Haarnoja et al. 2019, Learning to Walk via Deep Reinforcement Learning. Tuomas Haarnoja, Sehoon Ha, Aurick Zhou et al., RSS 2019. Parallel simulation covers the first regime.

Most GPU-simulation work used PPO for its simplicity and ease of scaling, while off-policy methods were more data-efficient but slower in wall-clock time; Parallel Q-Learning scales to tens of thousands of environments and beats PPO in wall-clock time^{Source 25}25 Li et al. 2023, Parallel Q-Learning: Scaling Off-policy Reinforcement Learning under Massively Parallel Simulation. Zechu Li, Tao Chen, Zhang-Wei Hong et al., ICML 2023, and FastSAC and FastTD3 train sim-to-real humanoid locomotion in 15 minutes on one RTX 4090^{Source 26}26 Seo et al. 2025, Learning Sim-to-Real Humanoid Locomotion in 15 Minutes. Younggyo Seo, Carmelo Sferrazza, Juyue Chen et al., 2025.

What sample efficiency looks like in step counts

Sample efficiencysample efficiencySample efficiency is how much environment interaction a method needs to reach a set performance, counted apart from wall-clock time and gradient updates. Minitaur walking used 160,000 control steps over about two hours of the whole training process. A constant-rate budget illustration can turn steps into modelled time; that does not establish algorithm eligibility or predict another robot task. is how much experience a method needs to learn. Haarnoja and colleagues learned Minitaur locomotion from 160,000 control steps in about two hours of real-world time that includes resets and the operator^{Source 3}3 Haarnoja et al. 2019, Learning to Walk via Deep Reinforcement Learning. Tuomas Haarnoja, Sehoon Ha, Aurick Zhou et al., RSS 2019. DayDreamer trained a Unitree A1 to roll over, stand and walk in one hour without a simulator, and ten more minutes of online learning taught it to withstand pushes; the authors intervened manually at the edge of the training area^{Source 6}6 Wu et al. 2022, DayDreamer: World Models for Physical Robot Learning. Philipp Wu, Alejandro Escontrela, Danijar Hafner et al., CoRL 2022.

QT-Opt collected over 580,000 grasp attempts across seven robots in about 800 robot hours^{Source 4}4 Kalashnikov et al. 2018, QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation. Dmitry Kalashnikov, Alex Irpan, Peter Pastor et al., CoRL 2018. The 2016 hand-eye preprint, which trained a supervised grasp-success predictor, reports about 800,000 grasp attempts over two months with 6 to 14 robots at a time^{Source 5}5 Levine et al. 2016, Learning Hand-Eye Coordination for Robotic Grasping with Deep Learning and Large-Scale Data Collection. Sergey Levine, Peter Pastor, Alex Krizhevsky et al., arXiv preprint (v4), 2016.

Offline reinforcement learning

Offline reinforcement learningoffline reinforcement learningOffline reinforcement learning is learning a policy from a fixed dataset of previously collected transitions, without new environment experience during training. The trained policy can then be deployed. Its central challenge is distributional shift, which is distinct from ordinary supervised generalization: policy improvement can query action values outside the data distribution, and errors can propagate through Bellman backups without environment feedback. Adequate data coverage and finite-data errors still matter. learns a policy from a fixed dataset of transitions without further environment interaction during training, then deploys it^{Source 10}10 Levine et al. 2020, Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems. Sergey Levine, Aviral Kumar, George Tucker et al., 2020. It can reuse corpora like those in the datasets module, but a learned policy can choose actions outside the behavior distribution, Bellman targets then use erroneously high Q-values for them, and later backups propagate the errors without fresh feedback^{Source 10}10 Levine et al. 2020, Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems. Sergey Levine, Aviral Kumar, George Tucker et al., 2020. BCQ showed that this extrapolation error leaves standard off-policy algorithms unable to learn from fixed batches^{Source 27}27 Fujimoto et al. 2018, Off-Policy Deep Reinforcement Learning without Exploration. Scott Fujimoto, David Meger, Doina Precup, arXiv preprint (ICML 2019), 2018, and CQL answers it by learning a lower bound on policy value, penalizing Q-values on out-of-distribution actions^{Source 11}11 Kumar et al. 2020, Conservative Q-Learning for Offline Reinforcement Learning. Aviral Kumar, Aurick Zhou, George Tucker et al., NeurIPS 2020.

Does any of this beat behavior cloning on robot data?

Mandlekar and colleagues found batch RL weak on human demonstrations: neither BCQ nor CQL did well on human datasets, while history-dependent BC-RNN was a strong baseline^{Source 13}13 Mandlekar et al. 2021, What Matters in Learning from Offline Human Demonstrations for Robot Manipulation. Ajay Mandlekar, Danfei Xu, Josiah Wong et al., CoRL 2021. Kumar, Hong, Singh and Levine give sufficient conditions under which offline RL beats BC, assuming coverage of the optimal policy; with noisy data that improves coverage, conservative offline RL has a better bound than BC on equal expert data, while on expert data similar performance is a worst-case result. Their drawer-manipulation comparison, in simulation, pits BC on expert successes against CQL on noisy scripted data. Naive CQL can perform no better than BC^{Source 12}12 Kumar et al. 2022, When Should We Prefer Offline Reinforcement Learning Over Behavioral Cloning?. Aviral Kumar, Joey Hong, Anikait Singh et al., ICLR 2022.

For teleoperation data, a supervised behavior-cloning baseline is the simpler starting point, though newer real-robot results favor RL. Generalist policies fine-tuned on RL-generated data beat the same policies trained on human demonstrations by up to 40% in success^{Source 28}28 Xu et al. 2024, RLDG: Robotic Generalist Policy Distillation via Reinforcement Learning. Charles Xu, Qiyang Li, Jianlan Luo et al., 2024, and Physical Intelligence's RECAP pretrains a VLA with offline RL and more than doubles task throughput on some of the hardest tasks^{Source 29}29 Amin et al. 2025, π*0.6: a VLA That Learns From Experience. Ali Amin, Raichelle Aniceto, Ashwin Balakrishna et al., 2025.

Sparse reward, and learning from failure

A robot that has never completed a task has no successful trajectory, so a completion-only reward gives its value function nothing to fit. Shaping is one answer (reward design). Hindsight experience replayhindsight experience replayHindsight experience replay (HER) is an off-policy technique that replays stored transitions with alternative goals and recomputed rewards, alongside original-goal replay. It trains goal-conditioned policies and value functions; relabeling is not restricted to failed episodes or final states. Andrychowicz and colleagues tested it with DDPG on a simulated Fetch arm for pushing, sliding and pick-and-place. Their binary-reward description is qualified by a pushing contact-penetration reward penalty and by starting half of pick-and-place training episodes from a grasped state. targets sparse binary rewards directly, storing transitions under their original goal and alternative goals and recomputing the reward for each, for example with goals achieved later in the same episode^{Source 14}14 Andrychowicz et al. 2017, Hindsight Experience Replay. Marcin Andrychowicz, Filip Wolski, Alex Ray et al., NeurIPS 2017.

Andrychowicz and colleagues trained DDPG with HER on a simulated seven-DoF Fetch arm for pushing, sliding and pick-and-place, where DDPG alone failed. Pushing added a contact-penetration penalty, and pick-and-place started half its training episodes with the box already grasped. A simulation-trained pick-and-place policy transferred to a physical Fetch without real-robot fine-tuning, and HER needs an achieved-goal mapping and a reward recomputable for any replacement goal^{Source 14}14 Andrychowicz et al. 2017, Hindsight Experience Replay. Marcin Andrychowicz, Filip Wolski, Alex Ray et al., NeurIPS 2017.

The reset problem

A simulated episode restarts from a fresh state automatically, at only computational cost^{Source 15}15 Sharma et al. 2022, Autonomous Reinforcement Learning: Formalism and Benchmarking. Archit Sharma, Kelvin Xu, Nikhil Sardana et al., ICLR 2022; a physical one ends wherever the robot is, and a person or a learned behavior has to put it back^{Source 30}30 Eysenbach et al. 2017, Leave no Trace: Learning to Reset for Safe and Autonomous Reinforcement Learning. Benjamin Eysenbach, Shixiang Gu, Julian Ibarz et al., 2017.

Zhu and colleagues name learning from raw sensory inputs, assigning rewards without hand-designed instrumentation, and learning without human episode resets as the requirements for real-world RL; their D'Claw hand trained unattended, but evaluation included manual resets for bead manipulation^{Source 17}17 Zhu et al. 2020, The Ingredients of Real-World Robotic Reinforcement Learning. Henry Zhu, Justin Yu, Abhishek Gupta et al., ICLR 2020. Leave No Trace learns a reset policy alongside the forward policy and aborts before non-reversible states^{Source 30}30 Eysenbach et al. 2017, Leave no Trace: Learning to Reset for Safe and Autonomous Reinforcement Learning. Benjamin Eysenbach, Shixiang Gu, Julian Ibarz et al., 2017. Sharma and colleagues' EARL benchmark sums gaps to optimal deployment return over training and treats interventions as constrained costs, though its experiments still schedule resets during training^{Source 15}15 Sharma et al. 2022, Autonomous Reinforcement Learning: Formalism and Benchmarking. Archit Sharma, Kelvin Xu, Nikhil Sardana et al., ICLR 2022.

Reset-free learningreset-free learningReset-free learning is training on real hardware without a human resetting the scene between attempts. Sharma and colleagues separate deployed-policy regret over training from continuing average reward. Their base setting has no episodic resets; a non-ergodic extension adds a bounded, cost-decremented intervention budget. EARL’s experiments instead schedule occasional resets, reporting learning curves and final deployed-policy performance. Gupta and colleagues chain task-specific policies through a designer-provided task graph, so learned re-centering, pickup and flip-up can restore in-hand reorientation conditions after a drop. Their reported D’Hand/Sawyer training, which also covered pipe insertion, still used supplied rewards and task graphs, motion capture for in-hand manipulation and scripted pickup-arm motion. replaces manual resets with learned recovery behaviors. Gupta and colleagues' MTRF sequences task-specific policies with a designer-provided task graph, and a D'Hand on a Sawyer arm learned valve reorientation and pipe insertion without manual episode resets. The setup still used supplied rewards, motion capture for the in-hand object, scripted arm motion for pickup and a user who positioned the objects, pipe insertion used frozen fingers, and the method assumes that successful upstream tasks leave valid starting states for downstream ones^{Source 16}16 Gupta et al. 2021, Reset-Free Reinforcement Learning via Multi-Task Learning: Learning Dexterous Manipulation Behaviors without Human Intervention. Abhishek Gupta, Justin Yu, Tony Z. Zhao et al., ICRA 2021. Without such a task graph a person handles the reset, a cost the human-in-the-loop line pays deliberately^{Source 18}18 Luo et al. 2024, Precise and Dexterous Robotic Manipulation via Human-in-the-Loop Reinforcement Learning. Jianlan Luo, Charles Xu, Jeffrey Wu et al., 2024.

Where this goes from here

Parallel simulation explains the sampler behind on-policy training, why RL won locomotion which problems simulation represents faithfully, and sim-to-real transfer deployment. RL fine-tuning covers RL on pretrained vision-language-action policies: pi_RL applies online PPO to flow-based pi0 and pi0.5 policies, replacing flow-matching regression with a policy-gradient update^{Source 19}19 Chen et al. 2026, π_RL: Online RL Fine-tuning for Flow-based Vision-Language-Action Models. Kang Chen, Zhihao Liu, Tonghe Zhang et al., 2026, and PLD fine-tunes a generalist on hybrid recovery data from residual specialists^{Source 20}20 Xiao et al. 2025, Self-Improving Vision-Language-Action Models with Data Generation via Residual RL. Wenli Xiao, Haotian Lin, Andy Peng et al., 2025.

HIL-SERL reports 100% success in 100 evaluation trials per task, except IKEA whole assembly at 10 of 10. Nearly all tasks took one to two and a half hours of real-world training and timing-belt assembly six, counting scripted motion, rollouts and intended stops^{Source 18}18 Luo et al. 2024, Precise and Dexterous Robotic Manipulation via Human-in-the-Loop Reinforcement Learning. Jianlan Luo, Charles Xu, Jeffrey Wu et al., 2024. DROID^{Source 21}21 Khazatsky et al. 2024, DROID: A Large-Scale In-The-Wild Robot Manipulation Dataset. Alexander Khazatsky, Karl Pertsch, Suraj Nair, 2024 and Open X-Embodiment^{Source 22}22 Open X-Embodiment Collaboration et al. 2023, Open X-Embodiment: Robotic Learning Datasets and RT-X Models. Open X-Embodiment Collaboration, Abby O'Neill, Abdul Rehman et al., 2023 supply large real-robot demonstration corpora, but both papers evaluate imitation policies, so offline RL on them needs reward labels the datasets do not define.