Offline Reinforcement Learning for Robotics

Offline reinforcement learning is training a robot policy from a fixed dataset of past experience, without new transitions. Levine and colleagues name its central difficulty: a learned value function can overrate actions the data never tested, and no live environment corrects it^{Source 1}1 Levine et al. 2020, Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems. Sergey Levine, Aviral Kumar, George Tucker et al., 2020.

Paper figure: Viktor Makoviychuk et al. / arXiv. Licence: CC BY 4.0.

The fixed-data bargain

Offline RLoffline reinforcement learningOffline reinforcement learning is learning a policy from a fixed dataset of previously collected transitions, without new environment experience during training. The trained policy can then be deployed. Its central challenge is distributional shift, which is distinct from ordinary supervised generalization: policy improvement can query action values outside the data distribution, and errors can propagate through Bellman backups without environment feedback. Adequate data coverage and finite-data errors still matter. learns from a static set of transitions (observation, action, reward, next observation, termination) produced by one or more behavior policies^{Source 1}1 Levine et al. 2020, Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems. Sergey Levine, Aviral Kumar, George Tucker et al., 2020. Value learning bootstraps, so error from actions outside the data accumulates through the Bellman backup^{Source 8}8 Kumar et al. 2019, Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reduction. Aviral Kumar, Justin Fu, George Tucker et al., arXiv preprint (NeurIPS 2019). For robots, support also spans controller mode, action magnitude, contact regime and task stage, and Real-ORL, built on over 6,500 trajectories from 800 robot hours, found that offline RL and imitation prefer different action spaces^{Source 9}9 Zhou et al. 2022, Real World Offline Reinforcement Learning with Realistic Data Source. Gaoyue Zhou, Liyiming Ke, Siddhartha Srinivasa et al., 2022.

Audit the dataset before choosing a method

Behavior coverage

Map each episode's source (expert, novice, scripted controller, older policy or exploration) and its actions by task stage. D4RL builds its datasets from hand-designed controllers, human demonstrators, multitask data and policy mixtures^{Source 10}10 Fu et al. 2020, D4RL: Datasets for Deep Data-Driven Reinforcement Learning. Justin Fu, Aviral Kumar, Ofir Nachum et al., 2020.

Outcome coverage

The reward must separate successes, partial progress and failures, though a completion label can suffice: COG chained four robot skills under a +1/0 reward, from prior data that never solved the new task^{Source 11}11 Singh et al. 2020, COG: Connecting New Skills to Past Experience with Offline Reinforcement Learning. Avi Singh, Albert Yu, Jonathan Yang et al., arXiv preprint (CoRL 2020).

Transition integrity

Check timestamps, episode boundaries, terminations and action units. Mishandled time limits cause state aliasing and invalidate experience replay^{Source 12}12 Pardo et al. 2017, Time Limits in Reinforcement Learning. Fabio Pardo, Arash Tavakoli, Vitaly Levdik et al., arXiv preprint (ICML 2018), 2017.

Deployment overlap

Judge support in the policy's input representation and in physical operating variables, where a camera or control-rate change can outweigh the task label. PTR learned new tasks in a new domain on a real WidowX from as few as 10 demonstrations after offline RL pretraining^{Source 13}13 Kumar et al. 2022, Pre-Training for Robots: Offline RL Enables Learning New Tasks from a Handful of Trials. Aviral Kumar, Anikait Singh, Frederik Ebert et al., 2022. The datasets guide and the robot-learning stack cover selection and provenance.

Four ways to remain close to evidence

Offline-RL methods differ in how they keep improvement away from unsupported values, and Park and colleagues find the policy-extraction step often matters more than the value objective^{Source 14}14 Park et al. 2024, Is Value Learning Really the Main Bottleneck in Offline RL?. Seohong Park, Kevin Frans, Sergey Levine et al., arXiv preprint (NeurIPS 2024).

Conservative value estimation

Conservative Q-Learning penalizes high values on actions outside the dataset while preserving observed ones^{Source 2}2 Kumar et al. 2020, Conservative Q-Learning for Offline Reinforcement Learning. Aviral Kumar, Aurick Zhou, George Tucker et al., NeurIPS 2020. RankQ's authors argue that such pessimism anchors policies to suboptimal data, and their action ranking raised a VLA's real cube-stacking success from 43.1% to 88.9%^{Source 15}15 Choi et al. 2026, RankQ: Offline-to-Online Reinforcement Learning via Self-Supervised Action Ranking. Andrew Choi, Wei Xu, 2026.

Avoiding unseen-action queries

Implicit Q-Learning fits a state value by expectile regression over dataset actions and extracts a policy by advantage weighting, so value fitting never queries a new action^{Source 3}3 Kostrikov et al. 2022, Offline Reinforcement Learning with Implicit Q-Learning. Ilya Kostrikov, Ashvin Nair, Sergey Levine, ICLR 2022.

Regularizing the policy toward behavior

TD3+BC adds a cloning term and state normalization to an off-policy actor-critic and matches more elaborate algorithms^{Source 4}4 Fujimoto et al. 2021, A Minimalist Approach to Offline Reinforcement Learning. Scott Fujimoto, Shixiang Shane Gu, NeurIPS 2021.

Discretizing and scaling the value function

QT-Opt trained a Q-function on over 580,000 real grasp attempts and reached 96% success on unseen objects^{Source 16}16 Kalashnikov et al. 2018, QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation. Dmitry Kalashnikov, Alex Irpan, Peter Pastor et al., CoRL 2018. With datasets up to 1000 times the usual size, many offline RL algorithms still saturate well short of maximum performance, and long horizons form a fundamental barrier^{Source 17}17 Park et al. 2025, Horizon Reduction Makes RL Scalable. Seohong Park, Kevin Frans, Deepinder Mann et al., arXiv preprint (NeurIPS 2025).

When behavior cloning is the better tool

Behavior cloning is the default for clean expert demonstrations: it fits the action labels and has no bootstrapped critic. Robomimic found recurrent cloning a strong baseline on human demonstrations, where batch-RL methods struggled^{Source 6}6 Mandlekar et al. 2021, What Matters in Learning from Offline Human Demonstrations for Robot Manipulation. Ajay Mandlekar, Danfei Xu, Josiah Wong et al., CoRL 2021, and Kumar and colleagues favor offline RL for sparse rewards, suboptimal or noisy data, and long horizons with composable trajectories^{Source 5}5 Kumar et al. 2022, When Should We Prefer Offline Reinforcement Learning Over Behavioral Cloning?. Aviral Kumar, Joey Hong, Anikait Singh et al., ICLR 2022.

Neither method repairs missing support. Over 240 hours of demonstrations and over 3,000 rollouts, Robo-ValueRL ties success to value reliability, and reliable values let offline RL scale better than quality-agnostic cloning^{Source 18}18 Xia et al. 2026, Robo-ValueRL: Reliable Value Estimation for Offline-to-Online Reinforcement Learning. Wenke Xia, Pei Ren, Wenbo Yu et al., 2026. The behavior-cloning guide covers the baseline.

A six-step evaluation protocol

Do not choose checkpoints by the Q-values they maximize. Paine and colleagues found offline RL algorithms not robust to hyperparameters, yet able to rank policies reliably once the algorithm and Q-estimation method are controlled^{Source 19}19 Paine et al. 2020, Hyperparameter Selection for Offline Reinforcement Learning. Tom Le Paine, Cosmin Paduraru, Andrea Michi et al., 2020.

1. Lock the data split by deployment factor

Hold out environments, objects, operators or periods to match the intended generalization; random transition splits leak adjacent frames. Kumar and colleagues adapt cross-validation to tune offline RL without online evaluation^{Source 20}20 Kumar et al. 2021, A Workflow for Offline Model-Free Robotic Reinforcement Learning. Aviral Kumar, Anikait Singh, Stephen Tian et al., arXiv preprint (CoRL 2021).

2. Establish three baselines

Use cloning, a recurrent cloner when history matters, and the simplest regularized offline-RL method. RvS's conditioned two-layer MLP competes with TD and transformer methods but is weak on random data^{Source 21}21 Emmons et al. 2021, RvS: What is Essential for Offline RL via Supervised Learning?. Scott Emmons, Benjamin Eysenbach, Ilya Kostrikov et al., 2021.

3. Report support-aware diagnostics

Report action distance from the data, critic disagreement, saturation rate and per-source performance; EDAC, for one, penalizes out-of-distribution actions through Q-ensemble uncertainty^{Source 22}22 An et al. 2021, Uncertainty-Based Offline Reinforcement Learning with Diversified Q-Ensemble. Gaon An, Seungyong Moon, Jang-Hyun Kim et al., arXiv preprint (NeurIPS 2021).

4. Use simulation only where it is predictive

Simulation ranks close checkpoints only where its link to hardware is measured, as SIMPLER did with paired real evaluations^{Source 23}23 Li et al. 2024, Evaluating Real-World Robot Manipulation Policies in Simulation. Xuanlin Li, Kyle Hsu, Jiayuan Gu et al., 2024. SCORE limits simulated RL to the support of a real-data policy, raising eight real dexterous tasks from 37.8% to 89.9% average success, against 59.5% for the best baseline^{Source 24}24 Yu et al. 2026, Support-Constrained RL Enables Real-World Policy Improvement without Real-World Experience. Raymond Yu, William Huey, Mustafa Mukadam et al., 2026.

Paper figure: Xuanlin Li et al. / arXiv. Licence: CC BY 4.0.

5. Run blinded physical trials

Fix the protocol before policy identities are known, randomize conditions, and count interventions, collisions and resets, as TRI did^{Source 25}25 TRI LBM Team et al. 2025, A Careful Examination of Large Behavior Models for Multitask Dexterous Manipulation. TRI LBM Team, Jose Barreiros, Andrew Beaulieu et al., 2025.

6. Keep a fixed canary set

Run one repeatable physical suite for every candidate and keep the episode IDs; the TriFinger benchmark offers a reproducible real-hardware setup^{Source 26}26 Gürtler et al. 2023, Benchmarking Offline Reinforcement Learning on Real-Robot Hardware. Nico Gürtler, Sebastian Blaes, Pavel Kolev et al., ICLR 2023.

From a fixed dataset to controlled online improvement

RLPD mixes offline data into off-policy online learning without separate conservative pretraining^{Source 7}7 Ball et al. 2023, Efficient Online Reinforcement Learning with Offline Data. Philip J. Ball, Laura Smith, Ilya Kostrikov et al., ICML 2023. Conservative critics fine-tune poorly, so Cal-QL calibrates CQL with a one-line change and leads on 9 of 11 benchmarks^{Source 27}27 Nakamoto et al. 2023, Cal-QL: Calibrated Offline RL Pre-Training for Efficient Online Fine-Tuning. Mitsuhiko Nakamoto, Yuexiang Zhai, Anikait Singh et al., arXiv preprint (NeurIPS 2023). Retaining offline data in replay is a safe default, though WSRL shows it is unnecessary with a well-designed online method^{Source 28}28 Zhou et al. 2024, Efficient Online Reinforcement Learning Fine-Tuning Need Not Retain Offline Data. Zhiyuan Zhou, Andy Peng, Qiyang Li et al., arXiv preprint (ICLR 2025), 2024.

Ishir Garg, Benjamin Eysenbach, Sergey Levine and colleagues at UC Berkeley and Princeton (October 2026) use prior data to decide where to explore: they estimate empowerment, a measure of how many future states an agent can reach from a state, from unlabeled offline trajectories without any environment interaction^{Source 31}31 Garg et al. 2026, Measuring Empowerment from Offline Data for Intrinsically Motivated Exploration. Ishir Garg, Vivek Myers, Qiyang Li et al., 2026. Used as an exploration bonus for online goal-conditioned contrastive RL, it sped up learning and raised success on five of the OGBench task and dataset pairs and matched the baseline on the rest, over 5 seeds; every test ran in simulation^{Source 31}31 Garg et al. 2026, Measuring Empowerment from Offline Data for Intrinsically Motivated Exploration. Ishir Garg, Vivek Myers, Qiyang Li et al., 2026.

Keep action bounds and command validation outside the learned policy, define interventions and resets, and compare each promotion with the frozen offline checkpoint. PA-RL fine-tuned the 7B OpenVLA with Cal-QL from 40% to 70% real-world success in 40 minutes^{Source 29}29 Mark et al. 2024, Policy Agnostic RL: Offline RL and Online RL Fine-Tuning of Any Class and Backbone. Max Sobol Mark, Tian Gao, Georgia Gabriela Sampaio et al., 2024. LWD applies offline-to-online RL to a pretrained VLA across a fleet of 16 dual-arm robots and reports 95% average success over eight tasks^{Source 30}30 Wang et al. 2026, Learning While Deploying: Fleet-Scale Reinforcement Learning for Generalist Robot Policies. Yi Wang, Xinchen Li, Pengwei Xie et al., 2026. The RL fine-tuning guide covers this phase.

A compact go/no-go checklist

Run an offline-RL experiment only when actions, rewards, terminations and timing are trustworthy, the data holds behavior that cloning should not imitate uniformly, and deployment lies inside the data's support, since generalization to out-of-support states is often a big barrier^{Source 14}14 Park et al. 2024, Is Value Learning Really the Main Bottleneck in Offline RL?. Seohong Park, Kevin Frans, Sergey Levine et al., arXiv preprint (NeurIPS 2024).