RL for Robotics

Reinforcement learning is one idea with two entirely different economies. The idea is a policy improved from its own experience of acting in an environment, formalized as a Markov decision process: states, actions, a transition rule, a reward Sutton 2018↓Reinforcement Learning: An IntroductionRichard S. Sutton, Andrew G. Barto, MIT Press, second edition, 2018. The economies are what a step of that experience costs. In a GPU simulator, a step costs under ten microseconds of wall clock, amortized across the four thousand robots taking one at once Rudin 2021↓Learning to Walk in Minutes Using Massively Parallel Deep Reinforcement LearningNikita Rudin, David Hoeller, Philipp Reist et al., CoRL 2021. On a real robot, a step is a real moment of a real machine's life, and the machine has to be reset afterwards.

Most of what reads as an algorithmic preference in robot learning is that cost difference showing up somewhere else.

What a budget buys, and what it forbids

Set the environment-step budget below, then switch from simulation to a single robot without changing the budget. The instrument holds collection rates constant to illustrate a cost difference. Its family bands are editorial toy categories, not rules about which algorithms a robot can use.

The toy isolates collection cost. It does not measure task performance, predict a new robot’s collection rate or establish which learning algorithm will work.

Two ways to spend a transition

The MDP formulation admits two families of method, and the useful way to name the difference is not the taxonomy label but the property the label stands for: whether a transition can be used more than once.

An on-policyon-policyA learning method whose updates are only valid for data collected by the policy currently being improved, so each batch of experience is used once and then discarded. PPO is the robotics default of this kind: it alternates between sampling interaction data and optimizing a surrogate objective, and the surrogate is what allows several minibatch epochs per batch rather than the single gradient step plain policy gradient takes. The property is a cost when samples are expensive and nearly free when a GPU simulator produces them by the hundred million. method's update is only valid for data the current policy generated, so a batch is collected, consumed, and thrown away. PPO is the canonical instance, alternating between sampling interaction data and optimizing a surrogate objective, where the surrogate is precisely what buys several minibatch epochs from one batch instead of the single gradient step plain policy gradient allows Schulman 2017↓Proximal Policy Optimization AlgorithmsJohn Schulman, Filip Wolski, Prafulla Dhariwal et al., 2017. An off-policyoff-policyA learning method that can improve one policy from data generated by another, which means old experience stays usable after the policy has moved on. Soft actor-critic is the robotics reference point: Haarnoja and colleagues combined off-policy updates with a maximum-entropy objective specifically to attack the sample complexity that keeps model-free deep RL off real hardware, and reported substantially better sample efficiency than on-policy methods on the same tasks. The distinction is about data reuse rather than taxonomy, and on a real robot it is the difference between a feasible experiment and an impossible one. method can improve one policy from data another produced, which means experience survives the policy that made it and accumulates in a replay bufferreplay bufferThe store of past transitions an off-policy learner samples its gradient batches from, which is the mechanism that makes data reuse concrete. Its contents need not come from the current policy or even from the learner at all: RLPD trains from a buffer holding prior offline data alongside fresh online experience, and HIL-SERL seeds one with human demonstrations before any autonomous collection starts. The buffer is also where the distinction between online and offline learning collapses to a question of whether anything new is still being added.. The mechanism underneath is Q-learningQ-learningLearning the value of taking an action in a state, rather than learning a policy directly: the agent fits a function that predicts the return available from a state-action pair, and acts by picking the action its own estimate rates highest. Sutton and Barto set it out as the archetypal off-policy temporal-difference control method, where the update uses the best available next action instead of the action the behaviour policy actually took, which is exactly what lets the method learn from experience it did not generate. Every off-policy and offline algorithm the robotics literature uses is a descendant of that update, with machinery added to keep the value estimate from running away on data it cannot verify.: fit the value of a state-action pair, and bootstrap from the best available next action rather than the one that was taken Sutton 2018↓Reinforcement Learning: An IntroductionRichard S. Sutton, Andrew G. Barto, MIT Press, second edition, 2018.

The continuous-control lineage of that idea is short and worth knowing by name. DDPG adapted deterministic policy gradients to deep function approximators and learned pixel-to-torque policies off-policy Lillicrap 2016↓Continuous Control with Deep Reinforcement LearningTimothy P. Lillicrap, Jonathan J. Hunt, Alexander Pritzel et al., ICLR 2016. TD3 identified overestimation bias in DDPG and reduced its effects with clipped double critics, delayed policy updates and target policy smoothing Fujimoto 2018↓Addressing Function Approximation Error in Actor-Critic MethodsScott Fujimoto, Herke van Hoof, David Meger, ICML 2018. Soft actor-critic added a maximum-entropy objective, and its authors were explicit that the target was sample complexity, one of the two challenges they named as keeping model-free deep RL out of complex real-world domains Haarnoja 2018↓Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic ActorTuomas Haarnoja, Aurick Zhou, Pieter Abbeel et al., ICML 2018.

Why robotics defaulted to PPO anyway

On-policy learning wastes data by construction, so its dominance in simulation-trained robotics looks like a mistake until you price it. When the sampler is a GPU running thousands of environments, environment steps are close to free and the scarce resource is the researcher's patience with a training run that diverges. The SAC paper says that on-policy training "tends to improve stability but results in poor sample complexity"; it also calls the combination of off-policy learning and neural-network function approximation "a major challenge for stability and convergence" Haarnoja 2018↓Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic ActorTuomas Haarnoja, Aurick Zhou, Pieter Abbeel et al., ICML 2018. Rudin and colleagues trained ANYmal with 4,096 parallel robots at 98,304 steps per policy update, 1,500 updates in under twenty minutes on one workstation GPU Rudin 2021↓Learning to Walk in Minutes Using Massively Parallel Deep Reinforcement LearningNikita Rudin, David Hoeller, Philipp Reist et al., CoRL 2021. Discarding every one of those 147 million transitions after a single update is not extravagance, it is a rounding error.

The reported collection budgets differ sharply. Rudin’s 98,304 samples per update over 1,500 updates in under 1,200 seconds imply an end-to-end average greater than 122,880 environment steps per second on an i9-11900k CPU and RTX A6000 GPU Rudin 2021↓Learning to Walk in Minutes Using Massively Parallel Deep Reinforcement LearningNikita Rudin, David Hoeller, Philipp Reist et al., CoRL 2021. Minitaur’s 160,000 control steps over about two hours imply approximately 22 control steps per second averaged over the whole training process, not its 50 Hz rollout rate Haarnoja 2019↓Learning to Walk via Deep Reinforcement LearningTuomas Haarnoja, Sehoon Ha, Aurick Zhou et al., RSS 2019. The parallel-simulation module is about the machinery that makes the first regime possible; the rest of this article is about what you do when you are in the second.

What sample efficiency looks like in step counts

The papers that trained on hardware published their budgets, and the numbers are small enough to be surprising.

Sample efficiencysample efficiencyHow much environment interaction a method needs to reach a specified performance level, counted separately from wall-clock time and gradient updates. Minitaur walking used 160,000 control steps over about two hours of the whole training process. A constant-rate budget illustration can translate steps into modelled time, but that translation does not establish algorithm eligibility or predict another robot task. is the axis these results turn on. Haarnoja and colleagues learned Minitaur locomotion from 160,000 control steps, roughly 400 rollouts, in about two hours of real-world time, and that figure is end to end: it contains the resets and the operator, which is why it sits far below the control rate the rollouts ran at Haarnoja 2019↓Learning to Walk via Deep Reinforcement LearningTuomas Haarnoja, Sehoon Ha, Aurick Zhou et al., RSS 2019. DayDreamer reports one Unitree A1 training run in which the robot learned to roll over, stand and walk in one hour without a simulator. After that hour, ten minutes of additional online learning under pushes produced resistance to light pushes or recovery after harder pushes. The authors manually intervened at the edge of the training area without changing the robot’s joint configuration or orientation, so “without resets” does not mean intervention-free operation Wu 2022↓DayDreamer: World Models for Physical Robot LearningPhilipp Wu, Alejandro Escontrela, Danijar Hafner et al., CoRL 2022.

Scale changes the shape of the cost rather than removing it. QT-Opt collected over 580,000 grasp attempts across seven robots in about 800 robot hours Kalashnikov 2018↓QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic ManipulationDmitry Kalashnikov, Alex Irpan, Peter Pastor et al., CoRL 2018; Section 5.2 of the 2016 hand-eye coordination preprint reports about 800,000 grasp attempts over two months using between 6 and 14 robots at a time. Its abstract says “over 800,000,” while its introduction describes “several months”; these are the source’s own differing descriptions, not an exact campaign count or duration Levine 2016↓Learning Hand-Eye Coordination for Robotic Grasping with Deep Learning and Large-Scale Data CollectionSergey Levine, Peter Pastor, Alex Krizhevsky et al., arXiv preprint (v4), 2016. The two papers describe separate collection campaigns, not interchangeable step-rate measurements.

Offline reinforcement learning

If collecting new data is too costly, Offline reinforcement learningoffline reinforcement learningLearning a policy from a fixed dataset of previously collected transitions, without collecting additional environment experience during training. The trained policy can then be deployed. A central challenge is distributional shift: policy improvement can query action values outside the data distribution, and erroneous estimates can propagate through Bellman backups without fresh environment feedback. This is distinct from ordinary supervised generalization; adequate data coverage and finite-data errors still matter. learns a policy from a fixed dataset of transitions without additional environment interaction during training. The dataset is not updated during that phase; the trained policy is then deployed Levine 2020↓Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open ProblemsSergey Levine, Aviral Kumar, George Tucker et al., 2020. This makes offline learning a way to reuse the corpora catalogued in the datasets module.

Levine, Kumar, Tucker and Fu identify distributional shift as a central challenge in offline RL. It is not just ordinary supervised-learning generalization: improving on the behavior in a fixed dataset requires counterfactual queries, and a learned policy may choose actions outside the behavior distribution. In value-based methods, Bellman targets can then use erroneously high Q-values for those actions; policy improvement favors the overestimates, and later backups can propagate the errors without fresh environment feedback Levine 2020↓Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open ProblemsSergey Levine, Aviral Kumar, George Tucker et al., 2020. The tutorial assumes enough coverage of high-reward transitions for learning to be feasible; finite-data and function-approximation errors can remain even when out-of-distribution queries are controlled.

Conservative value estimation pushes the estimate down where the data is thin. CQL learns a lower bound on the policy's value by penalizing Q-values on out-of-distribution actions and maximizing them under the data distribution, which is the term that tightens the bound Kumar 2020↓Conservative Q-Learning for Offline Reinforcement LearningAviral Kumar, Aurick Zhou, George Tucker et al., NeurIPS 2020.

Implicit and expectile approaches avoid the query entirely. IQL never evaluates an unseen action: it fits an expectile of the value distribution over the actions the dataset does contain, treating the state value function as an implicit maximum, and extracts the policy by advantage-weighted regression Kostrikov 2022↓Offline Reinforcement Learning with Implicit Q-LearningIlya Kostrikov, Ashvin Nair, Sergey Levine, ICLR 2022.

Behavior-regularized approaches constrain the policy rather than the critic. TD3+BC adds a behavior-cloning term to a standard off-policy actor-critic and normalizes the states, and its authors' point is deliberately deflationary: that minimal change matches the performance of far more elaborate offline methods Fujimoto 2021↓A Minimalist Approach to Offline Reinforcement LearningScott Fujimoto, Shixiang Shane Gu, NeurIPS 2021.

Q-Transformer combines scale with regularization. It discretizes each action dimension and uses a Transformer to represent Q-values conditioned on the observation history and earlier dimensions of the current action, backing them up autoregressively. Its offline objective also uses conservative regularization and augments temporal-difference learning with Monte Carlo and n-step returns. The real-robot experiments combine successful human demonstrations with failed autonomous trials for language-conditioned manipulation, rather than establishing that any mixed-quality dataset will work Chebotar 2023↓Q-Transformer: Scalable Offline Reinforcement Learning via Autoregressive Q-FunctionsYevgen Chebotar, Quan Vuong, Alex Irpan et al., CoRL 2023. The paper studies sparse success/failure rewards and leaves online fine-tuning for future work.

Does any of this beat behavior cloning on robot data?

This is where the field genuinely disagrees, and the disagreement is worth stating rather than resolving.

Mandlekar and colleagues ran the head-to-head study on human demonstration data and reported that batch RL methods perform poorly there. BCQ in particular performed strongly on agent-generated data of mixed quality, neither BCQ nor CQL performed particularly well on the human-generated datasets, and a recurrent behavior-cloning baseline was consistently strong Mandlekar 2021↓What Matters in Learning from Offline Human Demonstrations for Robot ManipulationAjay Mandlekar, Danfei Xu, Josiah Wong et al., CoRL 2021. Their proposed explanation is that human data is multimodal and non-Markovian in ways these algorithms were not built for.

Kumar, Hong, Singh and Levine ask when offline RL can outperform BC, including when expert demonstrations are available. Their analysis gives sufficient conditions for particular algorithms, not a rule that sparse rewards or suboptimal data alone make RL superior. The theoretical setting assumes coverage of the optimal policy and bounds each trajectory's return independently of the horizon. For their near-expert critical-state result, the simplified proof additionally assumes coverage of every state-action pair and state marginals matching the optimal policy. They test tabular algorithms on diagnostic gridworlds and compare BC with CQL on simulated manipulation and navigation tasks and Atari. Offline tuning matters: naive CQL can perform no better than BC, and even the tuned results do not establish an across-the-board winner Kumar 2022↓When Should We Prefer Offline Reinforcement Learning Over Behavioral Cloning?Aviral Kumar, Joey Hong, Anikait Singh et al., ICLR 2022.

Under the paper's additional noisy-data coverage condition, conservative offline RL has a more favorable horizon-dependent suboptimality bound than BC trained on an equal amount of expert data. The noise helps by improving coverage, not because arbitrary bad demonstrations are useful. Their simulated drawer-manipulation comparison uses equally sized datasets of expert successes for BC and noisy scripted trajectories for CQL; it is not a result about deployed robots Kumar 2022↓When Should We Prefer Offline Reinforcement Learning Over Behavioral Cloning?Aviral Kumar, Joey Hong, Anikait Singh et al., ICLR 2022.

On expert data, their similar-performance statement is a worst-case result without additional environment-structure assumptions. They also identify conditions under which offline RL can improve on that result; Section 6 leaves understanding when BC is preferable as further work Kumar 2022↓When Should We Prefer Offline Reinforcement Learning Over Behavioral Cloning?Aviral Kumar, Joey Hong, Anikait Singh et al., ICLR 2022.

Editorial recommendation: for a teleoperation dataset, start with a supervised behavior-cloning baseline as the simpler starting point, then test a tuned offline RL alternative under the same evaluation protocol. That is this article's advice, not Kumar and colleagues' endorsement of BC or a guarantee that short-horizon expert data favors cloning.

The pragmatic settlement is to stop choosing. RLPD initializes online off-policy training from a buffer already holding offline data, with no pretraining phase and no explicit conservatism, and gets sample-efficient online improvement out of it Ball 2023↓Efficient Online Reinforcement Learning with Offline DataPhilip J. Ball, Laura Smith, Ilya Kostrikov et al., ICML 2023. HIL-SERL builds on RLPD and incorporates human demonstrations and online corrections Luo 2024↓Precise and Dexterous Robotic Manipulation via Human-in-the-Loop Reinforcement LearningJianlan Luo, Charles Xu, Jeffrey Wu et al., 2024.

Sparse reward, and learning from failure

Offline data does not solve exploration. A robot that has never once completed a task has no successful trajectory in its buffer, and a reward that fires only on completion gives a value function nothing to fit. Reward shaping is one answer and it belongs to the reward design module. The other answer changes the question asked of the data.

Andrychowicz 2017↓Hindsight Experience ReplayMarcin Andrychowicz, Filip Wolski, Alex Ray et al., NeurIPS 2017 Hindsight experience replayhindsight experience replayAn off-policy replay technique that adds alternative goals to stored transitions and recomputes their rewards, while retaining original-goal replay. It trains goal-conditioned policies and value functions; relabeling is not restricted to failed episodes or final states. Andrychowicz and colleagues tested HER with DDPG on a simulated Fetch arm for pushing, sliding and pick-and-place. Their binary-reward description is qualified by a pushing contact-penetration reward penalty and by starting half of the pick-and-place training episodes from a grasped state. stores transitions under their original goal and selected alternative goals, recomputing the reward for each alternative. It is not restricted to failed episodes. The paper's simplest strategy uses the goal achieved in the final state; its future strategy instead samples goals achieved later in the same episode.

Andrychowicz and colleagues evaluated DDPG with HER on a seven-DoF Fetch arm simulated in MuJoCo for pushing, sliding and pick-and-place. The future-goal variant learned all three tasks whereas DDPG without HER did not, under their reported evaluation protocol. The main text describes sparse binary goal rewards, but Appendix A adds a squared-contact-penetration reward penalty in pushing. Pick-and-place also started half of its training episodes from a recorded state with the box already grasped. These settings do not support an unqualified claim of learning without any shaping or task-specific assistance. A separate pick-and-place deployment transferred a simulation-trained policy to a physical Fetch without real-robot fine-tuning; it used a separately trained camera estimator and simulation retraining with noisy observations, not on-hardware RL Andrychowicz 2017↓Hindsight Experience ReplayMarcin Andrychowicz, Filip Wolski, Alex Ray et al., NeurIPS 2017.

The original method trains policies and value functions conditioned on both state and goal. Goals may specify only some properties of a state, but the method needs an achieved-goal mapping and a reward that can be recomputed for the replacement goal. Training with multiple goals can help even when only one goal matters at deployment Andrychowicz 2017↓Hindsight Experience ReplayMarcin Andrychowicz, Filip Wolski, Alex Ray et al., NeurIPS 2017.

The reset problem

The difference between simulation RL and real-world RL that no algorithm addresses is janitorial. A simulated episode ends and the next one begins from a fresh initial state, for free. A physical episode ends with the robot somewhere arbitrary, possibly having knocked the workspace apart, and something has to put it back. That something is usually a person, and a person who has to intervene between episodes is the real limit on how long an experiment can run.

Zhu and colleagues identify three capabilities for a real-world RL system: learning from raw sensory inputs, assigning rewards without hand-designed perception systems or instrumentation, and learning continuously without human episode resets. Their experiments investigate the difficulty of combining those requirements. A D’Claw hand learned valve rotation and bead manipulation from RGB images after people supplied goal examples. Training was unattended, but evaluation was performed afterwards and included manual resets for bead manipulation. The authors leave sample complexity, exploration on harder tasks and safe operation among the problems for future work Zhu 2020↓The Ingredients of Real-World Robotic Reinforcement LearningHenry Zhu, Justin Yu, Abhishek Gupta et al., ICLR 2020.

Sharma and colleagues distinguish deployed-policy evaluation from continuing-policy evaluation for non-episodic learning. The former sums gaps to optimal expected discounted deployment return over training, using evaluation rollouts from the initial-state distribution that never enter training. The latter is the limiting average reward from the agent’s actual actions, with no separate deployment phase. In their extension for irreversible, non-ergodic settings, an intervention budget h falls by c(s,a) for each requested or enforced intervention; h - c(s,a) <= 0 sends the environment to an absorbing state and ends training. Interventions are therefore constrained costs, not the evaluation metric Sharma 2022↓Autonomous Reinforcement Learning: Formalism and BenchmarkingArchit Sharma, Kelvin Xu, Nikhil Sardana et al., ICLR 2022.

EARL’s experiments instead use scheduled resets during training and stop after a fixed step budget. They estimate deployed-policy performance with finite-horizon, undiscounted returns and continuing performance with average reward accumulated so far. They show learning curves and also report final deployed-policy returns. The paper notes that the two objectives can favor different behavior and that trading some human supervision for faster learning remains a practical question Sharma 2022↓Autonomous Reinforcement Learning: Formalism and BenchmarkingArchit Sharma, Kelvin Xu, Nikhil Sardana et al., ICLR 2022.

Reset-free learningreset-free learningTraining on real hardware without a human returning the scene to a start state between attempts, which is what stands between a working algorithm and an unattended experiment. Sharma and colleagues distinguish deployed-policy regret over training from continuing average reward. Their base setting has no episodic resets; a separate extension for non-ergodic settings adds a bounded, cost-decremented intervention budget. EARL’s experiments instead schedule occasional resets and report both learning curves and final deployed-policy performance. Gupta and colleagues sequence task-specific policies with a designer-provided task graph, so learned re-centering, pickup and flip-up can restore conditions for in-hand reorientation after a dropped object. Their D’Hand/Sawyer experiments also covered pipe insertion; the reported reset-free training still used supplied rewards and task graphs, motion capture for in-hand manipulation and scripted pickup-arm motion. can replace manual episode resets with learned recovery behaviors. Gupta and colleagues' MTRF learns task-specific policies and uses a designer-provided task graph to sequence them: re-centering, pickup and flip-up can restore the conditions for in-hand reorientation after a dropped object. They evaluated a D’Hand mounted on a Sawyer arm on valve reorientation and pipe insertion, and report learning these tasks without manual episode resets during the training runs Gupta 2021↓Reset-Free Reinforcement Learning via Multi-Task Learning: Learning Dexterous Manipulation Behaviors without Human InterventionAbhishek Gupta, Justin Yu, Tony Z. Zhao et al., ICRA 2021. The setup still used supplied rewards and task graphs; the in-hand experiment tracked the object with motion capture, and pickup used scripted arm motion while learning the hand controls. Pipe insertion used frozen fingers and separate approach and insertion tasks to reduce collisions Gupta 2021↓Reset-Free Reinforcement Learning via Multi-Task Learning: Learning Dexterous Manipulation Behaviors without Human InterventionAbhishek Gupta, Justin Yu, Tony Z. Zhao et al., ICRA 2021. For the in-hand run, the user positioned the objects and started the robot. The method assumes that successful upstream tasks produce valid starting states for downstream ones; it does not guarantee coverage of every downstream starting state. This needs a suitable task family and a supplied task graph Gupta 2021↓Reset-Free Reinforcement Learning via Multi-Task Learning: Learning Dexterous Manipulation Behaviors without Human InterventionAbhishek Gupta, Justin Yu, Tony Z. Zhao et al., ICRA 2021. Absent one, the reset is a person, which is the cost the human-in-the-loop line chose to pay deliberately rather than engineer away Luo 2024↓Precise and Dexterous Robotic Manipulation via Human-in-the-Loop Reinforcement LearningJianlan Luo, Charles Xu, Jeffrey Wu et al., 2024.

Where this goes from here

Everything above is the machinery. The rest of this domain is what the field built with it, and each module picks up one thread from this one.

The parallel simulation module is the sampler that makes on-policy learning affordable, in detail: which simulators, what throughput, and where the remaining bottleneck sits. Why RL won locomotion is the argument about which problems that sampler can faithfully represent, which is the reason the same recipe produced walking robots and not manipulating ones. Sim-to-real transfer covers what happens between a trained simulation policy and a working machine. Reward design takes up the specification problem this article deferred.

On the manipulation side, RL fine-tuning covers reinforcement learning on pretrained vision-language-action policies. The January 2026 v3 preprint pi_RL applies online PPO to supervised-fine-tuned pi0 and pi0.5 flow-based policies; this is a policy-gradient update, not the supervised flow-matching regression loss Chen 2026↓π_RL: Online RL Fine-tuning for Flow-based Vision-Language-Action ModelsKang Chen, Zhihao Liu, Tonghe Zhang et al., 2026. Other approaches include PLD training off-policy residual specialists against a frozen VLA, collecting hybrid recovery data, then fine-tuning the generalist by SFT Xiao 2025↓Self-Improving Vision-Language-Action Models with Data Generation via Residual RLWenli Xiao, Haotian Lin, Andy Peng et al., 2025, and HIL-SERL reporting 100% observed success in 100 evaluation trials per task, except IKEA whole assembly at 10/10. HIL-SERL reports one to two and a half hours of real-world training for nearly all tasks; Table 1 lists six hours for timing-belt assembly. Training time includes scripted motion, policy rollouts, intended stops and onboard computation on one RTX 4090 GPU Luo 2024↓Precise and Dexterous Robotic Manipulation via Human-in-the-Loop Reinforcement LearningJianlan Luo, Charles Xu, Jeffrey Wu et al., 2024. And the offline-RL question is only interesting because the datasets exist: DROID Khazatsky 2024↓DROID: A Large-Scale In-The-Wild Robot Manipulation DatasetAlexander Khazatsky, Karl Pertsch, Suraj Nair, 2024 and Open X-Embodiment Open X-Embodiment Collaboration 2023↓Open X-Embodiment: Robotic Learning Datasets and RT-X ModelsOpen X-Embodiment Collaboration, Abby O'Neill, Abdul Rehman et al., 2023 are the corpora on which the argument about offline RL versus cloning is currently being settled.