RL Fine-Tuning of Policies

RL fine-tuning is the use of reinforcement learning to improve a pretrained robot policy from its own successes and failures. Imitation learningimitation learningImitation learning is learning a policy from expert demonstrations instead of a reward signal, by supervised fitting of recorded state-action pairs. Pomerleau's ALVINN (1988) trained on simulated road images before steering NAVLAB; the recipe still underlies most learned manipulation. Its structural weakness: training states come from the expert while deployment visits the learner's own, the mismatch DAgger was designed to repair. is brittle off the demonstration distribution because errors compound^{Source 1}1 Ross et al. 2011, A Reduction of Imitation Learning and Structured Prediction to No-Regret Online Learning. Stéphane Ross, Geoffrey J. Gordon, J. Andrew Bagnell, AISTATS 2011, and Physical Intelligence finds reliable success much harder than occasional success^{Source 6}6 Amin et al. 2025, π*0.6: a VLA that Learns from Experience. Ali Amin, Raichelle Aniceto, Ashwin Balakrishna et al., 2025.

Paper figure: Jianlan Luo et al. / arXiv. Licence: CC BY 4.0.

Switch to “What it learns”.

The flow-matching log-likelihood problem

PPOPPOPPO (proximal policy optimization) is the on-policy reinforcement learning algorithm the locomotion literature standardized on. Schulman and colleagues introduced it as a policy gradient method alternating between sampling environment data and optimizing a surrogate objective by stochastic gradient ascent. The surrogate allows several minibatch epochs per batch where standard policy gradient takes one step. It keeps most of trust region policy optimization's reliability without second-order machinery, and suits massively parallel simulation, whose thousands of robots supply its enormous on-policy batches. runs several epochs of minibatch updates on a clipped surrogate objective^{Source 18}18 Schulman et al. 2017, Proximal Policy Optimization Algorithms. John Schulman, Filip Wolski, Prafulla Dhariwal et al., 2017, which needs the likelihood of each action taken. Flow matchingflow matchingFlow matching is a generative modeling recipe that learns a time-dependent vector field carrying noise to data, by regressing against the conditional flow instead of estimating a score. In pi0, which brought it to robot control, an action expert trained this way denoises continuous action chunks. Integrating the field as an ODE for a handful of steps is fast enough for high-frequency control and keeps diffusion policies' multimodality. trains by regressing vector fields along fixed conditional probability paths^{Source 19}19 Lipman et al. 2022, Flow Matching for Generative Modeling. Yaron Lipman, Ricky T. Q. Chen, Heli Ben-Hamu et al., 2022, so the executed action's log-likelihood is intractable^{Source 4}4 Chen et al. 2026, π_RL: Online RL Fine-tuning for Flow-based Vision-Language-Action Models. Kang Chen, Zhihao Liu, Tonghe Zhang et al., 2026. Online RL applied directly to large VLAs is also unstable and needs compute beyond most local machines^{Source 20}20 Guo et al. 2025, Improving Vision-Language-Action Model with Online Reinforcement Learning. Yanjiang Guo, Jianke Zhang, Xiaoyu Chen et al., ICRA 2025.

DPPO (Ren et al., 2024) follows DDPO, which posed diffusion denoising as a multi-step decision problem^{Source 21}21 Black et al. 2023, Training Diffusion Models with Reinforcement Learning. Kevin Black, Michael Janner, Yilun Du et al., 2023. DPPO embeds denoising inside the environment MDP: each denoising transition has an analytic Gaussian likelihood, and the environment advances and pays reward only after the final step. Only the last selected denoising steps are fine-tuned, with a frozen copy of the network for earlier ones^{Source 2}2 Ren et al. 2024, Diffusion Policy Policy Optimization. Allen Z. Ren, Justin Lidard, Lars L. Ankile et al., 2024.

DPPO is the strongest of the compared diffusion-based RL methods on state-input Robomimic tasks, though off-policy methods are more sample-efficient on HalfCheetah and a Gaussian policy slightly beats it on the low-randomness Lamp task. A simulation-trained One-leg assembly policy transferred zero-shot to a Franka Emika Panda and succeeded in 16 of 20 trials at 10 Hz^{Source 2}2 Ren et al. 2024, Diffusion Policy Policy Optimization. Allen Z. Ren, Justin Lidard, Lars L. Ankile et al., 2024.

ReinFlow injects learnable noise into a flow policy's path for exact per-step likelihoods, saving 82.63% of wall time against DPPO on legged locomotion^{Source 22}22 Zhang et al. 2025, ReinFlow: Fine-tuning Flow Matching Policy with Online Reinforcement Learning. Tonghe Zhang, Chao Yu, Sichang Su et al., 2025, and FPO instead builds a PPO-clip ratio from the conditional flow-matching loss^{Source 23}23 McAllister et al. 2025, Flow Matching Policy Gradients. David McAllister, Songwei Ge, Brent Yi et al., 2025. QF3 (UC Berkeley and Amazon FAR, October 2026), an off-policy method that adds the critic's action gradient to the flow-matching loss, is reported as the first off-policy flow RL method to train humanoid locomotion from scratch and transfer it zero-shot to hardware, with a 10× wall-clock speedup over the on-policy FPO++^{Source 37}37 Kim et al. 2026, QF3: Fast Flow RL with Filtered Q-Gradients. Chung Min Kim, Brent Yi, David McAllister et al., 2026.

ConRFT (Chen et al., 2025) fine-tunes a consistency-policy action head, a sampler distilled from a diffusion policy^{Source 24}24 Prasad et al. 2024, Consistency Policy: Accelerated Visuomotor Policies via Consistency Distillation. Aaditya Prasad, Kevin Lin, Jimmy Wu et al., 2024, while freezing the visual encoders and transformer backbone of Octo-small on a 7-DoF Franka. Offline Cal-ConRFT pairs a consistency-based behavior cloningbehavior cloningBehavior cloning is supervised learning of a policy from expert demonstrations, with observations as inputs and the expert's actions as labels. The recipe predates deep learning: ALVINN (1988) used camera and laser-range inputs and trained on simulated road images before NAVLAB tests. Scaled up, it still underlies most learned manipulation policies; its known closed-loop failure mode is covariate shift. loss with a Cal-QL critic, from 20 to 30 demonstrations per task, and online HIL-ConRFT adds human takeovers to the demonstration buffer^{Source 3}3 Chen et al. 2025, ConRFT: A Reinforced Fine-tuning Method for VLA Models via Consistency Policy. Yuhui Chen, Shuai Tian, Shugao Liu et al., 2025.

ConRFT reports 96.3% mean success ratesuccess rateSuccess rate is the fraction of attempted episodes in which a policy completes the task, the standard headline metric of robot learning evaluation. A reported rate needs its trial count, time limit and scene distribution. Toyota Research Institute's Large Behavior Model study analyzed 1,800 real-world rollouts and warned of a significant risk of statistical noise from insufficient power. Snyder et al.'s STEP sequential test needs up to 32% fewer trials than the Lai and SAVI baselines. across eight tasks, 20 trials each, after 15 to 90 minutes of listed online training. ConRFT's headline 144% is a rounded relative gain over offline Cal-ConRFT's 39.4% mean, which the SFT baseline also scores. Online learning curves can include successes produced by human takeovers, and the paper warns of reward-classifier hacking and limited generalization^{Source 3}3 Chen et al. 2025, ConRFT: A Reinforced Fine-tuning Method for VLA Models via Consistency Policy. Yuhui Chen, Shuai Tian, Shugao Liu et al., 2025. ConRFT says every compared method used interventions, but PA-RL reports fine-tuning OpenVLA autonomously with Cal-QL, from 40% to 70% in 40 real-world minutes^{Source 25}25 Mark et al. 2024, Policy Agnostic RL: Offline RL and Online RL Fine-Tuning of Any Class and Backbone. Max Sobol Mark, Tian Gao, Georgia Gabriela Sampaio et al., 2024.

pi_RL (Chen et al., 2026) fine-tunes flow-based policies with online PPO. Flow-Noise learns Gaussian noise for denoising transitions jointly with the velocity network and uses the joint likelihood of the denoising path in place of the executed action's marginal likelihood. Flow-SDE uses Gaussian transition likelihoods in a two-layer denoising and environment MDP, with one randomly chosen denoising step stochastic and the rest deterministic^{Source 4}4 Chen et al. 2026, π_RL: Online RL Fine-tuning for Flow-based Vision-Language-Action Models. Kang Chen, Zhihao Liu, Tonghe Zhang et al., 2026.

Main experiments fine-tune π0 and π0.5 in simulation on LIBERO, ManiSkill, MetaWorld and CALVIN, and gains under distribution shift do not consistently reach the five unseen MetaWorld ML45 tasks. A Real2Sim2Real case transfers zero-shot to a Franka Panda, reporting 40% success without a trial count^{Source 4}4 Chen et al. 2026, π_RL: Online RL Fine-tuning for Flow-based Vision-Language-Action Models. Kang Chen, Zhihao Liu, Tonghe Zhang et al., 2026.

Recap: advantage-conditioned policy learning

Recap, the method behind π*0.6, combines demonstrations, expert teleoperation corrections during rollouts and autonomous experience, in rounds of collection and retraining. A language-conditioned distributional value function predicts negative remaining steps for successes and a large negative value for failures. In post-training the advantage uses rewards over the next N = 50 steps plus the later value minus the current value, and a task-dependent threshold turns it into a binary indicator, with human corrections assigned the positive one^{Source 5}5 Amin et al. 2025, π*0.6: a VLA That Learns From Experience. Ali Amin, Raichelle Aniceto, Ashwin Balakrishna et al., 2025.

The text "Advantage: positive" or "Advantage: negative" precedes the actions, so good and bad data both train the policy, and evaluation uses the positive-conditioned policy; indicator dropout permits classifier-free guidance^{Source 5}5 Amin et al. 2025, π*0.6: a VLA That Learns From Experience. Ali Amin, Raichelle Aniceto, Ashwin Balakrishna et al., 2025, which combines jointly trained conditional and unconditional estimates^{Source 26}26 Ho et al. 2022, Classifier-Free Diffusion Guidance. Jonathan Ho, Tim Salimans, 2022. The KL-regularized objective has the exponential solution:

Advantage-weighted regression fits that target with two supervised steps^{Source 27}27 Peng et al. 2019, Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learning. Xue Bin Peng, Aviral Kumar, Grace Zhang et al., 2019. Recap's binary conditioning instead writes the improved policy as

At $\beta_{\mathrm{CFG}}=1$ this reduces to the indicator-conditioned policy. Recap thereby avoids a PPO-style objective: training combines next-token prediction with the flow-matching loss, and Knowledge Insulation stops the action expert's gradient at the backbone interface^{Source 5}5 Amin et al. 2025, π*0.6: a VLA That Learns From Experience. Ali Amin, Raichelle Aniceto, Ashwin Balakrishna et al., 2025 ^{Source 7}7 Physical Intelligence 2025, π0.6 Model Card. Physical Intelligence, 2025.

On-robot experience more than doubles successful completions per hour for double-shot espresso and diverse laundry, against offline RL plus demonstration fine-tuning, on a static bimanual setup with a button-up-shirt test in place of the 11-item laundry set. Failures fall by "about a factor of two" ("more than 2×" elsewhere in the report); the "90%+" summary excludes diverse laundry, and the box chart reports stages separately^{Source 5}5 Amin et al. 2025, π*0.6: a VLA That Learns From Experience. Ali Amin, Raichelle Aniceto, Ashwin Balakrishna et al., 2025. The pi line lists the hardware and time limits.

The companion blog's video caption gives 5:30am to 11:30pm of espresso making, while the report says "13 hours straight"^{Source 6}6 Amin et al. 2025, π*0.6: a VLA that Learns from Experience. Ali Amin, Raichelle Aniceto, Ashwin Balakrishna et al., 2025 ^{Source 5}5 Amin et al. 2025, π*0.6: a VLA That Learns From Experience. Ali Amin, Raichelle Aniceto, Ashwin Balakrishna et al., 2025. The results are Physical Intelligence's own, with human reward labeling, interventions, resets and batch offline updates^{Source 5}5 Amin et al. 2025, π*0.6: a VLA That Learns From Experience. Ali Amin, Raichelle Aniceto, Ashwin Balakrishna et al., 2025. In March 2026 Physical Intelligence added RL tokens: small actor and critic networks train online on a compact token from π0.6, speeding the most precise stage of four tasks by up to 3×^{Source 28}28 Xu et al. 2026, Precise Manipulation with Efficient Online RL. Charles Xu, Jost Tobias Springenberg, Michael Equi et al., Physical Intelligence, 2026.

HIL-SERL: a human in the loop, RL on the robot

HIL-SERL (Luo et al., Science Robotics 2025) follows SERL, which learned PCB assembly, cable routing and object relocation in 25 to 50 minutes per policy^{Source 29}29 Luo et al. 2024, SERL: A Software Suite for Sample-Efficient Robotic Reinforcement Learning. Jianlan Luo, Zheyuan Hu, Charles Xu et al., ICRA 2024, and uses RLPD, off-policy RL with offline data mixed in^{Source 30}30 Ball et al. 2023, Efficient Online Reinforcement Learning with Offline Data. Philip J. Ball, Laura Smith, Ilya Kostrikov et al., ICML 2023. Intervention transitions enter both the demonstration and RL buffers, autonomous ones only the RL buffer, and training typically starts from 20 to 30 demonstrations; Jenga whipping starts from 30 with no real-time takeovers^{Source 8}8 Luo et al. 2024, Precise and Dexterous Robotic Manipulation via Human-in-the-Loop Reinforcement Learning. Jianlan Luo, Charles Xu, Jeffrey Wu et al., 2024.

HIL-SERL completes 100% of observed evaluation trials, 100 per task, except IKEA whole assembly with 10 trials and at most two attempts per sub-policy. Training takes 1 to 2.5 hours for nearly all tasks and 6 for timing-belt assembly, including scripted motion, rollouts, intended stops and computation. Against HG-DAgger with matched episodes and interventions, average success is 49.7% versus 100% and cycle time 9.6 versus 5.4 seconds^{Source 8}8 Luo et al. 2024, Precise and Dexterous Robotic Manipulation via Human-in-the-Loop Reinforcement Learning. Jianlan Luo, Charles Xu, Jeffrey Wu et al., 2024.

Online learning curves can include human-assisted successes, the paper reports DP's dashboard-assembly success as both 18% and 28%, and Luo and colleagues did not test unstructured environments or extensive randomization^{Source 8}8 Luo et al. 2024, Precise and Dexterous Robotic Manipulation via Human-in-the-Loop Reinforcement Learning. Jianlan Luo, Charles Xu, Jeffrey Wu et al., 2024. The RL for Robotics module covers the off-policy context, and parallel simulation RL the simulation route.

Mulligan (Stanford, October 2026) starts each round of supervised collection at the initial states where the policy failed or has not been tried; with value-based action selection, it raised final-round success on three real tasks by 10 to 34 percentage points over HG-DAgger, scored on 2,550 held-out, blinded episodes^{Source 38}38 Ankile et al. 2026, Mulligan: Performance-Guided Data Collection for Efficient On-Robot Learning. Lars Ankile, Perry Dong, Rohan Bhowmik et al., 2026.

Residual RL and distillation back into the generalist

Residual RL adds a learned correction to a fixed controller^{Source 31}31 Johannink et al. 2018, Residual Reinforcement Learning for Robot Control. Tobias Johannink, Shikhar Bahl, Ashvin Nair et al., 2018. PLD (Probe, Learn, Distill; Xiao et al., 2025) freezes a VLA, trains task-specific residual actors with off-policy RL, lets them recover from the states the base policy reaches, and distills the trajectories back into the generalist through SFT. Residual exploration carries no safety guarantee^{Source 9}9 Xiao et al. 2025, Self-Improving Vision-Language-Action Models with Data Generation via Residual RL. Wenli Xiao, Haotian Lin, Andy Peng et al., 2025.

Paper figure: Wenli Xiao et al. / arXiv. Licence: CC BY 4.0.

On LIBERO, averages are 99.2% for the model labelled OpenVLA against 91.8% for its baseline, and 97.2% against 93.4% for π0. On a four-task SimplerEnv suite, Octo-SFT rises from 71.8% to 96.6% and the WidowX carrot task from 43.3% to 93.9%; the printed average delta is +24.9, though the means differ by 24.8^{Source 9}9 Xiao et al. 2025, Self-Improving Vision-Language-Action Models with Data Generation via Residual RL. Wenli Xiao, Haotian Lin, Andy Peng et al., 2025.

On a Franka, PLD records 30/30 randomized trials for cube pick-up and peg insertion, but 28/30 for a blue cube and 20/30 for a red one under changed clutter. On YAM, four GPU insertion and removal stages distill into one BC policy that runs at least an hour continuously with recovery, though per-stage one-shot success stays below 100%^{Source 9}9 Xiao et al. 2025, Self-Improving Vision-Language-Action Models with Data Generation via Residual RL. Wenli Xiao, Haotian Lin, Andy Peng et al., 2025.

RLDG (Xu et al., 2024) uses HIL-SERL specialists to generate data for fine-tuning OpenVLA and Octo, so the generalist never receives RL updates. In FMB single-object assembly, RL supplies insertion data while humans supply grasping and transport, and OpenVLA succeeds in 20/20 trials against 12/20 with human demonstrations^{Source 10}10 Xu et al. 2024, RLDG: Robotic Generalist Policy Distillation via Reinforcement Learning. Charles Xu, Qiyang Li, Jianlan Luo et al., 2024. Relabeling suggests action quality, more than state distribution, drives the benefit; the method needs task rewards, speed-optimized RL data can cause premature object drops, and OpenVLA commands actions at 4 Hz while RL runs at 10 Hz^{Source 10}10 Xu et al. 2024, RLDG: Robotic Generalist Policy Distillation via Reinforcement Learning. Charles Xu, Qiyang Li, Jianlan Luo et al., 2024.

EXPO-FT: value-guided edits on a frontier VLA

EXPO (Perry Dong, Qiyang Li, Dorsa Sadigh and Chelsea Finn; ICLR 2026) trains a large flow or diffusion base policy with a stable imitation objective plus a lightweight Gaussian edit policy that nudges sampled actions toward higher value, avoiding gradients through the denoising chain, and reports up to 2 to 3× better sample efficiency on average^{Source 14}14 Dong et al. 2025, EXPO: Stable Reinforcement Learning with Expressive Policies. Perry Dong, Qiyang Li, Dorsa Sadigh et al., 2025.

EXPO-FT, from Dong, Kuo-Han Hung, Tian Gao, Sadigh and Finn, extends EXPO to VLA action chunks and human teleoperation corrections on π0.5. Tasks the base cannot do zero-shot get imitation fine-tuning to around 40% success or above before online RL, and sparse binary rewards come from rule-based detectors over 95% accurate. It reports 30/30 on every evaluated task within an average of 19.1 minutes of online robot data across eight task variants^{Source 13}13 Dong et al. 2026, EXPO-FT: Sample-Efficient Reinforcement Learning Finetuning for Vision-Language-Action Models. Perry Dong, Kuo-Han Hung, Tian Gao et al., 2026.

The paper calls HIL-SERL highly reliable in its original evaluations; this suite randomizes a much larger initial-state space, and the standard budget was too small for learning to start on Cube Pick and Pool Shot. With extra samples, HIL-SERL reaches 27/30 and 13/30 there^{Source 13}13 Dong et al. 2026, EXPO-FT: Sample-Efficient Reinforcement Learning Finetuning for Vision-Language-Action Models. Perry Dong, Kuo-Han Hung, Tian Gao et al., 2026 ^{Source 8}8 Luo et al. 2024, Precise and Dexterous Robotic Manipulation via Human-in-the-Loop Reinforcement Learning. Jianlan Luo, Charles Xu, Jeffrey Wu et al., 2024.

DSRL keeps the base policy frozen and runs RL over its latent-noise space^{Source 15}15 Wagenmaker et al. 2025, Steering Your Diffusion Policy with Latent Space Reinforcement Learning. Andrew Wagenmaker, Mitsuhiko Nakamoto, Yunchu Zhang et al., 2025. Dong and Finn argue it has a ceiling: "If the skill a robot needs isn't already somewhere in what the base model learned during pretraining, no amount of noise-searching will produce the action ...". DSRL averages 19/30 in their comparison^{Source 17}17 Dong et al. 2026, Towards Universal Post-Training for Robotics. Perry Dong, Chelsea Finn, 2026 ^{Source 13}13 Dong et al. 2026, EXPO-FT: Sample-Efficient Reinforcement Learning Finetuning for Vision-Language-Action Models. Perry Dong, Kuo-Han Hung, Tian Gao et al., 2026.

Dong and Finn's September 2026 blog post says: "RL shows incredible promise in training frontier robotics models, but it is not yet a recipe. It's a craft." EXPO-FT still needs a human to define success, reset the scene and intervene, longer horizons strain the value function, and "Nineteen minutes of robot interaction is not nineteen minutes of post-training," because gradient updates dominate wall-clock time^{Source 17}17 Dong et al. 2026, Towards Universal Post-Training for Robotics. Perry Dong, Chelsea Finn, 2026. Real-Time EXPO-FT lets a lightweight edit policy react to the latest observation while a slow VLA proposes chunks, and on four dynamic real tasks average performance rises from 42% to 97%^{Source 16}16 Dong et al. 2026, Reinforcement Learning for Real-Time Vision-Language-Action Policies. Perry Dong, Kuo-Han Hung, Dorsa Sadigh et al., 2026; real-time execution covers the latency budgets.

The methods side by side

The protocols differ too much for a cross-paper ranking. DPPO, ConRFT, EXPO-FT, HIL-SERL and pi_RL state that code is available. The inspected PLD, DSRL and Recap sources establish no code, weight or license release^{Source 2}2 Ren et al. 2024, Diffusion Policy Policy Optimization. Allen Z. Ren, Justin Lidard, Lars L. Ankile et al., 2024 ^{Source 3}3 Chen et al. 2025, ConRFT: A Reinforced Fine-tuning Method for VLA Models via Consistency Policy. Yuhui Chen, Shuai Tian, Shugao Liu et al., 2025 ^{Source 8}8 Luo et al. 2024, Precise and Dexterous Robotic Manipulation via Human-in-the-Loop Reinforcement Learning. Jianlan Luo, Charles Xu, Jeffrey Wu et al., 2024 ^{Source 4}4 Chen et al. 2026, π_RL: Online RL Fine-tuning for Flow-based Vision-Language-Action Models. Kang Chen, Zhihao Liu, Tonghe Zhang et al., 2026 ^{Source 5}5 Amin et al. 2025, π*0.6: a VLA That Learns From Experience. Ali Amin, Raichelle Aniceto, Ashwin Balakrishna et al., 2025 ^{Source 6}6 Amin et al. 2025, π*0.6: a VLA that Learns from Experience. Ali Amin, Raichelle Aniceto, Ashwin Balakrishna et al., 2025 ^{Source 7}7 Physical Intelligence 2025, π0.6 Model Card. Physical Intelligence, 2025 ^{Source 9}9 Xiao et al. 2025, Self-Improving Vision-Language-Action Models with Data Generation via Residual RL. Wenli Xiao, Haotian Lin, Andy Peng et al., 2025 ^{Source 13}13 Dong et al. 2026, EXPO-FT: Sample-Efficient Reinforcement Learning Finetuning for Vision-Language-Action Models. Perry Dong, Kuo-Han Hung, Tian Gao et al., 2026 ^{Source 15}15 Wagenmaker et al. 2025, Steering Your Diffusion Policy with Latent Space Reinforcement Learning. Andrew Wagenmaker, Mitsuhiko Nakamoto, Yunchu Zhang et al., 2025.

Real-world and fleet-scale results

RL-100, published in Science Robotics in July 2026, applies one clipped-PPO objective within the denoising process across offline and online stages and reports 1000 of 1000 successful episodes on eight real tasks^{Source 32}32 Lei et al. 2025, RL-100: Performant Robotic Manipulation with Real-World Reinforcement Learning. Kun Lei, Huanyu Li, Dongjie Yu et al., 2025. LWD runs offline-to-online RL on a fleet of 16 dual-arm robots, where one generalist policy reaches 95% average success across eight real tasks^{Source 33}33 Wang et al. 2026, Learning While Deploying: Fleet-Scale Reinforcement Learning for Generalist Robot Policies. Yi Wang, Xinchen Li, Pengwei Xie et al., 2026.

Paper figure: Yuhui Chen et al. / arXiv. Licence: CC BY 4.0.

What RL buys, with caveats

Liu et al.'s NeurIPS 2025 study finds PPO beats the tested DPO/TPO and GRPO implementations for OpenVLA pick-and-place with sparse task rewards, with stronger semantic and execution generalization than supervised fine-tuning and comparable visual robustness. A September 2025 revision adds appendix tests, and SFT wins the articulated task's visual average and rephrased instructions, and the action-chunking multi-receptacle test; the main-text limitations still say all experiments are simulated pick-and-place^{Source 11}11 Liu et al. 2025, What Can RL Bring to VLA Generalization? An Empirical Study. Jijia Liu, Feng Gao, Bingwen Wei et al., NeurIPS 2025.

PAIR-VLA adds paired-view invariance and sensitivity objectives to PPO in simulated ManiSkill3 pick-and-place and reports better OOD success than PPO under the tested shifts. The sensitivity-only ablation loses to PPO on target-pose success, the camera-view variant trails PPO slightly in-distribution, the task-preserving views rely on simulator object masks, and real-world transfer is untested^{Source 12}12 Peng et al. 2026, What to Ignore, What to React: Visually Robust RL Fine-Tuning of VLA Models. Yuanfang Peng, Jingjing Fu, Chuheng Zhang et al., 2026.

SimpleVLA-RL lifts OpenVLA-OFT's LIBERO average from 91% to 99%^{Source 34}34 Li et al. 2025, SimpleVLA-RL: Scaling VLA Training via Reinforcement Learning. Haozhan Li, Yuxin Zuo, Jiale Yu et al., 2025. Reward specification remains open: a benchmark of open and proprietary VLMs as robot reward models finds that no model excels across tasks^{Source 35}35 Lee et al. 2026, RoboReward: General-Purpose Vision-Language Reward Models for Robotics. Tony Lee, Andrew Wagenmaker, Karl Pertsch et al., 2026. Physical Intelligence says π0.7 runs an espresso machine out of the box at a level matching much more specialized RL-finetuned models^{Source 36}36 Ai et al. 2026, π0.7: a Steerable Generalist Robotic Foundation Model with Emergent Capabilities. Bo Ai, Ali Amin, Raichelle Aniceto et al., 2026.