RL Fine-Tuning of Policies

Imitation learningimitation learningLearning a policy from expert demonstrations rather than from a reward signal: the expert's recorded state-action pairs become a supervised training set, and the fitted mapping from observed state to action is the policy. Pomerleau's ALVINN (1988) instead trained on simulated road images before steering NAVLAB, and the recipe still underlies most learned manipulation. Its structural weakness is that the training distribution comes from the expert while deployment visits the states the learner itself induces, the mismatch DAgger was designed to repair. gets a policy that is good on the demonstration distribution and brittle off it, because small per-step errors compound into states the demonstrator never visited Ross 2011↓A Reduction of Imitation Learning and Structured Prediction to No-Regret Online LearningStéphane Ross, Geoffrey J. Gordon, J. Andrew Bagnell, AISTATS 2011. Physical Intelligence states the operational version of the same problem: it is relatively easy to get a VLA to succeed at a task some of the time, and quite hard to make it succeed reliably Amin 2025↓π*0.6: a VLA that Learns from ExperienceAli Amin, Raichelle Aniceto, Ashwin Balakrishna et al., 2025. Reinforcement learning is the only mechanism in the toolbox that learns from failure rather than only from success. The obstacle is cost: naive online RL on a multi-billion-parameter VLA, on real hardware, at control rates, is infeasible. Everything in this module is a way around that constraint, and by 2026 the field has converged on a small set of recipes that demonstrably work.

The scrubber below uses a fictional espresso episode to illustrate conditioning. Its timings, values and stage tags are toy choices, not measurements from Recap. Drag the slider to compare keeping high- and low-tag examples during training with asking for the high-conditioned policy at execution.

The flow-matching log-likelihood problem

Modern policies emit actions through diffusion or flow-matchingflow matchingA generative modeling recipe that learns a time-dependent vector field carrying samples from a noise distribution to the data distribution, trained by regressing the model's field against the conditional flow rather than by estimating a score. pi0 brought it to robot control: a dedicated action expert is trained with the flow-matching objective to denoise continuous action chunks, and at inference the learned field is integrated as an ODE for a handful of steps, which keeps the policy fast enough for high-frequency control while keeping the multimodality that made diffusion policies attractive. heads, and that choice breaks the standard policy-gradient machinery. PPOPPOProximal policy optimization, the on-policy reinforcement learning algorithm the locomotion literature standardized on. Schulman and colleagues introduced it as a policy gradient method that alternates between sampling data through interaction with the environment and optimizing a surrogate objective with stochastic gradient ascent, where the surrogate is what allows several epochs of minibatch updates on each batch of samples instead of the single gradient step standard policy gradient takes. It keeps most of trust region policy optimization's reliability without the second-order machinery, and it pairs naturally with massively parallel simulation, where thousands of robots supply the enormous on-policy batches it consumes.-style updates need the log-likelihood of the action the policy actually took; for a flow-matching head the "action" is the endpoint of an ODE integration, and its likelihood under the model is intractable. For two years this was the standard argument for why RL could not touch the frontier VLAs directly. Three methods dissolve it in different ways Ren 2024↓Diffusion Policy Policy OptimizationAllen Z. Ren, Justin Lidard, Lars L. Ankile et al., 2024 Chen 2025↓ConRFT: A Reinforced Fine-tuning Method for VLA Models via Consistency PolicyYuhui Chen, Shuai Tian, Shugao Liu et al., 2025 Chen 2026↓π_RL: Online RL Fine-tuning for Flow-based Vision-Language-Action ModelsKang Chen, Zhihao Liu, Tonghe Zhang et al., 2026.

DPPO (Ren et al., 2024) embeds the diffusion denoising process inside the environment MDP. Each policy action in this combined MDP is a denoising transition with an analytically evaluable Gaussian likelihood; the environment advances and supplies reward only after the final denoising step. PPO updates the denoising policy, not the environment dynamics. Environment-discounted advantages are multiplied by a separate denoising discount that downweights earlier, noisier steps. The value function depends on the environment state. The paper also fine-tunes only the last selected denoising steps, using a frozen original network for earlier steps and a trainable copy for the selected steps Ren 2024↓Diffusion Policy Policy OptimizationAllen Z. Ren, Justin Lidard, Lars L. Ankile et al., 2024.

The v3 comparisons are specific to their benchmarks and protocols. DPPO is the strongest performer among the compared diffusion-based RL methods on the paper's state-input Robomimic tasks; the PPO comparisons also cover Gaussian and Gaussian-mixture policies with different network architectures. This is not a universal efficiency ranking: some Gym baselines are competitive, off-policy methods are more sample-efficient on HalfCheetah, and Appendix D reports slower wall-clock iterations than several baselines. The Gaussian baseline also slightly outperforms DPPO on the low-randomness Lamp task Ren 2024↓Diffusion Policy Policy OptimizationAllen Z. Ren, Justin Lidard, Lars L. Ankile et al., 2024.

Its physical demonstration is zero-shot transfer of a simulation-trained One-leg assembly policy to a Franka Emika Panda, not on-robot RL fine-tuning. At 10 Hz, DPPO succeeds in 16 of 20 hardware trials. The simulation comparisons, this small physical evaluation, and wall-clock time per training iteration measure different things Ren 2024↓Diffusion Policy Policy OptimizationAllen Z. Ren, Justin Lidard, Lars L. Ankile et al., 2024.

ConRFT (Chen et al., 2025) fine-tunes a consistency-policy action head while keeping the visual encoders and transformer backbone frozen. The main eight-task study uses Octo-small on a 7-DoF Franka Emika arm at 10 Hz. Offline Cal-ConRFT combines a consistency-based behavior cloningbehavior cloningTraining a policy by supervised learning on expert demonstrations: each recorded observation is an input, the action the expert took at that moment is the label, and the fitted mapping from state to action is the policy. The recipe predates deep learning; ALVINN (1988) used camera and laser-range inputs and trained on simulated road images before NAVLAB tests, and the same approach, scaled to modern networks and datasets, still underlies most learned manipulation policies. Its known failure mode in closed loop is covariate shift. loss and a negative-Q actor loss with a Cal-QL critic, starting from 20-30 demonstrations per task. Online HIL-ConRFT keeps both actor losses, changes their weights, and uses a standard temporal-difference critic loss. Training batches sample equally from the demonstration and online replay buffers; human takeover transitions enter the demonstration buffer Chen 2025↓ConRFT: A Reinforced Fine-tuning Method for VLA Models via Consistency PolicyYuhui Chen, Shuai Tian, Shugao Liu et al., 2025.

Do not confuse training noise levels with inference steps. Appendix B specifies 40 sub-intervals for the consistency policy's training noise grid. Its separate five-step setting describes the diffusion-policy baseline, not a measured ConRFT inference-call count Chen 2025↓ConRFT: A Reinforced Fine-tuning Method for VLA Models via Consistency PolicyYuhui Chen, Shuai Tian, Shugao Liu et al., 2025.

In v2, Table I reports 96.3% mean success ratesuccess rateThe standard headline metric of robot learning evaluation: the fraction of attempted episodes in which the policy completes the task. Trial count, time limit and scene distribution belong with the reported rate. Toyota Research Institute's Large Behavior Model study analyzed 1,800 real-world rollouts and warned of a significant risk of statistical noise from insufficient power, not a measured prevalence of incorrect papers. An inconclusive comparison does not establish policy equivalence. across eight tasks, with 20 trials per task. The listed online training times range from 15 to 90 minutes, with a reported mean of 48.8 minutes; the abstract and results prose instead state 45-90 minutes. Training time includes scripted motions, policy rollouts and onboard computation on an NVIDIA RTX A6000. The table's 144% is a rounded relative gain over the corresponding offline Cal-ConRFT mean of 39.4%, not a percentage-point gain. The separate SFT baseline happens to have the same reported mean Chen 2025↓ConRFT: A Reinforced Fine-tuning Method for VLA Models via Consistency PolicyYuhui Chen, Shuai Tian, Shugao Liu et al., 2025.

Keep the intervention caveats with that comparison. The online learning curves can include successes produced by human takeovers, so they are not autonomous-policy success measurements. Table I's caption says all methods use human interventions, but Figure 3 and the PA-RL discussion explicitly describe PA-RL without them. These are the authors' task-specific results, not independent replication; the paper also warns about reward-classifier hacking and limited generalization Chen 2025↓ConRFT: A Reinforced Fine-tuning Method for VLA Models via Consistency PolicyYuhui Chen, Shuai Tian, Shugao Liu et al., 2025.

pi_RL (Chen et al., 2026; v3) fine-tunes flow-based policies with online PPO. Flow-Noise learns Gaussian noise for denoising transitions jointly with the velocity network, then uses the joint likelihood of the denoising path. This is not an exact marginal likelihood for the final executed action. The noise network is discarded for deterministic inference. Flow-SDE uses Gaussian transition likelihoods in a two-layer denoising/environment MDP; its hybrid sampler makes one randomly chosen denoising step stochastic and leaves the others as deterministic ODE updates. The PPO objective is distinct from the conditional flow-matching regression used in supervised training Chen 2026↓π_RL: Online RL Fine-tuning for Flow-based Vision-Language-Action ModelsKang Chen, Zhihao Liu, Tonghe Zhang et al., 2026.

The main experiments use pi0 and pi0.5. Appendix B.2 reports full-model supervised fine-tuning followed by a frozen VLM and a trainable 300M-parameter action expert, on eight NVIDIA H100 80GB GPUs. Appendix F separately compares PPO with GRPO and studies VLM LoRA; Appendix H evaluates PPO with Flow-SDE on GR00T N1.5. These are different evaluated configurations, not a universal best recipe Chen 2026↓π_RL: Online RL Fine-tuning for Flow-based Vision-Language-Action ModelsKang Chen, Zhihao Liu, Tonghe Zhang et al., 2026.

The ID comparisons cover LIBERO, ManiSkill, MetaWorld and CALVIN. LIBERO reports 500 initial states per task suite; the CALVIN Scene D evaluation uses 1,000 five-subtask episodes. ManiSkill tests visual, semantic and execution shifts. Gains under distribution shifts do not consistently extend to the five unseen tasks in MetaWorld ML45, which trains on 45 tasks Chen 2026↓π_RL: Online RL Fine-tuning for Flow-based Vision-Language-Action ModelsKang Chen, Zhihao Liu, Tonghe Zhang et al., 2026.

Two setup descriptions disagree within v3. Section 4.3.2 puts pi0.5's proprioceptive state into the VLM, whereas Appendix B.2 says its pi0.5 setting omits state. Appendix D.2 describes CALVIN ABC-to-D evaluation, but its results also say D-to-D training. Neither conflict establishes one unambiguous input or OOD training protocol Chen 2026↓π_RL: Online RL Fine-tuning for Flow-based Vision-Language-Action ModelsKang Chen, Zhihao Liu, Tonghe Zhang et al., 2026.

RL training is simulated. A separate Real2Sim2Real case uses 20 motion-planner trajectories for supervised fine-tuning and 100 RL iterations, then transfers zero-shot to a physical Franka Panda with a RealSense D435 camera. Section E.2 reports 40% real-world success without stating an evaluation-trial denominator. This is a separate transfer test, not on-robot RL training or a general reliability guarantee Chen 2026↓π_RL: Online RL Fine-tuning for Flow-based Vision-Language-Action ModelsKang Chen, Zhihao Liu, Tonghe Zhang et al., 2026.

Recap: advantage-conditioned policy learning

Amin 2025↓π*0.6: a VLA That Learns From ExperienceAli Amin, Raichelle Aniceto, Ashwin Balakrishna et al., 2025 Recap, the method behind π*0.6, combines demonstrations, expert teleoperation corrections during rollouts, and autonomous experience. The training sequence is offline-RL pre-training, demonstration fine-tuning for a task, then batch collection and retraining; it is not a continuously updated online policy.

The language-conditioned distributional value function predicts negative remaining steps for successful episodes and a large negative value for failures, with per-task normalization. For post-training, Appendix F estimates advantage using rewards over the next N = 50 steps plus the later value minus the current value. Pre-training uses a separate episode-return estimator. A task-dependent threshold turns the estimate into a binary indicator, and human corrections are explicitly assigned the positive indicator Amin 2025↓π*0.6: a VLA That Learns From ExperienceAli Amin, Raichelle Aniceto, Ashwin Balakrishna et al., 2025.

The text input “Advantage: positive” or “Advantage: negative” appears after the predicted subtask and before the actions. Both good and bad data can train the policy. The default evaluation uses the positive-conditioned policy; indicator dropout during training also permits conditional/unconditional classifier-free guidance. A raw increase in value is not the complete advantage estimator or its threshold rule Amin 2025↓π*0.6: a VLA That Learns From ExperienceAli Amin, Raichelle Aniceto, Ashwin Balakrishna et al., 2025.

Section III first recalls the exponential solution for a KL-regularized objective:

Recap's practical binary-conditioning method is motivated by a related improvement-indicator construction, not by asserting that binarization exactly implements that exponential weighting. Equation 2 instead writes the improved policy as

At $\beta_{\mathrm{CFG}}=1$, this reduces to the indicator-conditioned policy. The distinct subscripts here separate the paper's two uses of β; they do not claim the two constructions are identical Amin 2025↓π*0.6: a VLA That Learns From ExperienceAli Amin, Raichelle Aniceto, Ashwin Balakrishna et al., 2025.

This avoids a PPO-style policy-extraction objective, not training gradients through the action expert. The actual objective combines discrete next-token prediction with a continuous flow-matching loss. Knowledge Insulation stops the action expert's gradient at the VLM-backbone interface; the action expert itself is trained Amin 2025↓π*0.6: a VLA That Learns From ExperienceAli Amin, Raichelle Aniceto, Ashwin Balakrishna et al., 2025 Physical Intelligence 2025↓π0.6 Model CardPhysical Intelligence, 2025.

The scrubber is a deterministic teaching example, not a recorded Recap episode or an implementation of its estimator. Its forty-second timeline, twenty-second credit-assignment arc, arbitrary positive values and stage-difference tags are chosen for illustration. Real Recap uses reward-inclusive estimates, task-dependent thresholds and a failure penalty. The execution view illustrates the requested conditioning, not a guarantee that every low-tag behavior disappears.

Recap's on-robot experience more than doubles successful completions per hour for double-shot espresso and the diverse-laundry evaluation, relative to offline RL plus task-specific demonstration fine-tuning. These are results on the report's static bimanual setup, with a button-up-shirt test rather than the full 11-item-type laundry training set Amin 2025↓π*0.6: a VLA That Learns From ExperienceAli Amin, Raichelle Aniceto, Ashwin Balakrishna et al., 2025. The pi-line evaluation details give the hardware, task protocols and time limits.

The paper qualifies its reliability claims: Section VI-C describes failure reduction as “about a factor of two,” while Figure 8 says “more than 2×.” Its “90%+” success summary excludes diverse laundry, and the box chart reports separate stages. This does not establish strictly greater than 90% end-to-end success for every application Amin 2025↓π*0.6: a VLA That Learns From ExperienceAli Amin, Raichelle Aniceto, Ashwin Balakrishna et al., 2025. See the task-by-task qualifications.

The companion blog separately says that espresso throughput and success rate more than doubled after adding on-robot experience. Its demonstration-video caption gives 5:30am to 11:30pm for making various espresso drinks, while the report's introduction says “13 hours straight.” These are different duration accounts; neither supplies a denominator for the quantitative double-espresso result, and they should not be merged into one continuous-run success statistic Amin 2025↓π*0.6: a VLA that Learns from ExperienceAli Amin, Raichelle Aniceto, Ashwin Balakrishna et al., 2025 Amin 2025↓π*0.6: a VLA That Learns From ExperienceAli Amin, Raichelle Aniceto, Ashwin Balakrishna et al., 2025.

These are Physical Intelligence's own reported results, not independent replication. The report describes human reward labeling, interventions and episode resets, and batch offline updates. Intervention use varies by task: the T-shirt/shorts improvement experiment uses autonomous evaluation data without expert corrections Amin 2025↓π*0.6: a VLA That Learns From ExperienceAli Amin, Raichelle Aniceto, Ashwin Balakrishna et al., 2025.

HIL-SERL: a human in the loop, RL on the robot

HIL-SERL (Luo et al., 2024) combines vision-based off-policy RL built on RLPD with demonstrations and human corrections. The learner samples equally from demonstration and online replay buffers. Intervention transitions enter both buffers; autonomous policy transitions enter only the RL buffer. Training typically starts with 20 to 30 demonstrations. Jenga whipping is an exception to online correction: it starts with 30 demonstrations and does not use real-time human takeovers Luo 2024↓Precise and Dexterous Robotic Manipulation via Human-in-the-Loop Reinforcement LearningJianlan Luo, Charles Xu, Jeffrey Wu et al., 2024.

Table 1a reports 100% observed success in 100 evaluation trials per task, except IKEA whole assembly, which uses 10 trials and allows at most two attempts per sub-policy. Training takes 1 to 2.5 hours for nearly all tasks, but timing-belt assembly takes 6 hours. The reported time includes scripted motion, policy rollouts, intended stops and onboard computation on one RTX 4090. Table 1a reports average success of 49.7% for the imitation baseline and 100% for HIL-SERL, about twice the success rate, and average cycle times of 9.6 versus 5.4 seconds, described as 1.8× faster. These averages compare against HG-DAgger with matched episodes and interventions, except for flat BC on Jenga whipping and object flipping, trained on 50 and 200 demonstrations respectively. The separate Table 1b comparison uses 200 demonstrations for DP, BC, IBRL, Residual RL and DAPG; that is not HIL-SERL's universal initialization Luo 2024↓Precise and Dexterous Robotic Manipulation via Human-in-the-Loop Reinforcement LearningJianlan Luo, Charles Xu, Jeffrey Wu et al., 2024.

The online learning curves can include human-assisted successes and are not the same measurement as Table 1a evaluation. A separate baseline inconsistency also remains: Table 1b gives DP 18% on dashboard assembly, while the prose says 28%; neither value is silently chosen as definitive Luo 2024↓Precise and Dexterous Robotic Manipulation via Human-in-the-Loop Reinforcement LearningJianlan Luo, Charles Xu, Jeffrey Wu et al., 2024.

Luo and colleagues did not test generalization in unstructured environments or perform extensive randomization, and they leave substantially longer-horizon tasks uncertain. The paper reports code availability, not a field-wide ranking or a fleet-scale staffing result. The RL for Robotics module covers the off-policy context Luo 2024↓Precise and Dexterous Robotic Manipulation via Human-in-the-Loop Reinforcement LearningJianlan Luo, Charles Xu, Jeffrey Wu et al., 2024.

Residual RL and distillation back into the generalist

PLD (Probe, Learn, Distill; Xiao et al., 2025) freezes a VLA while training task-specific residual actors with off-policy RL. A hybrid collection stage first rolls out the base policy, then lets a residual specialist recover from the states it reaches. The resulting trajectories fine-tune the generalist through SFT. Freezing the base during specialist training does not mean it stays frozen during distillation, or that residual exploration guarantees safety Xiao 2025↓Self-Improving Vision-Language-Action Models with Data Generation via Residual RLWenli Xiao, Haotian Lin, Andy Peng et al., 2025.

Table 1 reports 99.2% average success for the model labelled OpenVLA, versus 91.8% for its baseline; pi0 reaches 97.2% versus 93.4%. These results cover LIBERO-Spatial, Object and Goal, ten tasks per suite, with 50 evaluation episodes per task, not every LIBERO task. On a separate four-task SimplerEnv suite, Table 2 reports Octo-SFT improving from 71.8% to 96.6% on average. The WidowX carrot task rises from 43.3% to 93.9%, a 50.6-percentage-point gain. The table's printed average delta is +24.9, although its displayed means differ by 24.8 points; neither is a benchmark-wide relative gain above 50% Xiao 2025↓Self-Improving Vision-Language-Action Models with Data Generation via Residual RLWenli Xiao, Haotian Lin, Andy Peng et al., 2025.

The Franka experiment starts with 200 teleoperated trajectories. After RL data generation and distillation, PLD records 30/30 randomized trials each for cube pick-up and peg insertion; peg insertion also reaches 30/30 with the comparison data sources. Under changed clutter and object conditions, PLD records 28/30 for a blue cube and 20/30 for a red cube. These finite trial results do not imply perfect reliability across tasks Xiao 2025↓Self-Improving Vision-Language-Action Models with Data Generation via Residual RLWenli Xiao, Haotian Lin, Andy Peng et al., 2025.

The abstract's 100% real-robot headline needs a further qualification for YAM. The detailed experiment describes four GPU insertion/removal stages, at most eight hours of training per subtask, and distillation into one BC policy. It reports at least one hour of continuous operation with recovery, while explicitly stating that per-stage one-shot success is not 100%. Human setup is not absent: the Franka peg experiment uses human hole repositioning. These results support a bounded data-generation recipe, not universal self-improvement or a safety guarantee Xiao 2025↓Self-Improving Vision-Language-Action Models with Data Generation via Residual RLWenli Xiao, Haotian Lin, Andy Peng et al., 2025.

RLDG (Xu et al., 2024) uses task-specific RL policies to generate data for fine-tuning generalists, rather than applying RL updates directly to the generalist. The paper implements its specialists with HIL-SERL and evaluates fine-tuned OpenVLA and Octo. Its comparisons match the task setup, training configuration, observation and action spaces, and number of successful training episodes, changing the data source. In the FMB single-object assembly experiment, RL supplies the insertion-stage data while human demonstrations still supply grasping and transport. OpenVLA succeeds in 20/20 trials with this mixed dataset versus 12/20 with human demonstrations: 100% versus 60%, a 40-percentage-point gain, not a 40% relative increase Xu 2024↓RLDG: Robotic Generalist Policy Distillation via Reinforcement LearningCharles Xu, Qiyang Li, Jianlan Luo et al., 2024.

Xu and colleagues' FMB-insertion ablation relabels human-data actions with an RL policy. Its results suggest that both action quality and state distribution contribute, with action quality accounting for most of the benefit; they do not establish that recovery-state coverage is always the cause. The method still requires task rewards, and the authors report that speed-optimized RL data can produce premature object drops after distillation Xu 2024↓RLDG: Robotic Generalist Policy Distillation via Reinforcement LearningCharles Xu, Qiyang Li, Jianlan Luo et al., 2024.

The paper's OpenVLA setup commands actions at 4 Hz; data collection, RL and Octo use 10 Hz. The separate low-level impedance controller runs at 1 kHz. These are setup-specific rates, not a cross-model throughput ranking Xu 2024↓RLDG: Robotic Generalist Policy Distillation via Reinforcement LearningCharles Xu, Qiyang Li, Jianlan Luo et al., 2024.

The six methods side by side

Read these as source-specific reports, not a cross-paper ranking. Recap's results are the lab's own reports on its own hardware, not independent replication. Publication type alone does not establish independent verification.

DPPO, ConRFT, HIL-SERL and pi_RL state that code is available. Those statements do not verify repository contents, particular downloadable weight checkpoints or license terms. The inspected PLD preprint does not establish a code or weight release. Recap's inspected report, blog and model card do not establish model-specific checkpoint availability or licensing; this is not a claim that the model is closed. The openness notes keep those source scopes with each row Ren 2024↓Diffusion Policy Policy OptimizationAllen Z. Ren, Justin Lidard, Lars L. Ankile et al., 2024 Chen 2025↓ConRFT: A Reinforced Fine-tuning Method for VLA Models via Consistency PolicyYuhui Chen, Shuai Tian, Shugao Liu et al., 2025 Luo 2024↓Precise and Dexterous Robotic Manipulation via Human-in-the-Loop Reinforcement LearningJianlan Luo, Charles Xu, Jeffrey Wu et al., 2024 Chen 2026↓π_RL: Online RL Fine-tuning for Flow-based Vision-Language-Action ModelsKang Chen, Zhihao Liu, Tonghe Zhang et al., 2026 Amin 2025↓π*0.6: a VLA That Learns From ExperienceAli Amin, Raichelle Aniceto, Ashwin Balakrishna et al., 2025 Amin 2025↓π*0.6: a VLA that Learns from ExperienceAli Amin, Raichelle Aniceto, Ashwin Balakrishna et al., 2025 Physical Intelligence 2025↓π0.6 Model CardPhysical Intelligence, 2025 Xiao 2025↓Self-Improving Vision-Language-Action Models with Data Generation via Residual RLWenli Xiao, Haotian Lin, Andy Peng et al., 2025.

What RL actually buys, and the caveats

Liu et al.'s NeurIPS 2025 study reports that PPO beat the tested DPO/TPO and GRPO implementations in its OpenVLA pick-and-place setup, where preferences and returns came from sparse task rewards. The authors summarize the main OOD comparison with supervised fine-tuning as stronger semantic and execution generalization, with comparable visual robustness; this is not a test of arbitrary new-task generalization. The inspected January 2026 v4 appendix also includes a preliminary sim-to-real trial and an articulated-object task, despite the main text's narrower limitation statement. Its results are not uniform: SFT wins on the articulated-task visual average and rephrased instructions, and on the action-chunking multi-receptacle test Liu 2025↓What Can RL Bring to VLA Generalization? An Empirical StudyJijia Liu, Feng Gao, Bingwen Wei et al., NeurIPS 2025.

Peng et al.'s PAIR-VLA study addresses a related limitation rather than showing that RL newly creates visual fragility. In simulated ManiSkill3 pick-and-place experiments with OpenVLA and π0.5, it adds paired-view invariance and sensitivity objectives to PPO: preserve action distributions for task-preserving changes and separate them for target-pose changes. The authors argue that task rewards alone give limited guidance about this distinction and report improved OOD success over standard PPO under the tested visual shifts. The sensitivity-only ablation loses to PPO on target-pose success; the separate camera-view experiment, using a different task-preserving view construction, also trails PPO slightly in-distribution despite its OOD gains. That comparison does not isolate reward design as the cause of a pre-to-post-RL loss of robustness; its task-preserving views also rely on simulator object masks, and real-world transfer remains untested Peng 2026↓What to Ignore, What to React: Visually Robust RL Fine-Tuning of VLA ModelsYuanfang Peng, Jingjing Fu, Chuheng Zhang et al., 2026.