NVIDIA and KAIST Cut Per-Task Robot Foundation Model RL Training Time by Up to 68.5%

Researchers from NVIDIA and KAIST have published ACCRUE, a continual off-policy reinforcement learning framework that reduces the real-robot training time required for each new task when deploying robot foundation models in sequence. In experiments on contact-rich assembly hardware, the authors report that later tasks in a multi-task RL sequence reached a 95-percent success threshold in 31.0 to 68.5 percent less robot interaction time on the AutoMate assembly suite, and 38.7 to 42.9 percent less on the NIST assembly suite, compared with retraining a fresh actor-critic from scratch for every task.

The preprint, submitted to arXiv on October 13, 2026, is co-authored by Changyeon Kim (NVIDIA and KAIST), Kimin Lee (KAIST), Yashraj Narang (NVIDIA), and Yijie Guo (NVIDIA). All quantitative results are author-reported from an unreviewed preprint; independent replication has not yet been conducted.

Why Real-Robot RL Time Is the Binding Constraint for Foundation Model Deployment

Robot foundation models such as NVIDIA's GR00T and Physical Intelligence's π0.5 arrive pretrained on large demonstration datasets, but they typically require task-specific reinforcement learning post-training on actual hardware before they can reliably perform precision manipulation in a given environment. Unlike simulation, real-robot RL cannot be parallelized across hundreds of virtual instances. Each policy update requires the physical robot arm to attempt the task, fail, recover, and retry — a cycle that makes wall-clock time on the robot the scarce resource that determines how quickly a model can be deployed across a new task library.

The standard approach is to treat each new task as an independent RL problem: initialize a fresh actor-critic on top of the frozen base model, collect trajectories until the policy converges, then discard that learner and repeat for the next task. That approach wastes the behavioral and value-function knowledge accumulated during prior tasks and forces full convergence costs each time.

How ACCRUE Carries the Learner Forward

ACCRUE's central idea is to carry the compact actor-critic module — not just the frozen base model — across tasks rather than reinitializing it. This design rests on three interlocking components.

First, the base model itself (π0.5 or GR00T, depending on setup) remains frozen throughout all RL stages. Only the lightweight actor-critic adapter trains, which keeps the problem tractable and the computational footprint small regardless of foundation model scale.

Second, the inherited critic from the previous task adapts using only current-task transitions collected off-policy. Because the critic already encodes a rough value landscape from prior assembly work, it guides actor updates more efficiently than a randomly initialized critic — an effect the authors describe as forward transfer. The off-policy nature of FastTD3 (used on real hardware) and FastSAC (used in simulation) means past transitions can be stored in a replay buffer and reused across gradient steps, improving sample efficiency relative to on-policy methods.

Third, to prevent the actor from drifting away from skills it already learned — the catastrophic forgetting problem — ACCRUE distills the actor toward cached behavioral targets drawn from a small set of retained episodes from each prior task. This acts as a regularization constraint rather than full experience replay: the cache is deliberately small, keeping storage requirements modest.

A subtler design choice links all three components. The visual features extracted by the frozen base model are fixed across tasks, which ensures that cached prior-task transitions remain valid replay data even if the base model is updated between deployment cycles. This means task-specific RL modules and foundation model upgrades can proceed on independent schedules — a separation with practical value for real fleet management.

Read more: World Action Agent: VLMs rehearse robot moves before physical execution

Tested on Contact-Rich Assembly Sequences

In real-robot experiments using π0.5 as the base model, the authors report that later tasks reached the stopping criterion — 95 percent success across the 20 most recent episodes — in roughly four to twelve minutes of robot interaction. Across a six-task sequence, the final policies achieved 118 out of 120 successes on the six earlier tasks, a retention rate of 98.3 percent, according to the ACCRUE project page.

In simulation using GR00T as the base model with FastSAC, the training-time reductions followed a similar pattern. The use of two different model-algorithm pairings is notable: it suggests the framework is not tuned to a single architecture, though systematic tests across a wider range of base models and task families have not yet been published.

What Sets This Apart From Prior Continual RL on Robots

The closest published comparison point is LifeLong-RFT (Liu et al., RSS 2026), which also targets continual learning for VLA models using reinforcement fine-tuning. LifeLong-RFT applies on-policy RL with a multi-dimensional process reward and reports a 22 percent success-rate improvement over supervised fine-tuning on LIBERO benchmarks. But on-policy updates discard trajectory data after each policy iteration, limiting reuse of prior experience and eliminating the forward-transfer advantage ACCRUE derives from its inherited critic. LifeLong-RFT does not report wall-clock training-time reductions.

The broader continual RL literature typically addresses catastrophic forgetting through replay (storing full trajectories), elastic weight consolidation (penalizing weight drift), or progressive networks (adding new network branches per task). ACCRUE's approach is lighter: it stores only small behavioral caches for distillation, carries the actor-critic rather than the full model, and uses off-policy transitions for both efficiency and forward transfer — a combination the paper presents as novel in the robot foundation model context.

Limitations and Open Questions

Several limits apply to the published results. The experiments cover assembly task families — structured, contact-rich manipulation with relatively bounded variation — and it is not yet clear how the actor-critic carry-forward performs on tasks with larger domain shifts, such as moving from assembly to flexible-object handling or from table manipulation to mobile manipulation. The training-time figures exclude demonstration collection and base-model fine-tuning, which are real costs in deployment. Cache size, a key hyperparameter governing the forgetting/efficiency trade-off, has not been characterized systematically. And with six tasks in the test sequence, the framework has not yet been demonstrated at the task-library scales a commercial robot operator might require.

Read more: Robot AI policies learn to fix hardware errors without retraining

The Modular Architecture Implication

The most significant architectural consequence of ACCRUE's design is one the abstract does not state directly: by keeping the base model frozen and making the RL adapter the unit of transfer, the framework implies a modular deployment architecture in which task-specific competencies and foundation model upgrades operate on separate cycles. A team could update the underlying foundation model — replacing π0.5 with a newer checkpoint — without invalidating the task-specific actor-critics carried from prior RL runs, provided the fixed visual feature assumption continues to hold. Whether that assumption survives major foundation model updates is the central open question for production deployment, and it is likely to drive follow-on research as robot fleet operators begin accumulating multi-task RL libraries at scale.