Robot foundation models are getting remarkably capable

Bimanual manipulation

Humanoids

Language-conditioned skills

One generalist policy — many tasks, many embodiments, out of the box. footage: Google DeepMind, Gemini Robotics

In language, capability already lives behind an API

$ curl https://api.openai.com/v1/fine_tuning/jobs \
    -H "Authorization: Bearer $OPENAI_API_KEY" \
    -d '{
      "model": "gpt-4o",
      "training_file": "file-demos-abc123"
    }'

{ "object": "fine_tuning.job", "status": "queued", ... }
→ ft:gpt-4o:your-org::a1b2c3

Robotics is following the same playbook

• The most capable robot foundation models may not be fully open-weight
• Capability lives behind a service, not a checkpoint
• Developers may get only one interface: a managed SFT API

Gemini Robotics On‑Device: a powerful foundation

• A powerful robot foundation model, designed to run locally on robots
• Exposed through a managed SFT API
• Fine-tuned on the same demonstrations, it already far outperforms open-weight alternatives

GROD (SFT) π_0.5 (SFT)

Box packing

93

59

Cup insertion

70

50

Bimanual plate handover

53

5

success rate % · 100 trials · same demonstrations

Imitation alone leaves a deployment gap

Success rate slides as contact gets richer

after deployment covariate shift real-world dynamics controller-specific failures

A limit of SFT-only adaptation — for any policy, not any particular model.

Every door into closed-loop training is locked

• Mastery takes practice: the robot must learn in closed loop
• Standard closed-loop learning needs gradients, losses, action likelihoods, custom objectives
• A closed-weight model exposes none of them — by design

The only access point: a managed SFT API.

CLIFT — Closed-Loop Iterative Fine-Tuning

• We cannot open the model —
  but we can change what it learns from
• Not just expert demonstrations: the robot’s entire deployment experience
• Successes, failures, and everything in between

problem setting

The entire interface: a dataset in, a policy out

// the managed SFT interface, abstractly
FSFT :  D  →  π

D = { (ot,  ℓ,  at) }  obs · instruction · action chunk

→ returns a fine-tuned policy π

exposed:  submit data · receive policy
hidden:   weights ✕  gradients ✕  likelihoods ✕  losses ✕

problem setting

The goal is reinforcement — the access is imitation

Every route that needs internal signals is ruled out. F_SFT only accepts data.

A deployment-time data flywheel — API-only, non-invasive

the flywheel

Four moves, all through the same API

↻ 04 hands the new policy back to 01 — the loop is the method

01Score

Judging rollouts like a human is the hard part

100 pairwise comparisons is the entire per-rollout labeling budget — plus one short rubric per task.

01Score

A reward model calibrated to human judgment

Select: keep the candidates whose induced return rankings best match human preferences · calibrate once, score every rollout.

01Score

Zoom in: a VLM becomes a dense reward generator

You are an expert reward labeler for robot manipulation videos.
TASK: {TASK_INSTRUCTION}

REWARD RUBRIC (anchor scores in [0,100]; interpolate):
  - 0: no relevant action; robot has not engaged
  - {RUBRIC}   ← milestones → anchor scores

HARD FAILURES (override the rubric): if visible in a
frame, its reward MUST be 0 — no matter what
milestone was reached earlier.

Frames arrive in SHUFFLED order — judge each purely
on visual content.

OUTPUT: JSON — { frame_index, task_reward, frame_description }

01Score

Zoom in: keep what humans agree with, distill it once

The human budget is spent exactly once — the distilled R_θ scores every rollout of every later cycle for free.

01Score

The reward model watches every rollout

• Scores task progress, execution quality, and safety at every step
• Judges rollouts like a human operator
• Credit assigned chunk by chunk, not per episode

01Score

On real rollouts, credit lands chunk by chunk

A failed packing episode still contains well-executed chunks · a successful insertion contains chunks a human would reject — episode-level labels can express neither.

02Relabel

A return only means something relative to its state

02Relabel

How a chunk earns its advantage token

Return

G(τ) = Σ γ^Δt R_θ(o_t; ℓ)

the chunk’s discounted return over its window

→retrieve

Similar states

DINOv3 cosine · best frame per rollout
their returns form the comparison set

→rank

✦ Binarize

this chunk

top 30%

Advantage=Trueabove the cut

Advantage=Falsebelow the cut

The percentile threshold auto-calibrates credit to state difficulty · human demonstrations are always Advantage=True.

02Relabel

Zoom in: retrieval instead of a value function

No value network, nothing retrained per cycle — the baseline is the robot’s own accumulated experience.

02Relabel

Reward feedback becomes ordinary SFT data

• Each chunk compared, via visual retrieval, against chunks from similar states → advantage token
• Observation and action exactly as executed; the advantage written into the instruction, in plain text
• To the API, this is just supervised data

02Relabel

After relabeling: four bins, one plain-text format

Observations and actions exactly as executed · both labels are trained on — to F_SFT, all of it is ordinary supervised data.

03Condition

At inference, the token becomes a switch

• During training the advantage token was a label
• At deployment it is flipped one way — True
• The policy steers toward what humans prefer, away from what the reward model rejected

03Condition

Zoom in: what deployment actually looks like

// every inference call, every task, every cycle
o = head cameras (regular + wide) + proprioception
ℓ = "Stack the plate from the rack, Advantage=True"

a = π(o, ℓ)   classifier-free guidance, β = 0.2

execute the 1.6 s action chunk → observe → replan

03Condition

Why a single token is enough

04Iterate

Deploy → score → relabel → fine‑tune — then again

• Relabeled rollouts appended to a cumulative dataset
• Base model re-fine-tuned from scratch each cycle — avoids distributional drift
• Each cycle: a stronger task specialist

04Iterate

Each cycle: grow the data, retrain from the base

04Iterate

Why keep the failures? The hardest task answers

Failure is not waste — it is exactly the training signal the hard tasks need.

04Iterate

Zoom out: the whole loop, one more time

Everything travels through F_SFT as ordinary supervised data — the model box never opens.

Real Unitree G1 humanoid · three contact-rich tasks

Whole-body VR teleoperation for demonstrations

Box packing · cup insertion · bimanual plate handover

Whole-body balancing throughout — end-effector pose and contact geometry keep shifting.

Generalist → specialist, in two cycles

Box packing

93%→100%

Cup insertion

70%→98%

Bimanual plate handover

53%→96%

Success rate over 100 trials per task · API-only · no additional human demonstrations

results

One chart, the whole story — starting from SFT

100 trials per bar · hatching = flywheel cycle, still dimmed · SFT alone: GROD 93/70/53 · π_0.5 59/50/5

results · finding 1

Two cycles turn GROD into a task specialist

Both GROD recipes beat the SFT baseline at every cycle · dense chunk relabeling wins the hardest task (96 vs ≈84)

results · finding 2

The same pipeline lifts π_0.5 too

With the high-return recipe, two cycles net a gain on every task — zero model surgery — but progress is uneven, and the ceiling sits far below GROD’s.

results · finding 3

The base model sets the ceiling — access doesn’t

π_0.5 granted invasive FiLM conditioning (green) — an architectural change CLIFT forgoes — still trails API-only GROD (blue) on every task (one invasive baseline evaluated).

Emergent behaviors from closed-loop practice

reorients the box in-hand to set up an easier grasp

recovers from a failed first insertion — retries and succeeds

Behaviors absent from the teleoperated demonstrations.

Same CLIFT recipe, opposite outcomes

why the collapse

π_0.5 cannot see its own failures

→

→

hand is empty ✕

after identical sft · bimanual plate handover GROD53%vsπ_0.55%

hypothesis — over-reliance on proprioceptive cues; on-policy SFT may reinforce those correlations.

What’s next: the RL pyramid for Physical AI

Learning to reason, act, and self-improve

On-device self-improve

on-device RL that keeps improving the model at deployment

Act · closed loop

real-to-sim closed-loop RL for robots

Learning in sensor-physics joint sim

Reason

RL for task understanding, planning & decision-making

Agentic research

agents propose, run, and analyze the experiments at every layer

Acknowledgments