Correct Robots Before They Make Mistakes: Proactive Human-in-the-Loop Intervention via Preference Learning

Xinkai Jiang^*

Jing Jiang

Hongyi Zhou

Pankhuri Vanjani

Ahmad Baki

Jakub Suliga

Ge Li

Zhuoyue Li

Gerhard Neumann

Rudolf Lioutikov

Karlsruhe Institute of Technology (KIT)

^*Corresponding author

arXiv Paper (PDF)

Abstract

Proactive Human-in-the-Loop (PHIL) is a framework for proactive policy correction and preference-based post-training. It lets operators preview policy-predicted future motion and intervene before anticipated undesirable actions. During takeover, PHIL continues querying the policy and displaying its predicted trajectories in the physical workspace through an extended reality (XR) interface. PHIL pairs rejected policy predictions with corrective action chunks subsequently executed from the same physical state over the same horizon, and preference-based post-training encourages the policy to favor these corrections.

We evaluate PHIL on real-world manipulation with a diffusion policy and a pretrained vision-language-action policy. With five correction episodes per condition, PHIL increases mean success rates from 44.0% to 70.3% for Diffusion Policy and from 57.8% to 76.9% for X-VLA, compared with HG-DAgger. These are unweighted means across each policy’s evaluated tasks and demonstration budgets: four tasks for Diffusion Policy and three for X-VLA. Separately, proactive-trained policies show fewer unnecessary recovery behaviors in successful episodes.

How PHIL works

PHIL framework: initialize a base policy, preview future motion and collect rejected–corrected action pairs, then post-train with imitation and preference objectives. — PHIL combines trajectory previews, proactive human corrections, and preference-based post-training. Select the figure to view it at full size.

Reactive vs. proactive correction

Reactive correction responds after an error. PHIL previews future motion, allowing the operator to intervene before an anticipated undesirable action. The clip below illustrates the difference.

Proactive human intervention in XR

A Meta Quest 3 interface aligns predicted end-effector trajectories with the physical workspace. Holding the controller trigger enables correction; releasing it returns control to the policy. Predictions continue to update during human takeover.

Preference learning from corrections

PHIL pairs a rejected prediction with the next executed corrective action chunk, starting from the same query state and spanning the same horizon. Post-training combines imitation on the original demonstrations with a preference objective that favors the corrective continuation.

The positive chunk may extend beyond the end of takeover; the prediction accepted at release is excluded from negative samples. Only complete pairs are used for training. Actions run at 20 Hz, with a 30-step prediction horizon and a query every 10 control steps.

Real-world demonstrations

These selected rollouts compare HG-DAgger with reactive data, HG-DAgger with proactive data, and PHIL with proactive data. They illustrate behavior; aggregate results are reported below. All clips retain the source video’s on-screen annotations, pauses, and speed labels. There is no audio.

Towel Folding

Fold the towel in a single-stage manipulation task.

Cook the Carrot

Place the pot, add the carrot, and close the lid. The clip includes an annotated example of unnecessary lid-handling behavior.

Lemon in the Drawer

Open the drawer, place the lemon, and close it. The clip includes an annotated example of repeated drawer manipulation.

Experiments and results

Each condition uses five correction rollout episodes and 25 evaluation rollouts. Multiple interventions are allowed within a correction episode. Diffusion Policy is evaluated on Pick Up Banana, Towel Folding, Cook the Carrot, and Lemon in the Drawer. X-VLA is evaluated on the latter three tasks. Base-demonstration budgets are 3/6/9 for Diffusion Policy on Pick Up Banana, and 10/20/30 elsewhere.

Policy	HG-DAgger	PHIL	Evaluation scope
Diffusion Policy	44.0%	70.3%	4 tasks × 3 demonstration budgets
X-VLA	57.8%	76.9%	3 tasks × 3 demonstration budgets

Mean full-task success rates, averaged over each policy’s evaluated tasks and demonstration budgets. HG-DAgger uses reactive corrections; PHIL uses proactive corrections.

Diffusion Policy

PHIL matches or exceeds the compared adaptation baselines across the evaluated tasks, budgets, and metrics. HG-DAgger and Residual Policy use reactive corrections; PHIL uses proactive corrections.

Six-panel comparison of Base Policy, HG-DAgger, Residual Policy, and PHIL across demonstration budgets. — Left to right: success rate for Banana, Towel, Carrot, and Lemon; then task completion score for Carrot and Lemon. Zero-demonstration origins are visual guides, not measured conditions. Select the figure to view it at full size.

For Cook the Carrot with 10 demonstrations, PHIL reaches 52% success, compared with 24% for HG-DAgger and 4% for the base policy.

X-VLA

PHIL matches or exceeds HG-DAgger across the evaluated X-VLA tasks, budgets, and metrics, reaching 100% full-task success on Lemon in the Drawer with 30 base demonstrations.

All table values are percentages. SR is full-task success rate; TCS is normalized task completion score across three ordered stages. TCS is not reported for Towel Folding. Each condition is evaluated over 25 rollouts.

Towel Folding

Method	10 demos SR	10 demos TCS	20 demos SR	20 demos TCS	30 demos SR	30 demos TCS
Base Policy	32	—	36	—	60	—
HG-DAgger	32	—	40	—	64	—
PHIL	64	—	72	—	72	—

Cook the Carrot

Method	10 demos SR	10 demos TCS	20 demos SR	20 demos TCS	30 demos SR	30 demos TCS
Base Policy	40	73.3	56	77.3	56	85.3
HG-DAgger	56	78.7	48	76.0	76	90.7
PHIL	56	85.3	68	89.3	76	92.0

Lemon in the Drawer

Method	10 demos SR	10 demos TCS	20 demos SR	20 demos TCS	30 demos SR	30 demos TCS
Base Policy	0	37.3	4	46.7	0	53.3
HG-DAgger	68	72.0	56	78.7	80	88.0
PHIL	92	96.0	92	97.3	100	100.0

Correction collection and post-training

Proactive collection alone does not consistently improve success under behavior cloning. On the same proactive data, preference post-training matches or improves on HG-DAgger across all four tasks in the lowest-data regime.

Success-rate comparison of reactive, hybrid, and proactive data with HG-DAgger and PHIL on four tasks. — Three base demonstrations for Banana and ten for each other task; five correction episodes per condition. Hybrid combines two proactive and three reactive episodes. Mixing improves HG-DAgger here but reduces PHIL performance, showing that the benefit depends on the objective.

Unnecessary recovery behavior

For both HG-DAgger and PHIL, proactive-trained Diffusion Policies show fewer unnecessary recovery behaviors in every evaluated task–demonstration setting.

Unnecessary recovery counts across four manipulation tasks for policies trained with reactive or proactive corrections. — Counts are averaged over ten successful episodes per condition. Necessary recovery after actual errors is excluded. The horizontal axis is the share of correction frames in the combined training data and decreases to the right.

Takeaways

Preview before intervening. XR makes a policy’s proposed motion visible before it is executed.
Learn from the comparison. Rejected predictions and executed corrections provide context-aligned preference pairs.
Improve with a small correction budget. Five correction episodes improve mean success over HG-DAgger in the evaluated Diffusion Policy and X-VLA settings.