DUO Policy: Learning Fine-Grained Manipulation
from Human Videos via Dual-Actor Policy

Abstract

Behavioral cloning is an effective way to learn visuomotor policies, but it depends on slow and expensive robot data collection. Prior work has shown that human videos can be a useful alternative data source for manipulation. However, embodiment and precision gaps limit the resulting cross-embodiment policies to coarse behaviors, so they cannot learn precise tasks such as insertion. We propose DUO Policy (Dual-Actor Policy), a visuomotor policy learning approach that reduces how much robot data is needed to train a robust policy. It does this by splitting the learned skill into a human hand policy and a robot policy. DUO Policy uses a dual-actor architecture with an adaptive gate. The gate combines a hand policy pretrained on low-cost human videos, which handles coarse manipulation, with a robot policy trained on sparse robot data, which handles fine-grained manipulation. We evaluate the approach on complex manipulation tasks, including coffee pod insertion, laptop closing, and grasping in cluttered scenes. DUO Policy outperforms the best baseline by an average of 17% on four simulation tasks and 52% on three challenging real-world tasks, and it is 3× more efficient in wall-clock data-collection time.

Real-World Tasks

We evaluate on three real-world tasks that each need fine-grained control. The videos show DUO Policy rollouts (playback speed is marked in the corner).

Supplementary Video

Not loading? Watch on Google Drive.

Why Is Fine-Grained Manipulation Hard?

A fine-grained task naturally splits into a coarse, free-space phase and a precise, contact-rich phase. Robot data is precise but sparse, so a robot-only policy easily drifts out of distribution in free space. Human hand data covers the workspace densely but is imprecise because of the embodiment gap, so a hand-only policy fails at narrow "chokepoints" such as inserting a pod. DUO Policy uses each data source where it is strongest.

Method

DUO Policy has two branches. The hand policy π_hand is pretrained on human videos and frozen. The robot policy π_robot is trained on a small amount of robot data. A state-dependent binary policy gate g decides which branch acts at each step:

π(o) = g(o) π_robot(o) + (1 − g(o)) π_hand(o)

From human video to robot-ready point clouds

Real-World Results

Success rates on the three real-world tasks shown in the task videos above.

Success rates

Data-collection efficiency

On Candy Sorting, 10 minutes of human hand video (216 demonstrations) plus 30 minutes of human-in-the-loop correction brings DUO Policy to 90% success. With the same 40-minute budget of SpaceMouse teleoperation, DP3 reaches only 35%, and it needs about 70 more minutes of robot data to become competitive.

Simulation Results

On MimicGen tasks, every method is pretrained on noisy Panda demonstrations without in-hand views, to mimic human video. Each is then finetuned with 20, 30 or 40 UR5 demonstrations. Numbers are success rates (%), and parentheses show the difference from the best baseline.

Spatial generalization

Convergence and robustness to noisy human data

Ablations (Coffee D2, 3 seeds)

BibTeX

@article{duopolicy2026,
  title   = {Learning Fine-Grained Manipulation from Human Videos via Dual-Actor Policy},
  author  = {Jia, Mingxi and Yin, Patton and Huang, Haojie and He, Ivy Xiao and Liu, Xuanru and Liu, Gefei and Wang, Dian and Tellex, Stefanie},
  journal = {arXiv preprint},
  year    = {2026}
}