Earlier this year we taught our humanoid Asimov to walk with reinforcement learning. We expected that teaching it to manipulate objects would require an entirely new RL system. Instead, the lessons and machinery we developed for locomotion became the foundation of our manipulation pipeline. The RL core stayed largely the same, but we redesigned the pipeline around the manipulation setting and the sensors available at deployment. This post explains how the locomotion foundation shaped that pipeline and where manipulation required a different approach.
The task that we decided to work on is flipping cartons on a moving conveyor: on the production line, cartons ride the belt standing on their narrow edge and have to arrive downstream lying flat on their largest face. Achieving this requires a 90-degree flip about the carton's long axis, performed while the belt keeps moving. We wanted our Asimov dual-arm robot to intercept a carton, pinch it between two grippers, rotate it, and release it so that it settles as it continues down the belt.
Our locomotion work showed us why RL in simulation is useful: training across thousands of parallel environments exposed the policy to variations and edge cases that made it robust outside a single nominal setup. The carton flip presents a different set of variations. Cartons can arrive at different poses and orientations, and the belt can run at different speeds. We chose RL so the policy could experience this range at scale in simulation, applying the central lesson from locomotion rather than assuming that the walking skill itself would transfer.
Why manipulation is where RL from vision gets expensive
Robots run on real sensors. While flat-terrain locomotion can often rely on proprioception alone, manipulation tasks commonly require the robot to interact with objects, so vision is typically needed to perceive information such as an object's pose and motion.
Before training a manipulation policy, almost every robotics team faces one design decision: how should visual information enter the policy pipeline? The field has largely converged on three answers.
The first is to feed RGB straight into the RL loop and learn perception and control together from reward. This works, and QT-Opt and OpenAI's Rubik's cube hand both did it, but the sample cost is enormous. High-dimensional images inside the loop inflate GPU memory, which caps the number of parallel simulations you can run and drops training efficiency.
The second is teacher-student distillation: train a state-based teacher with RL using privileged simulator information, then distill its behavior into a vision-based student through supervised imitation. This has become a common recipe in recent work on vision-based, contact-rich manipulation, used by OmniReset, DextrAH-RGB, From Imitation to Refinement, and bimanual dexterous humanoid work, among others.
The third is modular perception plus state RL: convert RGB-D into compact object poses with an off-the-shelf estimator, then feed those poses to a state policy, as in IKER. Training stays cheap, but deployment is fragile, because the controller depends entirely on the pose estimator: occlusion, symmetry, calibration error, and latency all corrupt the policy input directly.
We chose the second approach:
- Train an RL teacher that cheats by reading the simulator's exact object state.
- Freeze the teacher and roll it out while recording its actions, the robot's proprioception, and the same camera views the student will use.
- Train a vision student to predict the teacher's actions from those camera observations and proprioception.
- Continue training with DAgger: roll out the student, ask the teacher what it would do in the states the student visits, add those labels to the dataset, and retrain.
- Deploy only the vision student.
DAgger, short for Dataset Aggregation, addresses a well-known weakness of plain imitation: a student trained only on the teacher's own trajectories drifts into states the teacher never visited, where it has never been shown what to do. By repeatedly labeling the states the student actually reaches with the teacher's action and retraining on them, DAgger teaches the student to recover from its own mistakes.
Only the first step is reinforcement learning. The rest is supervised imitation, including the iterative data aggregation in DAgger, which is stable and cheap by comparison. Recording camera views for imitation is much cheaper than rendering them inside the RL loop. The teacher is never deployed to the real robot; it supplies action labels for both its own demonstrations and the states encountered by the student. The policy that ships is the vision student. This lets the teacher read perfect simulator state while the student learns from deployable observations. That efficiency is what makes the flip tractable: we train the teacher in MuJoCo at 500 Hz physics across 16,384 parallel environments on a single GPU, which would be impossible with cameras rendering inside the RL loop.
Simulation
↓
RL teacher (reads privileged object state)
↓
Successful demonstrations
↓
Vision student (reads cameras + proprioception)
↓
Real robotLocomotion reaches the same goal, privileged in training and deployable at inference, in a single stage. A walking policy runs from an IMU and joint encoders, which the real robot already has, so we keep the privileged signals in the critic, train once, and deploy the actor as is. The flip cannot work that way, because the task is defined by where the carton is, and the robot can only recover that through vision. So the flip needs the extra distillation stage that locomotion never did.
Putting cameras inside the RL loop is what makes manipulation expensive, and the teacher-student split is how we avoid it.
What carried over
The teacher is a state-based RL policy built on the same core training setup as our locomotion work: PPO with an asymmetric actor-critic architecture. Its observations, actions, and task rewards are adapted for manipulation. Both policies train in mjlab, a GPU-accelerated MuJoCo environment, across 16,384 parallel environments.
Shared with locomotion
Algorithm PPO
Actor / critic MLP 512 / 256 / 128, ELU
Discount 0.99
GAE lambda 0.95
PPO clip 0.2
Learning rate 1e-3, adaptive
Environments 16,384 parallelWe also carried over the adversarial motion prior (AMP) approach as a style reward. In locomotion, we introduced AMP to guide the walking policy toward natural human gaits using retargeted human motion as the reference. For manipulation, we use the same idea to guide the policy toward the grasp-and-flip gesture we want, rather than leaving it to discover any motion that earns task reward. In both cases, a discriminator learns to distinguish the policy's motion from the reference, and the policy earns a small style reward for producing similar motion. Both projects compute that reward with the same formula:
r_style = clamp(1 - 0.25 * (d - 1)^2, 0, 1)The reference motion and task-specific features changed for the carton flip, but the AMP mechanism remained part of the shared RL foundation.
Designing the manipulation task
Observations for a privileged teacher
For manipulation, we designed the observations around a privileged state teacher with no vision. The actor receives five frames at 50 Hz covering the robot's state and previous action, the tool-center and gripper state, the carton's pose and motion, contact geometry, belt speed, and task progress. Most quantities are expressed in the robot's base frame, and equivalent carton orientations are canonicalized. Observation corruption can be enabled during robust training.
The asymmetric critic receives only the current frame, but adds exact simulator information such as contacts, success conditions, task timers, reset state, and sampled physical parameters. Its inputs remain uncorrupted. Because the teacher's actor directly observes the carton's state, it cannot run on the real robot; that is why we distill it into a separate vision student.
From vanilla rewards to AMP-guided rewards
The core training setup carried over, but the locomotion reward design did not. Locomotion is relatively forgiving to shape: a gait is periodic, ground contact is continuous, and dense rewards for tracking velocity and staying upright provide useful feedback throughout an episode. The carton flip is different. It is a staged, contact-rich event against a moving target, and the policy must discover a narrow sequence of reaching, grasping, rotating, and releasing.
Our first attempt was ordinary PPO with hand-designed dense rewards: approach the carton, penalize pose error, reward success. The first policy learned that standing next to the carton paid better than flipping it. It parked beside the carton and farmed the proximity reward forever.
We fixed that. The next policy learned to shove the carton over without grasping it, then collect the go-home bonus.
We fixed that too. The next policy learned to drag the carton slowly along the belt, staying just above the stall threshold to farm progress rewards for the whole episode.
By the end of the week we had eighteen reward terms and were spending more time tuning them against each other than solving the task. Each patch closed one exploit and opened the next.
Reward: stay close
↓
Policy hovers next to the carton forever
↓
Patch the reward
↓
Policy shoves the carton without grasping
↓
Patch the reward
↓
Policy drags the carton slowly, farming progress
↓
Eighteen reward terms, still no clean flipThe problem was not a lack of reward terms. Each dense signal created a place where the policy could earn value without completing the sequence. The optimizer repeatedly found the cheapest behavior that satisfied part of the objective. RL optimizes the reward you write, not the behavior you intend.
Instead of inventing a nineteenth reward term, we changed the design in two ways. First, we used AMP to guide exploration toward the kind of motion we wanted. Using a browser teleoperation tool that drives the arms through inverse kinematics, we recorded one successful flip. Rather than force the policy to copy it exactly, we gave the policy a small style bonus whenever its motion resembled the demonstration.
As in locomotion, the AMP discriminator learns online to distinguish the policy's motion from the reference. The policy earns a reward for fooling it, which provides guidance throughout the motion without turning the demonstration into a fixed sequence of waypoints.
The demonstration was collected with the belt stationary. The AMP features are expressed in the carton frame and relative to the belt surface, so the style prior still transfers to episodes where the belt is moving. This is the same reason the locomotion prior works across walking speeds: describe motion in the right frame and one demonstration covers many conditions.
With the style reward in place, the policy discovered the two-arm pinch-and-rotate strategy that pure task rewards never found. But AMP guides how the robot moves; it does not define when the task is complete. Its weight also needs care. Set it too high and the policy can earn more by resembling the demonstration than by finishing the flip, so it holds the carton forever. Kept small, AMP guides exploration while the task rewards determine the outcome. One demonstration did what eighteen reward terms could not.
That leads to the second change: we rebuilt the task reward around the stages of the flip, using three categories:
- Task guidance rewards provide dense, signed progress toward the active subtask, such as reaching the carton contact points, aligning the grippers, or rotating the carton.
- Milestone rewards provide one-time bonuses for completing meaningful stages: reaching the grasp pose, flipping the carton, releasing it stably, and completing the task. Terminal failure is handled here as a negative outcome.
- Smoothness rewards apply small penalties to rapid action changes and joint velocity so the learned motion remains controlled.
AMP sits alongside these task rewards as a style signal: it guides the grasp-and-flip gesture, while the staged rewards determine whether the task is actually completed.
| Term | Weight | Purpose |
|---|---|---|
relative_tcp_position | +5.0 | TCP position shaping relative to carton |
relative_tcp_orientation | +4.0 | TCP orientation shaping |
relative_tcp_pose_milestone | +5.0 | Milestone for reaching grasp pose |
flip_pose_dense | +0.2 | Dense flip-progress shaping |
flipped_milestone | +20.0 | Flip milestone bonus |
stable_release_milestone | +40.0 | Stable release after flip |
success_terminal | +100.0 | Successful downstream flip and stable release |
hard_failure | -50.0 | Terminal failure penalty |
post_task_home | +3.0 | Return home after release; the only signal active post-release |
action_rate | -0.001 | Clamped action-rate L2 penalty |
joint_velocity | -0.0005 | Joint-velocity L2 penalty |
The stage logic is built around three one-way latches: secure grasp, flipped, and stable release. Each latch records that the policy has completed a stage and controls which rewards can remain active next:
- Pre-grasp:
relative_tcp_positionandrelative_tcp_orientationpay only for improvement toward the target grasp pose. Holding still pays zero and moving away pays negative, so hovering cannot generate income. Both terms switch off permanently once the secure-grasp latch is set. - Grasp:
relative_tcp_pose_milestonepays +5 when both tool-center poses remain within tolerance for five control steps, then sets the secure-grasp latch. - Flip:
flip_pose_denseis active only after a secure grasp and before the flipped latch. We deliberately shape only the carton-face error here: averaging yaw and centerline into the dense signal gave a grasped but unrotated carton roughly two-thirds of the maximum reward, which encouraged holding it until timeout.flipped_milestonepays +20 only after the grasp latch and a full pose check, then sets the flipped latch. - Release: once the carton is flipped and neither gripper remains in contact,
post_task_homebecomes the only active shaping term and guides the hands away safely.stable_release_milestonepays +40 when the flipped carton remains stable for the required window, then sets the stable-release latch. - Terminal:
success_terminalpays +100 only when the carton reaches the downstream success region after completing the full grasp, flip, and stable-release chain.
A carton that stalls or gets pinned earns no flip-shaping income because that reward is also scaled by its downstream speed relative to the belt. A drop or conveyor collision triggers the hard-failure penalty and ends the episode, while pose errors remain recoverable rather than causing immediate termination. Together, the latches make each intermediate reward temporary and the milestones one-time, while AMP guides the desired motion toward final task success.
Distilling to the student policy
The state-based teacher cannot run on the real robot because signals such as the exact carton pose, belt speed, and contact state exist only in simulation. We distill its behavior into a vision student with online DAgger. A ResNet-18 processes each camera view, combines the visual features with the robot's proprioception, and predicts the teacher's end-effector actions.
We use front and overhead cameras rather than wrist cameras, which the two-arm grasp occludes at exactly the moment contact matters most. During teacher rollouts, we record these deployment camera views alongside proprioception and teacher actions. We then roll out the student, ask the teacher to label the states the student actually visits, and use those samples for continued imitation training.
What comes next
The pipeline, a state-based teacher distilled into a vision student, now completes the carton flip in simulation. The next step is to close the sim-to-real gap before moving to hardware. We will harden the student with visual and dynamics randomization, varying factors such as carton mass, friction, lighting, observation noise, actuator behavior, and latency. We will also calibrate the real cameras, measure their intrinsics and extrinsics, and match the simulated camera setup to those measurements.
We see this as a promising direction rather than a settled answer. It reuses what worked in our locomotion RL, where exposing the policy to many simulated variations produced robustness that hardware training alone could not, and asks whether that same breadth carries over to manipulation. Whether it closes the sim-to-real gap here is what the next stage will test.
The main lesson was not that a walking policy transfers directly to manipulation. It was that the core RL ideas from locomotion, such as large-scale simulation, PPO, asymmetric training, and AMP, provided a strong foundation, while manipulation required new observation, action, reward, and distillation designs.
This is still early work. If you have pointers to prior work on contact-rich RL from vision, or want to compare notes, we would like to hear from you.
