Isaac Lab PegInsert Reward Search

Design a better training reward so a robot learns to align and insert a peg more reliably, with the simulator and training budget held fixed.

The climb

Both agents search for a better training reward. Each dot is the best recipe from one scored Judge submission, and the step lines show the best result reached so far. The dashed line is the original reward's validation-measured success rate. Four submissions have no official score; they remain in the tables as dashes, not zero-valued points.

The task

A robot must align a peg with a socket and insert it. The agent does not control the robot directly: it designs the reward that guides reinforcement learning. A better recipe should teach the same policy to succeed more often within the same training budget.

Environment

• A pinned Isaac Lab Factory PegInsert environment, offline simulator assets, and the original reward recipe are provided.
• The deliverable is a declarative reward graph, with one optional alternative recipe. The graph combines allowed sensor values, mathematical operations, and bounded memory; it cannot execute arbitrary code.
• Work has two H100 GPUs, and Judge has two separate H100 GPUs. Judge trains the submitted recipes in parallel, one per GPU.
• Research notes and diagnostic tools stay in Work and do not count as submitted policies.

Reference baseline

The dashed 0.378 line is the original Factory reward, measured during task validation: 387 successful terminal episodes out of 1,024. It rewards keypoint alignment, engagement, and successful insertion under the same fixed training and evaluation protocol used for candidates.

This task uses shorter episodes and wider reset randomization than the stock environment. The released reference checkpoint and the paper's results therefore are not comparable baselines. Neither published agent run submitted the original reward unchanged, and Judge does not rerun it alongside every candidate.

Research loop

1. Form a hypothesis about which behavior the reward should encourage or discourage.
2. Express it as a valid reward graph, optionally with a second recipe to compare.
3. Submit the recipes; Judge freshly trains and evaluates each one.
4. Compare the per-recipe success rates, then keep, revise, or replace the idea.

What the agent may change

• Reward terms, their weights, and how they are combined.
• Conditions that switch rewards between phases of the insertion.
• Bounded reward-only memory and how it is updated.
• Which training checkpoint is retained for evaluation.
• An optional alternative reward recipe in the same submission.
• Work-only research notes and diagnostic scripts.

What stays fixed

• The robot, peg and socket, simulator assets, physics, controller, observations, actions, and success condition.
• Episode duration and reset randomization: six-second episodes with the task's wider starting-position and orientation range.
• Policy architecture, PPO configuration, training seed, and fresh initialization.
• The fixed training budget: 50 epochs and 819,200 transitions per recipe; selecting an earlier checkpoint does not reduce the charged budget.
• The allowed reward-graph language, candidate-file limits, hidden evaluation episodes, and scoring rules. Work-side checkpoints are not scoring inputs.

Evaluation

Judge validates each recipe, trains a fresh policy for the complete fixed budget, retains the selected checkpoint, and measures successful terminal insertions / 1,024 evaluation episodes. The score lies between zero and one. With two recipes, the submission score is the better of the two, not their average.

Feedback includes each recipe's aggregate score, completion counts, selected epoch, timing, and bounded diagnostics. Invalid or interrupted submissions have no official score. In particular, the final in-flight round in each run retained candidate metrics but closed without a submission score when the run budget expired; those diagnostics are not substituted for a scored result. A completed evaluation with no successful insertions would instead be a genuine zero.

The Isaac Lab paper explains the underlying Factory tasks; this task's shorter episodes, wider resets, reward-search interface, and fixed evaluation protocol are defined by the linked task instructions.

Agent runs

Each figure shows one agent's scored submissions and running best. Both runs contain 32 submissions, of which 30 have an official score. Open the table beneath each figure for every submission; a dash means no score was recorded.

Links

• Task package (instruction, policy, tests) ↗
• Run logs — trajectories, submissions, judge feedback ↗
• Source project ↗
• Paper ↗