Sim-to-real and the Isaac stack

Sim-to-real transfer is not a solved problem. It is the central engineering challenge of every robot learning system in production. The question is not whether your policy trained in simulation will transfer to the real world — it is how large the performance gap will be and whether you can close it before your deadline. Domain randomization (Tobin et al., arXiv:1703.06907, March 20, 2017) is the primary tool: train the policy across a distribution of physics, lighting, and texture parameters so that the real world is just another sample from that distribution. NVIDIA's Isaac Sim, Isaac Lab, and the Newton physics engine (NVIDIA GTC, March 18, 2025) are now the open stack for this workflow. Newton on Warp delivers approximately 70x simulation speedup vs previous stacks. GR00T-Dreams demonstrates what this unlocks: 6,500 hours of synthetic humanoid training data generated in 11 hours from a small real-seed dataset — the direct driver of GR00T N1.5's improvement from 17.4% to 47.5% on the RoboCasa 30-demo benchmark.

Domain randomization — the core mechanism

Domain randomization (Tobin et al., March 2017) addresses the sim-to-real gap by training across a distribution P(xi) of simulator parameters xi rather than a single fixed simulator. If the policy achieves high reward across the full distribution — friction coefficients from 0.3 to 1.2, object masses from 0.8x to 1.5x nominal, 50 different lighting textures and material reflectances — then the specific parameters of the real world, which are one sample from that distribution, should be within the policy's generalization capability. The key engineering question is how wide to make the distribution: too narrow and the real world falls outside; too wide and the task becomes unsolvable (a robot cannot grasp if friction can be arbitrarily low).

Three parameter axes are most commonly randomized for robot manipulation: physics parameters (friction, damping, mass, moment of inertia), visual parameters (texture, lighting, camera extrinsics), and object pose (initial position/orientation distribution, object shape variation). Isaac Lab's configuration system exposes all of these through Python dataclasses — you specify a distribution for each parameter and the training loop samples from it at episode reset. The Franka arm reach task in Isaac Lab has domain randomization pre-configured as a reference implementation.

System identification and real-to-sim-to-real

System identification (system ID) is the complementary approach: instead of randomizing broadly, measure your real robot's actual dynamics and set the simulator to match. Identify friction, damping, motor torque curves, sensor noise from real hardware experiments and fit simulator parameters to minimize the discrepancy between real and simulated trajectories. This narrows the sim-to-real gap at the cost of measurement effort. In practice, most production systems combine both: system ID to center the distribution, domain randomization to widen it appropriately.

The real-to-sim-to-real loop — collect real data, build or calibrate the sim from it, train in sim, deploy back to real — is the pattern that GR00T-Dreams operationalizes at scale. The NVIDIA pipeline takes real demonstration videos, uses video-to-motion retargeting and scene reconstruction (a 3DGS step) to build a sim-ready task specification, generates millions of synthetic rollouts in Isaac Lab with the Newton engine, and uses those rollouts to post-train GR00T. The result: 6,500 hours of synthetic data generated in 11 hours, driving GR00T N1.5's RoboCasa success from 17.4% to 47.5%.

Isaac Lab and Newton — the open stack

Isaac Sim is the photorealistic simulation environment (built on NVIDIA Omniverse). Isaac Lab is the robot learning framework on top of it: task definitions, reward functions, domain randomization APIs, and PPO/SAC training loops. Newton (announced at NVIDIA GTC, March 18, 2025) is a new physics engine built in Warp (NVIDIA's GPU-optimized Python DSL) in partnership with DeepMind and Disney Research — replacing the older PhysX stack for robot ML workloads. Newton delivers approximately 70x simulation throughput improvement, making large-scale domain randomization and population-based training feasible on a single 8-GPU node.

MuJoCo via MJX (JAX-based MuJoCo) is the alternative stack — faster to set up, excellent for research, widely used in academic robot learning. The choice between Isaac Lab (NVIDIA ecosystem, photorealistic, Newton physics) and MuJoCo (lighter, faster iteration, open-source) is primarily driven by whether you need photorealistic rendering and whether you are targeting NVIDIA hardware for deployment. For the JHU humanoid capstone, Isaac Lab is the target for sim training; MuJoCo is acceptable for algorithm development.

Capstone: the simulation tier

This is the simulation tier of the JHU humanoid capstone. Without it, you collect real robot data forever; with it, you can generate thousands of hours of synthetic demonstrations from a small real seed. The build for this chapter — Isaac Lab + PPO + domain randomization on the Franka arm — is the direct precursor to running GR00T N1.5 post-training on synthetic data from GR00T-Dreams. The capstone uses this pipeline to generate the synthetic manipulation data that supplements the 200-episode LeRobot SO-101 real dataset.