# Making this kitchen useful for robot training

The first interaction benchmark is now implemented beside the reference scene. It provides five Franka task setups, reset/scoring code, a sensor interface and a reduced coupled-pouring model. See the [evaluation page](review/benchmark.html) for actual outcomes, including failed trials, and the [benchmark protocol](benchmark/README.md) for reproducibility and limitations. The remaining priorities below concern reliability, real measurements and broader validation.

The current release is a small, reproducible **kitchen interaction benchmark**. Start with one reachable station and one supported arm. Establish reliable observations, actions, reset behavior and scoring before expanding to the whole kitchen or a humanoid.

## First: five tasks with observable outcomes

| Task | What the robot must actually do | What to measure |
| --- | --- | --- |
| Put a lid on a pot | Grasp the handle, align the lid, seat it and release | Final seating/alignment, stability after release, drops, peak contact force |
| Retrieve a refrigerated item | Pull the drawer, pick an item, place it and close the drawer | Object placement, drawer closure, open duration, collisions |
| Load the oven | Open the door by its handle, place a tray on a rack, close it, set controls | Tray support, door closure, achieved control state, collisions and unsafe heat exposure |
| Pour a specified amount | Grasp the jug, pour into an open pan, stop and set the jug down | Delivered mass error, spilled mass, retention after placement and completion time |
| Move a hot pan to a trivet | Select a suitable grasp, lift clear of the burner and place the pan | Support/stability, distance from active heat, gripper thermal exposure and spills |

Use state-based scoring for these measurable outcomes, with video review as an additional check. Task failure should distinguish a policy error from a simulator failure or an impossible reset. Record initial state, seed, model parameters, solver settings, asset hashes, policy version and observations/actions. Keep simulator ground truth available to the evaluator; expose only the chosen robot sensors to the policy.

Begin with scripted/teleoperated demonstrations to establish task feasibility. Then compare the same task with an agentic controller and a suitable learned policy. A language interface does not establish that a pretrained policy understands this robot, camera configuration or action space.

## Make the physical interactions credible

1. **Calibrate a few objects well.** Obtain dimensions and mass for a real jug, lid, pan and tray; measure center of mass and simple slide/tip behavior. Replace the current estimates with recorded measurements and uncertainty ranges. Preserve separate visual and collision meshes, but audit their fit at rims, handles, gripper contacts and rack supports.
2. **Connect controls to contact.** Give knobs angular joints and detents, buttons travel/activation thresholds, and doors/drawers suitable resistance and stops. A robot should change an appliance by moving its control. Keep direct API commands for resets and baselines, with the access mode recorded in the task.
3. **Audit forces and contact stability.** Check slow pushes, stacked dishes, lid seating, grip-slip thresholds, thin tray walls and objects resting on oven racks. Sweep timestep and solver settings to detect penetration, jitter, tunneling and excessive joint-drive forces. Stop scoring an episode when physics becomes invalid.
4. **Measure appliance response.** A real temperature trace and heater duty cycle would constrain power, thermal mass, door losses and food/vessel conductance. Until then, vary documented plausible parameters and describe the model as estimated. Calibrating one oven and one pan is more useful than adding guessed electronics to every visible appliance.

The source video gives strong appearance and layout clues. It does not identify exact friction, mass distribution, hidden internals, thermal response or every dimension. A few known measurements or additional calibrated photographs would reduce uncertainty much more than further unmeasured detail.

## Improve what the robot sees and feels

Add wrist and fixed RGB-D cameras with recorded intrinsics/extrinsics, occlusion, exposure changes, motion blur, depth holes around reflective steel, quantization, latency and frame drops. Add gripper/joint state and force/tactile observations only where the chosen robot has corresponding sensors. A clean renderer depth image should remain available for diagnosis, separate from the policy's sensor model.

Capture aligned RGB, depth, segmentation, contacts and action/state logs. Compare a real and simulated image from matched viewpoints before adjusting lights and materials again. Use the original kitchen layout as the reference version, with seeded variants for clutter, lighting, initial object poses, drawer positions and reasonable dynamics uncertainty. Hold out combinations of these variations for evaluation.

This prioritization is consistent with the actuator, multi-rate sensor, demonstration and randomization tooling described in [Isaac Lab](https://research.nvidia.com/publication/2025-09_isaac-lab-gpu-accelerated-simulation-framework-multi-modal-robot-learning). Contact-focused objectives and curricula have also supported demonstrated transfer in [IndustReal](https://arxiv.org/abs/2305.17110). Neither source validates this kitchen; they inform the proposed workflow.

## Liquids: couple the existing pieces before adding more solvers

The existing GPU jug test is a prescribed kinematic pour of 144.625 ml. The new robot task uses a separate reduced quasi-static/ballistic model that updates native vessel mass, COM and inertia, transfers momentum/enthalpy and exchanges heat with the jug. Its receiving pan must stay approximately upright, and packets do not collide with arbitrary scene geometry. This is not coupled PBD. The next work is controller reliability, model comparison across fills/rates and measured pouring data.

Benchmark several fill levels, pour rates, target positions and grasps. Include misses and overflow. Keep a cheaper approximate liquid mode for large training batches and qualify it against higher-fidelity particle runs; report which mode produced each result. Add wet-surface friction only after specifying how spilled liquid changes contact properties. Do not imply that rendering a wet patch changes physics.

## Where to stop for the next release

Keep the five-task, one-arm boundary. Maintain repeatable resets, explicit failures, sensor/action recordings and 50 held-out initial states per task. Use the reported success intervals, spills, sampled forces, temperatures and throughput to guide improvements. Tasks without successful trials remain experimental. Then compare a small real-world subset with the same physical props. Treat this as an acceptance target, not a promised policy success rate.

Defer full combustion and smoke flow, comprehensive cooking chemistry, arbitrary cutting, deformable food everywhere, a moving human crowd and full-kitchen humanoid locomotion. Each adds substantial modeling and validation work. Add one only when it is necessary for a defined task—for example cloth for wiping, deformable dough for kneading, or a mobile base when fixed-arm reach is the actual limitation.

After the benchmark is stable, broaden to bimanual manipulation and mobile manipulation. Keep the kitchen asset independent of embodiment, and attach robot-specific reach, sensor and controller configurations to each task setup.