Tabletop real-to-sim

Interactive demo prep

Won · 10 November 2026

A phone scan becomes a scene you can push around. One 26-second iPhone video of a table, four ways to get its 3D geometry, and the pipeline we kept: 6.4 minutes from capture to a MuJoCo scene for the hand demo.

TL;DR

I recorded with Stray Scanner, which gives LiDAR depth and the camera path (ARKit). VGGT-Omega turned out more accurate in shape than the LiDAR-based routes, but it has no real-world scale. The phone's camera path gives it the right scale; the LiDAR depth is only used to check the result.
The biggest gain was not the depth source but the object finder: Claude's object names + SAM 3 instead of automatic SAM 2 masks filtered by depth.
The pipeline so far: VGGT-Omega for geometry → Claude names the objects → SAM 3 masks them → SAM 3D shape and pose → SAM 3D meshes → export to MuJoCo.
Learned alternatives did not help. AnySplat renders poorly. WorldMirror 2.0 (HY-World 2.0) may not be used in the EU, the UK or South Korea under its licence, so it was not tried. Depth Anything 3 with the ARKit cameras as input is no better than VGGT-Omega.
Next: I will test more scenes with this new pipeline.

The input

Left: the video. Right: the phone's LiDAR depth, 256 × 192 pixels. The Stray Scanner app also records where the camera was in every frame (ARKit). A plate, a spoon, a mug and a jar.

Four ways to get the geometry

Same 32 frames, same object models after this step. Each scene is dropped into the simulator and left to settle. The tan table is the same size in every video. Videos 1–3 used the first, automatic object masks, which cut the spoon into a handle and a head; video 4 is the final pipeline.

1 · VGGT-Omega only

wrong size

Objects: automatic detection (SAM 2 + depth test)

Scale vs phone: ×1.72
Plate width: 33 cm
Models → scene: 6:18

Clean, upright shapes. A video alone cannot tell how big the table is.

2 · Fused LiDAR depth + ARKit cameras

objects eroded

Objects: automatic detection (SAM 2 + depth test)

Scale vs phone: metric
Plate width: 21.5 cm
Models → scene: 5:11

Merging all depth frames into one volume wears thin objects away. The mug and jar come out misshapen and tip over.

3 · Per-frame LiDAR depth + ARKit cameras

objects merged

Objects: automatic detection (SAM 2 + depth test)

Scale vs phone: metric
Plate width: 21 × 30 cm
Models → scene: 8:05

Right size, but the low-resolution depth glues the spoon to the plate and finds placemats as objects.

4 · VGGT-Omega + ARKit cameras

kept

Objects: Claude names + SAM 3

Scale vs phone: metric
Plate width: 22.4 cm
Whole run: 6:25

VGGT-Omega's shapes at the scale of the phone's camera path. No LiDAR needed. Shown as the final pipeline builds it: objects found by Claude + SAM 3, each resized to fit its masks.

Raw LiDAR depth beside fused depth for two frames — Why method 2 fails: raw LiDAR (second column) still shows the spoon handle; after fusion (third column) it is gone.

The main finding

Finding the objects mattered more than the depth source

Methods 1–4 all used the first object finder: SAM 2 proposes masks with no text, and a mask is kept if the depth says something stands on the table. It breaks in a different way with each depth source. Replacing it with Claude's object names + SAM 3 fixed the spoon, the placemats and the count in one step.

SAM 2 + LiDAR depth

10 found

Ten things found with per-frame LiDAR depth

Each tile tints one thing it kept. Placemats and chairs pass the test because 256 × 192 LiDAR is noisy at edges. Plate and spoon are one object.

SAM 2 + VGGT-Omega depth

5 found

No placemats, but the spoon is cut into a handle and a head. That is the odd spoon in video 1.

Claude names + SAM 3

4 found

Four objects found by SAM 3 from their names

Claude names what is on the table; SAM 3 masks exactly those names in every frame. Four objects, each whole.

What we kept

Method 4 for geometry, then three fixes that mattered more than the depth source: object names from Claude, masks from SAM 3, and a size fit to the masks.

The table scene. The spoon stays in the plate. Plate 22.4 cm, mug 9.8 cm tall, jar 12.3 cm tall.

A second capture: a pumpkin bucket and two duck toys on the floor.

Six frames sent to Claude — Claude sees six frames and answers: white plate, red spoon, white mug, vitamin jar.

Gaussian splat made from the point maps — The same point maps written as 3D Gaussians, in 6 seconds. Holes where nothing was seen.

Where the 6.4 minutes go

Frames, names, model loading

42 s VGGT-Omega + ARKit scale

8 s SAM 3 masks

15 s SAM 3D shape and pose

13 s SAM 3D meshes

138 s Texture from the video

49 s Size fit to masks

32 s MuJoCo collision shapes

86 s

GPU (RTX 5090)mostly CPU4 objects. One object: 4.6 min.

Tried and set aside

AnySplat: splats in one pass

dropped

Photo, AnySplat render, LiDAR depth, AnySplat depth

Time: 2–4 s
Depth vs LiDAR: 3.4 cm
Render quality: 15–16 dB

Rows: photo, AnySplat seen from the phone's camera, LiDAR depth, AnySplat depth. Streaky renders, and it cannot take the phone's cameras as input.

Depth Anything 3 with the phone's cameras

no gain

Depth and error maps for DA3 and VGGT-Omega against LiDAR

DA3 + cameras: 0.82 cm
DA3 alone: 0.99 cm
VGGT-Omega: 0.38 cm

Shape error against LiDAR after removing a common 4 % scale offset. Cameras help DA3, but VGGT-Omega without cameras is already more accurate. DA3 Large, 16 frames, one scene.

Splat after fitting to the photos — Fitting the splat to the photos (gsplat, 76 s) lifts unseen-view quality from 15–17 dB to 20–22 dB. Left out of the default run because of the extra time.

Run it

# Record with Stray Scanner (iPhone or iPad), copy the capture folder over, then:
python sm120.py fast_scene.py NAME CAPTURE_DIR

# options
  --objects claude                   # default: Claude names the objects, SAM 3 masks them
  --objects auto                     # no names: SAM 2 proposals filtered by depth (splits thin objects, keeps placemats)
  --prompts "plate,mug,toy duck*2"   # name the objects yourself instead of asking Claude
  --set-down                         # put every object flat on the table
  --no-export                        # stop before the MuJoCo scene

# the scene lands in ~/recon_scenes/NAME; in the hand demo:
uv run python scripts/live.py --world mujoco --scene NAME

Step	Script	Time
Frames and phone cameras	`phone_capture.py`	11 s
Geometry at phone scale	`run_video.py` (VGGT-Omega)	8 s
Object names	`caption_objects.py` (Claude)	6–16 s
Masks	`masks_sam3.py`	15 s
Object meshes	`run_fast.py`, `run_stage2.py` (SAM 3D)	151 s
Texture and size	`bake_texture.py`, `fit_silhouette.py`	81 s
MuJoCo scene, real table texture	`export_mujoco.py`, `surface_texture.py`	91 s
Splats	`scene_splat.py`	6 s

Code: experiments repo, projects/scene-reconstruction/2026-10-15_tabletop-recon, branch won/phone-depth. Needs one 32 GB GPU. Mass and friction of the objects are guesses.