Inside Magic Mirror: Achieving 60 FPS Offline AR Virtual Try-On with MediaPipe, Three.js PBR, and Automated 2D-to-3D Ingestion

Physical retail apparel stores lose millions of dollars every year to two pervasive problems: massive queues outside changing rooms during peak hours, and high return rates caused by sizing mismatches.

While virtual trial room apps have existed for years, almost all of them suffer from high latency, requiring video streams to be sent back and forth to expensive GPU cloud servers. When in-store Wi-Fi fluctuates, the frame rate stutters, completely breaking the illusion.

We engineered Magic Mirror to solve this at the hardware and software level: an in-store smart mirror kiosk that runs 100% offline at 60 FPS, paired with a cloud ingestion pipeline that turns 2D merchant photos into rigged 3D assets in under two minutes.

1. The Edge-Only Constraint: 60 FPS Offline

To make an AR mirror feel like a real optical reflection, motion-to-photon latency must stay strictly below 16 milliseconds (60 FPS). Any perceptual lag induces nausea and ruins the shopping experience.

Sending 1080p camera frames over the cloud introduces at least 150-400ms of roundtrip latency. Therefore, Magic Mirror enforces an Edge-Only Runtime Architecture:

[HD Front Camera Stream] 
       │
       ▼ (1080p @ 60 FPS)
[MediaPipe WebAssembly Engine] 
       │ ➔ 468 Facial Landmarks + 33 Skeletal Keypoints
       ▼
[Dynamic Kalman / 1-Euro Filter Smoothing] 
       │ ➔ Eliminates landmark jitter
       ▼
[Three.js / WebGL PBR Skinned Mesh Pipeline] 
       │ ➔ Garment bones deformed to match user pose
       ▼
[Hardware Display (Android TV / Expo)]

All computer vision inference and 3D rendering happens locally on the kiosk processor (e.g. Rockchip RK3588, Apple Silicon, or Intel Core i5 edge boxes).

2. Landmark Anchoring & Skeletal Deformation

Accurately fitting garments and eyewear onto moving humans requires multi-layered geometric anchoring:

Eyewear Anchoring (6 Degrees of Freedom)

Eyewear models are anchored to Landmark 168 (Nose Bridge):

• Roll, pitch, and yaw are derived in real-time from Eye Corners (Landmarks 33 and 263) and the Nose Tip (Landmark 1).
• Clearcoat shader properties simulate the optical glass reflections and subtle refractive distortion of prescription eyewear.

Torso Apparel Deformation

Shirts, jackets, and hoodies are anchored across:

• Left Shoulder (11) and Right Shoulder (12)
• Left Hip (23) and Right Hip (24)
• Neck Base (0)

// ar-engine/pose-deformer.ts
export function updateGarmentBones(skeleton: THREE.Skeleton, landmarks: Keypoint[]) {
  const leftShoulder = landmarks[11];
  const rightShoulder = landmarks[12];
  const leftHip = landmarks[23];
  const rightHip = landmarks[24];

  // Calculate spine vector and torso tilt
  const torsoWidth = Math.hypot(
    rightShoulder.x - leftShoulder.x, 
    rightShoulder.y - leftShoulder.y
  );

  // Dynamic mesh scale factor to match customer chest width
  const scale = torsoWidth / REFERENCE_TORSO_WIDTH;
  skeleton.bones[0].scale.set(scale, scale, scale);

  // Compute roll and yaw rotations
  const rollAngle = Math.atan2(rightShoulder.y - leftShoulder.y, rightShoulder.x - leftShoulder.x);
  skeleton.bones[0].rotation.z = rollAngle;
}

3. Real-Time Depth Occlusion via Selfie Segmentation

A notorious flaw in mobile AR clothing apps is that virtual clothes render on top of the user's hands. If a shopper raises their hand to adjust their hair or checks their phone, the 3D shirt awkwardly draws over their fingers.

Magic Mirror fixes this using Selfie Segmentation Depth Buffers:

1. MediaPipe Selfie Segmentation runs concurrently with the pose landmarker.
2. The user's arm and hand silhouettes are rendered to the WebGL depth buffer as invisible occlusion geometries with color writing disabled (colorWrite = false; depthWrite = true;).
3. When the 3D shirt is rendered, pixels behind the user's physical hands are automatically culled by the GPU hardware depth test.

4. The 2-Minute Merchant Ingestion Pipeline

Retail merchants cannot spend ₹15,000 per garment hiring 3D Blender artists to hand-model every SKU in their inventory.

To automate this, Magic Mirror utilizes a library of Canonical Pre-Rigged Garment Templates (Crewneck, Polo, Hoodie, Blazer, Denim Jacket):

1. Vendor Photo Ingestion: The merchant snaps flat-lay front and back photos using our mobile web portal.
2. AI Matting & Normal Generation: A cloud microservice running BiRefNet strips the background, while a Sobel-filter normal shader generates micro-wrinkle height maps.
3. Automated UV Projection: The textures are baked onto the canonical 3D template's UV map and exported as a lightweight, Draco-compressed .glb asset under 3MB.
4. Local CDN Cache: The in-store mirror downloads and caches the .glb files overnight. During business hours, items load instantly from local SSD storage.

Summary

Magic Mirror demonstrates the power of edge-native computer vision. By combining MediaPipe WebAssembly inference, Three.js physically based materials, and automated 2D-to-3D garment templating, retail shoppers enjoy a 60 FPS real-time fitting experience with zero latency and zero cloud dependence.

• Request Hardware & Software Demo: Contact Sarthak Srivastava
• Built by: Sarthak Srivastava