Magic Mirror — 60 FPS Offline AR Virtual Trial Room & Smart Retail Kiosk
An AI-powered AR virtual trial room and smart retail mirror rendering clothes, eyewear, and accessories in real-time at 60 FPS offline using MediaPipe, Three.js PBR, and canonical 3D garment templates.
Overview
Magic Mirror is an AI-powered AR virtual trial room and smart retail mirror engineered to eliminate in-store fitting queues and reduce fashion return rates. Running on edge kiosk hardware (Android TV, tablets, and interactive retail mirrors), the system tracks customers' faces and body postures to render apparel, eyewear, and jewelry in real time at 60 FPS completely offline with zero cloud streaming lag.
The ecosystem combines an edge-optimized WebGL/Three.js rendering engine with a merchant mobile photo ingestion portal that converts standard 2D flat-lay garment photos into rigged 3D PBR assets in under two minutes.
Technical Valuation: Replacement Cost: ₹52.0 Lakhs | Complexity Score: 230 FP. 60 FPS on-device computer vision, 468-point facial landmarking, 33-point skeletal pose tracking, and real-time depth occlusion.
Architectural Design: Hybrid Edge-Cloud
To deliver smooth 60 FPS rendering without burning bandwidth or failing when in-store Wi-Fi drops, Magic Mirror splits computation into an offline edge client and an automated merchant cloud ingestion pipeline:
┌─────────────────────────────────────────────────────────────┐
│ MERCHANT CLOUD BACKEND │
│ │
│ [Vendor 2D Photos] ──► [BiRefNet Matting] │
│ │ │
│ ▼ │
│ [Canonical 3D Garment Template] │
│ │ │
│ ▼ │
│ [UV Texture & Normal Bake] │
│ │ │
│ ▼ │
│ Rigged .GLB Asset (<3MB) │
└─────────────────────────────┬───────────────────────────────┘
│ Sync via Local Cache / CDN
▼
┌─────────────────────────────────────────────────────────────┐
│ IN-STORE SMART MIRROR RUNTIME (OFFLINE) │
│ │
│ [Camera Stream] (1080p 60fps Front / Mirror Mode) │
│ │ │
│ ▼ │
│ [MediaPipe Landmarker] (Face Mesh + 33 Body Keypoints) │
│ │ │
│ ▼ │
│ [Dynamic Calibration] (User Body Calibration & Smoothing) │
│ │ │
│ ▼ │
│ [Three.js / Filament 3D PBR WebGL Engine] │
│ │ │
│ ▼ │
│ [Full Screen Display @ 60 FPS - Zero Latency] │
└─────────────────────────────────────────────────────────────┘
Engineering Highlights
1. Zero-Latency Edge Computer Vision
The kiosk uses Google MediaPipe FaceMesh (468 landmarks) and Pose Landmarker (33 body keypoints) executing locally via WebAssembly and GPU shaders:
- Eyewear Tracking: Anchored precisely to Landmark 168 (Nose Bridge) with 6-DOF roll/pitch/yaw calculated from Eye Corners (33, 263) and Nose Tip (1).
- Garment Deformation: Anchored to Left/Right Shoulders (11, 12), Hips (23, 24), and Neck (0). Skeletal bones dynamically stretch and rotate to match the user's stance.
// ar-pose-tracker.ts
export function align3DGarmentToSkeleton(
garmentMesh: THREE.SkinnedMesh,
poseLandmarks: NormalizedLandmarkList
) {
const leftShoulder = poseLandmarks[11];
const rightShoulder = poseLandmarks[12];
const midSpine = {
x: (leftShoulder.x + rightShoulder.x) / 2,
y: (leftShoulder.y + rightShoulder.y) / 2,
z: (leftShoulder.z + rightShoulder.z) / 2,
};
// Compute shoulder tilt angle
const deltaY = rightShoulder.y - leftShoulder.y;
const deltaX = rightShoulder.x - leftShoulder.x;
const roll = Math.atan2(deltaY, deltaX);
garmentMesh.position.set(midSpine.x, midSpine.y, midSpine.z);
garmentMesh.rotation.z = roll;
// Apply One-Euro filter to prevent landmark jitter
smoothBoneTransforms(garmentMesh.skeleton);
}
2. Physically Based Rendering (PBR) Materials
To achieve lifelike fabric appearance without heavy offline ray-tracing:
- Normal Maps: Synthesize fabric weave, cotton textures, and denim micro-wrinkles in real-time.
- Roughness / Metallic Channels: Distinguish soft cotton, reflective leather, and metallic zipper accents.
- Clearcoat Optics: Adds realistic glass reflections and prescription lens tints to eyewear models.
3. Real-Time Depth Occlusion
To solve the classic AR problem where virtual clothes appear unnaturally floating over customer hands, Magic Mirror feeds the MediaPipe Selfie Segmentation mask into the depth buffer. When a user raises their hands or checks their pockets, their real limbs naturally occlude the 3D rendered fabric.
4. Frictionless Merchant Portal & QR Checkout
- 2-Minute Ingestion: Retail merchants snap front and back photos of garments; the pipeline runs BiRefNet background matting and wraps the texture onto canonical pre-rigged templates (T-Shirts, Hoodies, Blazers).
- Instant Buy QR: Customers scan an on-screen QR code to immediately purchase their selected fit via UPI or add it to their in-store bag.
Request Kiosk Demo: Email Sarthak