Surface relief grating (SRG) waveguides, championed by Microsoft (HoloLens 2), Magic Leap, and XREAL, have become the mainstream approach, achieving 50-70 degree diagonal FOV in sunglasses-like form factors. XREAL's latest Air 2 Ultra pushes 52° FOV in a 52g frame, but struggles with rainbow artifacts and non-uniform brightness across the eye box. Companies like Dispelix are developing proprietary SRG designs that claim improved uniformity, though no commercial device has yet broken 70° FOV in an all-day-wearable form factor without unacceptable bulk.

Birdbath optics remain the fallback for devices prioritizing immediate visual quality over form factor, used by XREAL (Air line) and Lenovo. They deliver 46-50° FOV with good brightness and color but require a chunky front-heavy chassis — a non-starter for all-day wear. Varjo's XR-4 employs a four-lens pancake stack to hit 120° FOV, but at 510g it's a tethered professional device, not a consumer wearable. The trade-off is increasingly untenable as Apple and Meta push consumer expectations toward sub-80g sunglasses.

Holographic optical elements (HOEs) and metasurface gratings represent the frontier, pursued by Mojo Vision (now refocused on microLED displays for others), XPANCEO, and academic labs. The pitch: replace the multi-layer waveguide stack with a single thin film. Mojo demonstrated a 14,000 PPI microLED with HOE combiners, but the FOV remains under 30° — acceptable for a contact-lens display but not for immersive AR. XPANCEO claims progress on metalenses that can steer light dynamically, but no working full-color prototype at binocular FOV over 40° has been shown publicly. The physics of diffraction-limited metalenses at visible wavelengths at scale remains a fundamental material science challenge.

PBP (Pancharatnam-Berry Phase) lenses, pursued by Magic Leap (via their patent portfolio) and several Japanese optics labs, use liquid crystal polarization gratings to achieve thin-film, polarization-dependent focusing. These promise higher efficiency and better color uniformity than SRGs, with fewer rainbow artifacts. Magic Leap 2's PBP-based approach achieved 70° FOV but required a bulky two-layer combiner and active dimming that added weight. No company has yet demonstrated PBP optics below 80g that maintain 70°+ FOV. Samsung is rumored to be exploring PBP for its upcoming consumer AR device, but has shown no prototype.

Camera-based tracking using deep learning has advanced rapidly, driven by Apple (Vision Pro hand tracking via downward-facing IR cameras), Meta (Quest 3/3S), and Ultraleap. Apple's system achieves sub-20ms latency in ideal lighting with 4 cameras, but fails predictably during hand-over-hand occlusion and in dim environments. Meta's Quest 3 uses 6 cameras with a semantic keypoint model trained on synthetic data, but fast gestures like typing or finger-snapping produce visible jitter. Both companies now employ a two-stage pipeline — a neural network for coarse keypoint estimation followed by an inverse kinematics solver — but the gap between 95th and 99th percentile reliability remains the critical barrier to controller-free productivity use.

Thalmic Labs (now part of Google's ATAP) and Meta (via the CTRL-Labs acquisition) are betting that wrist-mounted EMG or ultrasound provides the invariant signal that cameras miss. Meta's EMG wristband, announced alongside the Orion AR glasses prototype, measures motor neuron signals from the wrist to predict finger movements before they happen — effectively sub-millisecond intent detection. Google's ATAP demonstrates similar with Project Soli's radar-on-wrist approach. The catch: EMG requires calibration per user and drifts with sweat and arm position. No wrist-based system has yet matched vision's spatial tracking accuracy for 3D pointing (e.g., selecting a small button in space). The bet is that multimodal fusion — EMG for intent, cameras for spatial position — will converge, but no production device has shipped this fusion yet.

A newer approach, pioneered by research labs at Meta AI and Google DeepMind, replaces hand-crafted gesture classifiers with large foundation models trained on massive multi-modal datasets (video, synthetic, IMU). Meta's SAM-based models for hand segmentation in the wild achieve 98% IoU on standard benchmarks but at inference costs too high for a 3W XR processor. Google's MediaPipe Holistic remains the most deployed solution, running at 30 FPS on smartphones, but its gesture recognition is limited to a fixed vocabulary of ~30 poses. The frontier challenge is semantic understanding — distinguishing a 'point' from a 'poke' or a 'swipe' from a 'wave' — which foundation models handle well in research but not yet at the sub-5ms inference latency required for presence-critical XR. Snap's latest Spectacles use a distilled transformer model that claims 5W total system power for a 26-gesture vocabulary.

Every major shipping device — Apple Vision Pro (2 hours with external battery pack), Meta Quest 3 (2.2 hours), Snap Spectacles 5 (45 minutes) — punts the battery problem to an external pack or accepts runtime far below user expectations. Apple's approach, a tethered battery worn on the hip, is the industry's pragmatic compromise: the pack weighs 350g, nearly as much as the headset itself. XREAL's Beam Pro offers a similar compute puck. No company has shipped a self-contained all-day wearable. The workaround buys time but doesn't solve the root physics problem, and user reviews consistently cite runtime as the #1 barrier to daily adoption.

Qualcomm's XR2 Gen 3 and Apple's R1 coprocessor represent the silicon approach: dedicated spatial compute pipelines that draw 40-60% less power than general-purpose SoCs for the same workload. The XR2 Gen 3, used in Meta Quest 3 and Samsung's upcoming device, includes a dedicated depth engine and display processor that offloads work from the CPU. Apple's R1 chip processes camera and sensor data at 12ms latency using only 1.3W. NVIDIA's upcoming Thor platform targets 200 TOPS at sub-15W for spatial AI. These are iterative gains — Moore's Law slowdown means we're getting 1.5x efficiency every 3 years, not the 10x needed for all-day wear. More radical approaches like analog compute-in-memory for neural network workloads remain at the lab stage.

Solid-state batteries, pursued by TDK, Samsung SDI, and QuantumScape, promise 2-3x energy density over lithium-ion — enough for 6+ hours in a glasses form factor. TDK announced a solid-state cell in 2024 with 1,000 Wh/L, targeting wearable applications. But manufacturing at scale is 3-5 years out, and headset-specific form factors (thin, curved, safe near the eye) add constraints. Micro fuel cells (proposed by Intelligent Energy) and wireless resonant charging (used by Mojo Vision's charging case) are longer-shot alternatives. Even Realities and Vuzix ship glasses today with tiny batteries (150-400 mAh) that last 4-8 hours but can only power monochrome low-resolution displays — a reminder that battery tech, not display tech, is the true governor of AR capabilities.

NeRF and its successor, 3D Gaussian Splatting, have revolutionized offline scene reconstruction but remain too computationally expensive for real-time on-device use. Luma AI and KIRI Engine have built consumer apps that reconstruct scenes from phone video in minutes, but converting a Gaussian splat to a mesh or semantic map that an XR device can use in real time is unsolved. Niantic Spatial's Visual Positioning System (VPS) uses learned embeddings of ground-level imagery to achieve 1-meter global localization, but only in mapped cities. The gap between impressive demos (a Gaussian splat of your living room) and a live, updatable semantic map that a device can query at 60 Hz remains vast. Instant NGP approaches from NVIDIA have cut training time to seconds but still require cloud-grade GPUs.

Google's ScaNN, Meta's SAM with spatial extensions, and Apple's RoomPlan API represent a push toward embedding scene understanding into the device OS. Apple's RoomPlan uses LiDAR and the R1 chip to produce real-time floor plans with room dimensions, but it only classifies ~20 furniture categories and has no understanding of material properties (glass, mirror, fabric). Meta's Scene Model on Quest 3 uses depth sensor data and a lightweight neural network to label walls, floor, ceiling, and furniture in 3D, but misclassifies ~15% of surfaces in real-world testing. The core challenge is that transformer-based vision models require 5-20W of inference power at current efficiency levels, far above what a glasses-class device can spare. Distillation and quantization of models like DINOv2 for 1W TDP is the active research frontier.

An emerging architecture offloads heavy scene understanding to the cloud while the device maintains a lightweight local mesh. Magic Leap's Cloud Anchors and Google's Geospatial API both enable users to upload a scan and retrieve persistent anchors across sessions. Snap's Scaniverse (acquired 2023) lets users capture high-quality 3D scans on a phone and view them on Spectacles. The latency cost — typically 500ms-2s for a full scene upload and response — makes this useless for real-time occlusion or physics, but viable for persistent content placement (e.g., a digital note stays on the same real-world desk a week later). PTC's Vuforia and PTC's Spatial Toolbox use cloud-based marker recognition for industrial scenarios where the environment is known in advance. The tension: cloud dependence breaks in the field scenarios (construction sites, remote areas) where AR is most valuable.

ORB-SLAM3 and Apple's ARKit 6-based persistence use keyframe-based visual-inertial SLAM to store a map of the environment and relocalize on return. Apple's ARKit 6 can anchor objects that persist across sessions in the same room using location-triggered map merging, but the anchors drift by 3-10 cm after the device is powered off and restarted. Meta's Spatial Anchors on Quest 3 use similar keypoint matching but degrade significantly when lighting changes (day to night) or when furniture is moved. Microsoft's Azure Spatial Anchors provide cloud-backed relocalization but require the room to be re-indexed on each visit, taking 5-15 seconds. The fundamental issue is that visual SLAM does not handle large-scale scene changes (a new sofa, moved bookshelf) without full re-mapping — which defeats the purpose of persistence.

Real-time occlusion — rendering virtual objects behind real ones — requires dense depth at 60+ FPS with sub-5cm accuracy. Apple Vision Pro uses its 12 cameras and LiDAR scanner to generate a depth mesh at 60 Hz, achieving the best occlusion of any shipping device. But the mesh has visible latency (~80ms) on fast-moving objects like hands, causing virtual objects to 'jump' in front of real hands briefly. Meta Quest 3's occlusion is worse — it uses a single depth projector that only covers a 10m² area and cannot resolve thin objects (chair legs, plants). Magic Leap 2's active dimming avoids occlusion entirely by darkening real-world pixels behind virtual objects — a clever workaround that only works in well-lit rooms and makes the real world look conspicuously dim. The true solution — real-time neural depth reconstruction at 120 Hz with no latency — requires sensor compute not yet available in any mobile form factor.

A newer research direction, pursued by Meta Reality Labs and ETH Zurich, uses learned scene priors to fill in depth where sensors have gaps. The approach: train a neural network on millions of scenes to predict object boundaries and depth contours even when the depth sensor sees only a fragment. Meta's recent papers on semantic occlusion boundary detection achieve 3cm accuracy at object edges — a 2x improvement over raw sensor depth — but only in offline evaluation. The challenge is inference speed: these models require 100+ million parameters and run at 5-10 FPS on current XR chipsets. Snap's Spectacles 5 uses a distilled model that runs at 30 FPS but only handles simple occlusion planes (walls, floors), not complex objects like hands or plants. The convergence path is clear — on-device neural depth completion — but the compute budget of sub-2W for the depth pipeline means we're likely 2-3 hardware generations away.

Gaming engines — Unity and Epic Games (Unreal Engine) — have driven the state of the art for deterministic multiplayer simulation. For spatial apps built on these engines, the standard approach is client-side prediction with server reconciliation: each device simulates the physics locally, the server arbitrates conflicts, and the client corrects when the server disagrees. This works fine for 2-4 players in the same room on a shared 5 GHz WiFi (Meta Horizon Worlds, Sandbox VR's in-location experiences). But it breaks at 8+ participants or over cellular — prediction errors compound, rubber-banding becomes visible, and spatial anchors drift relative to each client's local coordinate system. The core unsolved problem is 'spatial consensus': two devices in the same room can agree on where a virtual cube is relative to each other, but not relative to the real-world shared anchor point, because each device's SLAM solution diverges independently.

A standards-driven approach via W3C and the OpenXR consortium aims to solve anchor consensus at the protocol level. The WebXR Anchors specification defines how anchors are shared across devices, but implementations (Google Chrome, Samsung Internet, Apple Safari) do not support cross-device anchor sharing at all — each device computes anchors independently. Niantic's Spatial Anchors and Magic Leap's Shared Anchors use a cloud arbiter: the first device uploads its anchor graph, other devices download it and attempt to relocalize. This works for static content (a virtual sculpture on a real table) but not for dynamic content shared between moving users. Custom solutions like Spatial.io's platform for virtual worlds bypass real-world anchoring entirely by using a fully virtual coordinate system, which defeats the purpose of shared spatial AR. The Multiplayer XR specification from the Khronos Group (still in draft as of early 2025) proposes shared coordinate systems via UWB and Bluetooth RTT ranging for sub-10cm relative pose, but no shipping product supports it.

NVIDIA's CloudXR and AWS's Wavelength deploy GPU compute at the network edge to reduce round-trip latency. The concept: run the physics and rendering on a server 5-10ms away, streaming frames to thin client headsets. This works spectacularly for single-user streaming (NVIDIA's demo at GDC 2024 showed 45ms end-to-end latency over 5G) but introduces the 'many eyes, one server' problem — two users in the same room hitting the same edge node must synchronize state through a server that has no understanding of their physical co-location. HTC's Vive XR Elite and Pimax's CloudXR support both use this architecture for enterprise training (Moth+Flame, Strivr, Virti), where all users are on the same managed network. For consumer use over the open internet, edge compute doesn't solve the anchor divergence problem — it just makes the simulation faster while the anchors still drift relative to each other. The frontier is 'distributed consensus for spatial anchors' — essentially a blockchain-like protocol but for 3D coordinates — which no major lab has convincingly demonstrated.

Meta's Codec Avatars represent the most advanced research in this space: a neural network encodes a user's face from camera streams, transmits a latent code (~1KB per frame), and decodes it to a photorealistic avatar on the receiving HMD. Meta has demonstrated avatars that capture micro-expressions, eye gaze, and subtle mouth movements, running at 30 Hz on a desktop GPU. The problem for real-time headset use is two-fold: the encoder requires high-quality depth cameras (not available on any shipping HMD), and the decoder requires ~50 GFLOPS per frame — too much for the XR2 Gen 2 chipset. Apple's Persona on Vision Pro uses a simpler approach — a textured 3D mesh generated from the front-facing sensors — which produces the uncanny-valley effect (skin looks waxy, eyes appear glassy). Neither approach works for eyewear-form-factor devices that lack outward-facing full-face cameras.

3D Gaussian Splatting has emerged as a compelling alternative for avatar rendering, pursued by Luma AI (via its avatar platform), KIRI Engine, and academic labs at UMD and TU Munich. The approach: learn a set of 3D Gaussian primitives that encode a person's appearance from multi-view video, then render them via rasterization at real-time rates. Recent papers achieve 60 FPS avatar rendering on an NVIDIA RTX 4090 with quality approaching Codec Avatars but at 1/10th the compute cost. The catch: the Gaussian splats must be pre-computed from studio-grade multi-camera rigs — they cannot yet be generated from consumer sensors in real time. Luma AI's current product streams cloud-rendered Gaussian avatars to phones but adds 200-300ms of latency, breaking conversational turn-taking. The frontier is on-device Gaussian avatar reconstruction from a single camera — a 3D Gaussian Head NeRF that trains in seconds rather than hours — which remains an open research problem.

The classical approach — reconstructing a 3D face mesh from a single RGB image using a parametric model (FLAME, DECA, MICA) — has seen a renaissance thanks to large-scale training data. Epic Games' MetaHuman Animator can produce a high-quality digital double from an iPhone face scan in minutes, and then drive it in real-time via face capture. Snap's Avatar system uses a lightweight 3D morphable model that runs entirely on-device on Spectacles, but the output is stylized (cartoonish), not photorealistic. The trade-off is fundamental: parametric models have ~10^4 parameters and run at 120 FPS on a phone, but cannot capture wrinkles, skin pores, or subsurface scattering. Neural implicit models have ~10^8 parameters and capture photorealism but run at 5 FPS. Magic Leap and NVIDIA are collaborating on 'pruned' neural face models that claim 30 FPS on next-generation AR chipsets by 2026, but no shipping silicon supports them. The solve likely requires a dedicated neural face coprocessor — an NPU designed specifically for a single task — similar to what Apple's R1 chip does for sensor fusion.