What happened
Apple Machine Learning presented Luce, a 3D representation designed for high-fidelity image-to-3D generation that captures both geometry and appearance.
The representation combines geometry with physically based rendering (PBR) materials—including albedo, metallic-roughness, and surface normals—to support relighting and integration into standard rendering pipelines.
Luce organizes geometry and PBR materials into a voxelized multimodal Gaussian cloud, where each modality gets its own dedicated Gaussian primitives, and a variational autoencoder compresses it into a unified material-aware latent space.
Why it matters
By incorporating PBR materials directly into the 3D representation, Luce aims to make generated assets compatible with standard rendering pipelines, which could make image-to-3D outputs more usable in real-world workflows that require relighting.
The unified material-aware latent space suggests a path toward generating geometry and appearance together rather than as separate, less coherent components.
The use of multimodal Gaussian primitives is a step toward representations that are both expressive enough for detailed assets and structured enough for efficient generation.
Key facts
Luce is a 3D representation that unifies geometry and PBR materials in a voxelized multimodal Gaussian cloud.
Dedicated Gaussian primitives are used for each modality, including albedo, metallic-roughness, and surface normals.
A variational autoencoder compresses the representation into a unified material-aware latent space.
What to watch next
The paper description is partial, so forthcoming details may clarify how Luce is used for actual image-to-3D generation and rendering.
Keep an eye on Apple's machine learning publications for experimental results and comparisons with other 3D generation methods.