Weird Lens Allows Light Field Passthrough for VR Headset [Hackaday]

View Article on Hackaday

Light Fields are a subtle but critical element to making 3D video look “real”, and it has little to do with either resolution or field of view. Meta (formerly Facebook) provides a look at a prototype VR headset that provides light field passthrough video to the user for a more realistic view of their surroundings, and it uses a nifty lens and aperture combination to make it happen.

As humans move our eyes (or our heads, for that matter) to take in a scene, we see things from slightly different perspectives in the process. These differences are important cues for our brains to interpret our world. But when cameras capture a scene, they capture it as a flat plane, which is different in a number of important ways from the manner in which our eyes work. A big reason stereoscopic 3D video doesn’t actually look particularly real is because the information it presents lacks these subtleties.

How is this connected to VR headsets? The video passthrough feature of VR — where one sees the real world via external cameras — is increasingly understood to be an important feature, but has limitations. Visual distortions from software processing are one, but video passthrough also suffers from the same issues that 3D videos have: they just don’t look real, and they don’t actually look 3D. This is more than just a cosmetic problem; it gets in the way of interacting with the world. That includes not just handling items but also things like walking around without bumping into corners, or going down stairs rather faster than one intended.

Light fields are the missing link to making 3D video captured by cameras look more real, and one way to capture light fields is to glue up a whole bunch of cameras. Each camera captures a scene from a slightly different perspective, and software can process the resulting data into a light field video that manages to confer all (or at least most) of the little details our brains are expecting to see.

Meta’s light field passthrough prototype (the “Flamera” headset, about halfway down that page) takes the clever approach of using a lens array combined with apertures to create an optic that modifies how a camera sees the world, instead of using an array of cameras and processing the results. The optic looks like a sort of compound eye, allowing the headset to deliver light field passthrough video that is of remarkably higher quality than the usual options.

Want to know more about light fields? We’ve seen fascinating work from Google on light field video as well as a past Hackaday Superconference talk that does a great job of explaining why it’s so important, and how light fields can be approached even as a hobbyist.