Not long ago I published my thoughts on how I think Magic Leap will work based on a number of sources but primarily on this patent. In the previous article, I make some bold claims about resolution and field of view that, while based on information from that patent, is largely speculative. To be honest, the whole article is largely speculative considering patents do not equal products. As one reader put it, I built my description of the technology “by reading patent applications and other tea leaves”. While admittedly flawed, these documents are the best information we have to make guesses at the coming products out of Magic Leap so I’m diving in again to see if I can put more weight behind some of these claims.
Many readers took issue with my claim of Magic Leap providing a 90° field of view and rightly so. It is the most speculative statement in a speculative article and there is a good chance it is grossly wrong. It is particularly hard to reconcile with the previous quote stating that it would require 50 Mpx to produce a high field of view display. 50 Mpx is absurdly large and is far beyond anything we are capable of today. So how can I stand by my statement? By reading more patents of course.
Why is field of view hard?
This patent describes in further detail many different configurations Magic Leap might employ to create their head mounted display. In particular it has lengthy sections on the difficulties of chromatic aberration, exit pupil and, you guessed it, field of view.
Why is field of view so challenging? This Magic Leap patent gives us some insight into the problems involved. From the patent: “In a conventional waveguide approach wherein total internal reflection is relied upon for light propagation, the field of view is restricted because there is only a certain angular range of rays propagating through the waveguide (others may escape out)”. So light bounces around in a waveguide at a low angle (think of a glancing blow off of the edge of the glass) and when it achieves a high angle it escapes and we can see it. Towards the edge of the waveguide you need a low angle to be able to aim the light at the eye. That low angle will cause reflection and not the desired transmission so it cannot escape and cannot contribute to the field of view.
Given this fundamental constraint, increasing field of view is challenging but possible. The patent outlines a number of methods to increase field of view and I honestly don’t fully comprehend most of them. I will try to relate what I can decipher and give you hints at the other solutions but I don’t think I can give you a complete description.
To overcome the issue of field of view, Magic Leap appears to be employing a tiling approach. The patent mentions many methods which may be used to tile images to generate a larger field of view. From the patent, “a narrow field of view sub-image which is tiled with other narrow field of view sub-images presented by the other reflective surfaces to form a composite wide field of view image” and “wherein 6 sub-images are to be presented to the eye frame-sequential to form a large tiled image”. Keep that number 6 in mind for later as I think it is telling. The methods for tiling range from using a number of reflective surfaces that can be turned on and off (similar to the DOEs explained in my previous article) to using “freeform” optics in ways that I can’t fully comprehend. Tiling is conceptually quite friendly as you can imagine using 6 different scanning projectors to provide 6 different sections of the images that take different optical paths to your eye allowing an expansion of field of view.
At no time since the science of optical system development was created in 1885 has the industry been so technologically dynamic.
The patent has an entire section on the use of freeform optics. Freeform optics appear to be any sort of optical interface that does not conform to simple shapes such as spheres, parabolas or cylinders. While this sounds fairly simple it means the surface geometry must be carefully calculated and manufactured. Optics people seem very excited about the prospects of freeform optics stating “At no time since the science of optical system development was created in 1885 has the industry been so technologically dynamic.”. For Magic Leap, this means they can produce specially constructed lens arrangements to generate a larger field of view. As the patent states: “With freeform lenses, rather than having a standard planar reflector, one has a combined reflector and lens with power, which means that the curved lens geometry determines the field of view … it is possible for a freeform arrangement to realize a significantly larger field of view for a given set of optical requirements”. It appears that these optics don’t make use of total internal reflection (or at least don’t use it exclusively) so they can avoid the fundamental limitation discussed above.
The patent goes on to indicate a combination of tiling and freeform optics may be used. In particular, it references 2 different tiles but goes on to show an image with 6 potential freeform wedges driven by 6 different displays.
Unfortunately, the science of freeform optics appears to be complex and hard to digest. The surface structure is defined by numerically solving complex differentials and then fitting high order polynomials to the solution. I wish I understood the concepts more thoroughly but regardless of how it works it does appear to be causing a revolution in the optics space and will give Magic Leap a much needed tool in generating a wide field of view.
Okay, time to wildly speculate
In a previous patent we get this quote: “To best match the capabilities of the average human visual system, an HMD should provide 20/20 visual acuity over a 40° by 40° FOV, so at an angular resolution of 50 arc-seconds this equates to about 8 megapixels (Mpx). To increase this to a desired 120° by 80° FOV would require nearly 50 Mpx”. The patent then ignores this statement and goes on to describe a system that provides 8 Mpx across a 40° field of view. This has always seemed odd to me. The start of the patent explicitly says that a small field of view is bad and that a desired field of view is 120° by 80° with nearly 50 Mpx. Why say this? Why pick those particular numbers especially if you can’t achieve them?
I think there is a chance they have achieved them. If you were to tile the 8 Mpx display described in a 3 by 2 configuration for a total of six tiles you would get a resolution of 48 Mpx and a FOV of 120° by 80°. The scanning fiber display itself takes up the space of ~1cm in height so it would only take 3cm on either side of the lens. This is well within the size constraints of a typical pair of glasses. The idea is in the patent only it isn’t state expressly. It often describes using multiple fiber scanning displays in many distinct arrangements.
There are other issues, such as no current hardware being able to drive such a display, but I think many of these problems can be overcome by the fact that most of the display will be dark. Magic Leap does not have to push every pixel, only the ones required for the current scene. Further, Magic Leap will certainly make heavy use of foveated rendering. At any given time only 8 Mpx worth of pixels will likely be illuminated while the rest of the display is idle. Because of this I speculate Magic Leap will not be able to be used as a VR headset or it won’t be able to at a theoretical resolution of 48 Mpx. It won’t be capable of filling the entire field of view with high resolution images. It will always be limited to a subset of the scene or lower resolution. How much of the scene it is able to fill might be a crucial metric that we look at in future iterations of this sort of product.
I know this is massively speculative and likely not true. I know it even sounds a little bit crazy but it does line up with what we have heard from people who have tried Magic Leap. Many have claimed it is not possible to see pixels or the dreaded “screen door effect”. This implies a high resolution. The field of view is clearly under NDA. This means they are hiding it either because it is bad or because they want to surprise people with it. We have consistently heard that people are blown away by the technology and these are the same people who grumble about hololens due to poor field of view. I think this is evidence that they simply want to surprise us.
Further evidence is the reaction to my previous article where I postulated a 90 degree FOV.
— Andy Fouché (@AndyFouche) July 28, 2016
This is the best technical guessimate of how Magic Leap’s VR technology works; it matches what I saw in person. https://t.co/crLSg4bUIp
— Kevin Kelly (@kevin2kelly) July 31, 2016
Andy Fouché is the head of PR at Magic Leap and Kevin Kelly is a wired editor who has personally tried Magic Leap. I did say quite a bit in my previous article so they might have been ignoring my estimation on field of view but I believe it does lend credence to the theory.
In any case, it is clear that Magic Leap is working to provide the highest resolution and field of view that it can. I don’t want to get peoples hopes up too high. If I were you, I would expect something closer to 50 degree field of view and don’t let the hype train sweep you away to inevitable disappointment. That said, it is fun to speculate about achieving a 90+ degree field of view. If they have produced this then they will have made a technical jump that is unprecedented in the industry of consumer electronics. Unprecedented in the same way their funding is unprecedented. It is okay to get a little excited about that possibility.