The field of view question

Many readers took issue with my claim of Magic Leap providing a 90° field of view and rightly so.  It is the most speculative statement in a speculative article and there is a good chance it is grossly wrong. It is particularly hard to reconcile with the quote stating that it would require 50 Mpx to produce a high field of view display.  50 Mpx is absurdly large and is far beyond anything we are capable of today.  So how can I stand by my statement? By reading more patents of course.

Not long ago I published my thoughts on how I think Magic Leap will work based on a number of sources but primarily on this patent.  In the previous article, I make some bold claims about resolution and field of view that, while based on information from that patent, is largely speculative.  To be honest, the whole article is largely speculative considering patents do not equal products.  As one reader put it, I built my description of the technology “by reading patent applications and other tea leaves”.  While admittedly flawed, these documents are the best information we have to make guesses at the coming products out of Magic Leap so I’m diving in again to see if I can put more weight behind some of these claims.  

Many readers took issue with my claim of Magic Leap providing a 90° field of view and rightly so.  It is the most speculative statement in a speculative article and there is a good chance it is grossly wrong. It is particularly hard to reconcile with the previous quote stating that it would require 50 Mpx to produce a high field of view display.  50 Mpx is absurdly large and is far beyond anything we are capable of today.  So how can I stand by my statement? By reading more patents of course.

Why is field of view hard?

This patent describes in further detail many different configurations Magic Leap might employ to create their head mounted display.  In particular it has lengthy sections on the difficulties of chromatic aberration, exit pupil and, you guessed it, field of view.  

Why is field of view so challenging? This Magic Leap patent gives us some insight into the problems involved.  From the patent: “In a conventional waveguide approach wherein total internal reflection is relied upon for light propagation, the field of view is restricted because there is only a certain angular range of rays propagating through the waveguide (others may escape out)”. So light bounces around in a waveguide at a low angle (think of a glancing blow off of the edge of the glass) and when it achieves a high angle it escapes and we can see it.  Towards the edge of the waveguide you need a low angle to be able to aim the light at the eye.  That low angle will cause reflection and not the desired transmission so it cannot escape and cannot contribute to the field of view.

 The top image shows total internal reflection happening.  All the angles hit by the light beam are low so they are reflected within the waveguide.  The second image has a mirror to redirect the light beam to hit the waveguide at a high angle.  This allows it to exit and such that we can see it.  The bottom image shows light escaping the waveguide and aimed at an eye.  The light near the center of the waveguide has a high enough angle to escape and hit the eye.  The light near the edge of the waveguide, the would-be dotted lines if it could escape, needs too low an angle to be aimed at the eye.  It cannot escape and continues to reflect instead of contributing to the field of view.  This is a major limiting factor for field of view. 
The top image shows total internal reflection happening.  All the angles hit by the light beam are low so they are reflected within the waveguide.  The second image has a mirror to redirect the light beam to hit the waveguide at a high angle.  This allows it to exit and such that we can see it.  The bottom image shows light escaping the waveguide and aimed at an eye.  The light near the center of the waveguide has a high enough angle to escape and hit the eye.  The light near the edge of the waveguide, the would-be dotted lines if it could escape, needs too low an angle to be aimed at the eye.  It cannot escape and continues to reflect instead of contributing to the field of view.  This is a major limiting factor for field of view. 

Given this fundamental constraint, increasing field of view is challenging but possible.  The patent outlines a number of methods to increase field of view and I honestly don’t fully comprehend most of them.  I will try to relate what I can decipher and give you hints at the other solutions but I don’t think I can give you a complete description.

Tiling

To overcome the issue of field of view, Magic Leap appears to be employing a tiling approach. The patent mentions many methods which may be used to tile images to generate a larger field of view.  From the patent, “a narrow field of view sub-image which is tiled with other narrow field of view sub-images presented by the other reflective surfaces to form a composite wide field of view image” and “wherein 6 sub-images are to be presented to the eye frame-sequential to form a large tiled image”.  Keep that number 6 in mind for later as I think it is telling. The methods for tiling range from using a number of reflective surfaces that can be turned on and off (similar to the DOEs explained in my previous article) to using “freeform” optics in ways that I can’t fully comprehend.  Tiling is conceptually quite friendly as you can imagine using 6 different scanning projectors to provide 6 different sections of the images that take different optical paths to your eye allowing an expansion of field of view. 

Freeform Optics


At no time since the science of optical system development was created in 1885 has the industry been so technologically dynamic.


The patent has an entire section on the use of freeform optics.  Freeform optics appear to be any sort of optical interface that does not conform to simple shapes such as spheres, parabolas or cylinders.  While this sounds fairly simple it means the surface geometry must be carefully calculated and manufactured.  Optics people seem very excited about the prospects of freeform optics stating “At no time since the science of optical system development was created in 1885 has the industry been so technologically dynamic.”. For Magic Leap, this means they can produce specially constructed lens arrangements to generate a larger field of view.  As the patent states: “With freeform lenses, rather than having a standard planar reflector, one has a combined reflector and lens with power, which means that the curved lens geometry determines the field of view … it is possible for a freeform arrangement to realize a significantly larger field of view for a given set of optical requirements”.   It appears that these optics don’t make use of total internal reflection (or at least don’t use it exclusively) so they can avoid the fundamental limitation discussed above.  

The patent goes on to indicate a combination of tiling and freeform optics may be used.  In particular, it references 2 different tiles but goes on to show an image with 6 potential freeform wedges driven by 6 different displays.  

Unfortunately, the science of freeform optics appears to be complex and hard to digest.  The surface structure is defined by numerically solving complex differentials and then fitting high order polynomials to the solution.  I wish I understood the concepts more thoroughly but regardless of how it works it does appear to be causing a revolution in the optics space and will give Magic Leap a much needed tool in generating a wide field of view.  

Okay, time to wildly speculate

In a previous patent we get this quote: “To best match the capabilities of the average human visual system, an HMD should provide 20/20 visual acuity over a 40° by 40° FOV, so at an angular resolution of 50 arc-seconds this equates to about 8 megapixels (Mpx). To increase this to a desired 120° by 80° FOV would require nearly 50 Mpx”.  The patent then ignores this statement and goes on to describe a system that provides 8 Mpx across a 40° field of view. This has always seemed odd to me.  The start of the patent explicitly says that a small field of view is bad and that a desired field of view is 120° by 80° with nearly 50 Mpx.  Why say this? Why pick those particular numbers especially if you can’t achieve them?  

I think there is a chance they have achieved them. If you were to tile the 8 Mpx display described in a 3 by 2 configuration for a total of six tiles you would get a resolution of 48 Mpx and a FOV of 120° by 80°.  The scanning fiber display itself takes up the space of ~1cm in height so it would only take 3cm on either side of the lens.  This is well within the size constraints of a typical pair of glasses. The idea is in the patent only it isn’t state expressly. It often describes using multiple fiber scanning displays in many distinct arrangements.

 So many scanning fibers.  So much resolution.  So much field of view. Is this at all possible? Probably not but a guy can dream right?
So many scanning fibers.  So much resolution.  So much field of view. Is this at all possible? Probably not but a guy can dream right?

There are other issues, such as no current hardware being able to drive such a display, but I think many of these problems can be overcome by the fact that most of the display will be dark.  Magic Leap does not have to push every pixel, only the ones required for the current scene.  Further, Magic Leap will certainly make heavy use of foveated rendering.  At any given time only 8 Mpx worth of pixels will likely be illuminated while the rest of the display is idle. Because of this I speculate Magic Leap will not be able to be used as a VR headset or it won’t be able to at a theoretical resolution of 48 Mpx.  It won’t be capable of filling the entire field of view with high resolution images.  It will always be limited to a subset of the scene or lower resolution. How much of the scene it is able to fill might be a crucial metric that we look at in future iterations of this sort of product.   

I know this is massively speculative and likely not true.  I know it even sounds a little bit crazy but it does line up with what we have heard from people who have tried Magic Leap.  Many have claimed it is not possible to see pixels or the dreaded “screen door effect”.  This implies a high resolution. The field of view is clearly under NDA. This means they are hiding it either because it is bad or because they want to surprise people with it. We have consistently heard that people are blown away by the technology and these are the same people who grumble about hololens due to poor field of view. I think this is evidence that they simply want to surprise us.

Further evidence is the reaction to my previous article where I postulated a 90 degree FOV.  

Andy Fouché is the head of PR at Magic Leap and Kevin Kelly is a wired editor who has personally tried Magic Leap.  I did say quite a bit in my previous article so they might have been ignoring my estimation on field of view but I believe it does lend credence to the theory.  

In any case, it is clear that Magic Leap is working to provide the highest resolution and field of view that it can. I don’t want to get peoples hopes up too high.  If I were you, I would expect something closer to 50 degree field of view and don’t let the hype train sweep you away to inevitable disappointment. That said, it is fun to speculate about achieving a 90+ degree field of view.  If they have produced this then they will have made a technical jump that is unprecedented in the industry of consumer electronics.  Unprecedented in the same way their funding is unprecedented. It is okay to get a little excited about that possibility.  

The mixed reality gamble: How the big tech companies are placing their bets

Finding the next big thing is hard. Billions are spent and wasted every year in its pursuit. Even with these large resources most tech companies fail to strike on anything interesting and the cost of failure can be high in both dollars spent and reputation.  So how are these companies reacting to the potential future of mixed reality?

Finding the next big thing is hard. Billions are spent and wasted every year in its pursuit.  Even with these large resources most tech companies fail to strike on anything interesting. The amount spent and ink spilled on the smartwatch, for example, should have all but assured its success but other than the initial boost from all the PR, it has floundered in the market.  Companies are desperate for that next great thing but the cost of failure can be high in both dollars spent and reputation.  

Some believe the next big thing is Mixed Reality. Certainly, Magic Leap believe it. But this is by no means a sure thing. Even if the technology delivers on the expectations, the public might not be willing to bite.  Yet, it cannot be ignored.  Right now, all the major tech companies are placing their bets on the potential of an MR future.  Let’s take a look at where they are placing their chips. 

Microsoft

Microsoft is at the forefront of mixed reality with Hololens.  You can buy a development kit today and see exactly how it works and what it can do.  They have clearly sunk significant resource into the project and continue to push the technology forward.  But this has become somewhat more complicated in recent months when Microsoft announced Windows Holographic. In traditional Microsoft style, they are creating and hoping to license the operating system for a mixed reality headset. This will solve a large problem for OEMs wanting to make their own headset and positions Microsoft to profit regardless of who can create the best headset as long as it is running Windows Holographic.  

While this is a smart move regardless it does show a lack of faith in Hololens as a product.  I would speculate that they have hit certain road blocks with Hololens, in particular with resolution and field of view.  They know overcoming these will be challenging and potentially a large distraction from their current business.  Letting the rest of the market sort these issues out will save them that trouble.  

In either case, Microsoft is making a heavy bet on MR.  If it takes off they will be well positioned to influence and profit from the future of Mixed Reality.  

Magic Leap

Well this one is obvious.  They are betting the farm on their mixed reality headset to be a success. If it flops, so does the company. 

But is it really that simple?  Did they really convince investors to give that much money on such a risky gamble?  I think Magic Leap can be successful even if their headset flops. They own a number of fundamentally new technology whose applications will likely go beyond an MR headset. Not only that, they will have the only manufacturing line that is able to mass produce this technology.  You could envision building television or smartphone screens using the fiber scanning projector. The advanced optics could similarly be incorporated into car windscreens and VR products. They could licence the technology to other MR headsets perhaps using windows holographic. Magic Leap is betting on hardware in the same way that Microsoft is betting on software.  

With that in mind, I envision a potential future where Magic Leap’s headset flops and the internet explodes with “I told you so” articles about what a failure they are.  Meanwhile, they start raking in money from the licensing of their many technologies.  This has put me in the odd position of being cynical of potential future cynicism. So I’ll stop there.  

Google

It could be argued that Google kicked off the current push for MR with Google Glass. Glass, while never seeing a consumer release, created huge excitement for the potential of the technology. When the product couldn’t deliver on expectations, that excitement soured.  But the original excitement was real. It showed companies like Microsoft and people like Rony Abovitz that the public is interested in this technology, if you can only deliver on the expectations people have. So while never being a consumer success we still have a lot to thank Glass for in terms of pushing technology forward.   

Fast forward to today and we see Google investing heavily in Magic Leap while simultaneously pursuing their own technology. Google has always been about creating technology for everyone as opposed to just the high end of the market and we see this in how they are looking at mixed reality. Project Tango is aiming to bring SLAM processing to any smartphone.  While projects like Soli hope to bring gesture recognition to a much smaller form factor.  By combining Daydream with Tango and Soli they could build a Google Cardboard like mixed reality headset that could introduce the world to the concept without the need to buy expensive hardware beyond what they already own.

With the stake that Google owns in Magic Leap, they could sit back and do nothing while MR takes off and still make a tidy profit off of it. Meanwhile they continue to be good technology citizens by fostering developers in this new field and pushing MR forward to the masses.  

Apple

No one knows what Apple is doing. They are certainly working on something, they would be crazy not to, but they won’t talk about it until it is ready to go.  If history is any tell, they will wait until others have tested the market. Apple tends look at what consumers see as pain points in an existing products and then bring out their own product addressing those issues. I don’t want to be too down on Apple but I just don’t believe they will bring anything to the table in the infancy of MR. They don’t have the same R&D as some of the other companies and tend to be far more conservative when bringing out new products. I would love to see Apple open up more and give back to the developer community by showing what they are working on but traditionally this is not something Apple does. We will just have to wait and see.

The Interface of the Future: How will we use Magic Leap?

With new technology comes new fundamental interface paradigms. We are currently in a rare moment where we can guess at what the next paradigm might be and where Magic Leap fits in.

With new technology comes new fundamental interface paradigms. These sorts of paradigms don’t change often and are intrinsically tied to hardware capabilities. As technology evolves, we tend to settle on a core interface that defines how developers build the products we come to love.  In the early days of computing, we used printers and punchcards.  Then, the mother of all demos showed us the future: the display, mouse and keyboard.  These are such powerful concepts that almost 50 years later we are still using them.  More recently, touchscreens, and in particular the capacitive touchscreen, have driven the smartphone revolution.  Wayne Westerman and John Elias brought these to the world and paved the way for companies like Apple and Google to build the modern smartphone.  

We are currently in a rare moment where we can see an approaching shift in technology. Outside of videogames, where Nintendo does something wacky every 5 years or so, it is hard to think of another time in recent memory when we saw this shift coming and had a time to ruminate on it before it emerged. 

Mixed Reality is one of these shifts.  In this post, I want to wildly speculate on what the core interface paradigm for this technology will be. How will we select information we care about? How will we input text? How do we navigate between different sections of the UI?  Of course, I am going to focus on Magic Leap but I want to start somewhere else.  Hololens.

Hololens

Wait a minute. This isn’t an approaching shift in technology.  It is already here.  You can buy a hololens dev kit for $3000 and start playing with it today. We have a fully working and thought out interface for an MR product right now.  

 Expect to see a lot of this in the future
Expect to see a lot of this in the future

Kind of.

Hololens isn’t yet a consumer product.  It is a work in progress.  Hands on articles, videos and, seemingly, even Microsoft have all agreed that the technology is amazing but the product isn’t quite ready for consumer release.  That said the interface seems to be fairly well thought out and I think Magic Leap will operate in a similar way.  


HoloLens uses the position and orientation of a user’s head, not their eyes, to determine their gaze vector


Hololens has 3 forms of interaction: gaze, gesture and voice.  To broadly put these in a context of how we use technology today we can think of gaze as similar to a mouse, gestures as similar to mouse buttons and voice behaves like the keyboard.  

I worry about these interaction paradigms.  We all know how annoying voice interaction is to use. We’ve all tried Siri and Google Now and then decided not to use them except if very rare situations.  If the Wii/Kinect/Playstation Move proved anything it is that gesture based input tends to be terrible. The reason we get annoyed using voice input and gestures is an issue of feedback.  Voice input is slow.  You don’t know if the system is correctly picking up the words you are saying until you are done saying them. Even if it works 90% of the time, the 10% that it doesn’t is so frustrating that you give up on it entirely. Gestures have the same problem. It can be hard to tell if the system has actually registered your gesture or if it is just slow to load in this instance. A mouse and keyboard would suffer the same issues if they were as unreliable but they tend to be the most responsive part of a computer. You know your computer is really struggling when you can type faster than the characters are being displayed. Gaze, on the other hand, is less proven and perhaps has the most potential.

Magic Leap

So what else can we do? Honestly, I have no idea. Gaze, Gesture and Voice are the most obvious way to approach this problem and I expect Magic Leap to utilize them. Perhaps the only thing they can do is try to make these inputs as reliable as a mouse or keyboard. We see hints of this already. Magic Leap needs to do frame by frame, highly accurate eye tracking to achieve a light field display. This can be used for the gaze portion of the UI. I imagine you will simply have to look at something to select it in some way. We can already do this today. There is a company today called Tobii that has built eye tracking for your laptop.

This shows what is available to consumers today but in quite a different form factor.  Given Magic Leap will be far closer to the eye I would speculate they might be able to do a better job than Tobii.  If it is highly accurate it may have the same reliability as the mouse and alleviate any concerns for the gaze section of the interface. 

Of course you don’t want to select everything you look at so gaze will have to be combined with a gesture likely similar to Hololens “airtap”. This seems to be confirmed in the patent images which list a number of gesture interactions.

Gesture and voice input are problems that many companies have put huge resources in already.  I don’t imagine Magic Leap will improve drastically on the status quo.  In fact, there is a decent chance they will be worse at solving these problems in the first iteration than other companies.  I think this will be okay, as long as they can nail at least one gesture.  They need the airtap gesture to work.  I imagine this is the primary way we will interact with objects around us. If this is unreliable then the system will be frustrating to use regardless of anything else.

The patent image above shows another important interaction and that is the home button. It is easy to get lost in a UI.  Things like the start button on Windows or the home button on smartphones are vital to be able to find your way back somewhere familiar.  

Totems 

Magic Leap’s patent documentation talks extensively about totems. 

“[Totems are] Physical objects which are manipulable by the user to allow input or interaction with the AR system. These physical objects are referred to herein as totems. Some totems may take the form of inanimate objects, for example a piece of metal or plastic, a wall, a surface of table. Alternatively, some totems may take the form of animate objects, for example a hand of the user.”

From this description a totems can be just about anything at all that is recognized by Magic Leap. Many totems described are blank slates. Simple objects that are given life by projecting holograms on them.  One such example is a blank rectangle on which a keyboard is projected. Another is a blank mouse. Since the dimensions and properties are known quantities it may allow for simpler recognition of gestures pertaining to the totem.  

 A squishy keyboard totem.  Remember the keys might not actually be there but just be holograms on a squishy slab.
A squishy keyboard totem.  Remember the keys might not actually be there but just be holograms on a squishy slab.

The point of totems seems to be the ability to give physical presence to holograms. They are described as empty shells, useless shapes, that are only useful when the context of a hologram is projected on to them.  One example is particularly interesting.  The patent document outlines how a blank piece of aluminum becomes a smartphone.  

“[T]he AR system may render the user interface of an Android phone onto a surface of an aluminum sheet. The AR system may detect interaction with the rendered virtual user interface, for instance via a front facing camera, and implement functions based on the detected interactions.”

So, you heard it here first folks. Your next smartphone might be a slab of aluminum.  

Wild Speculation

Magic Leap is not one to go down the path most travelled.  So what are they doing that might be crazy, out there and line up with some of the “GPU of the brain” type talk we keep hearing from them?  There is one line in the patent that implies an entirely different interface to what has been described thus far. They are building a device that sits on your head.  It might be the first time a company can reasonable build an EEG into their product.  From the patent:

“[T]he system may measure neurological signals and use that as an input for the system. The system may have a sensor that tracks brain signals and map it against a table of commands. In other words, the user input is simply the user’s thoughts, that may be measured by the user’s brain signals. This may also be referred to as subvocalization sensing. Such a system may also include apparatus for sensing EEG data to translate the user’s “thoughts” into brain signals that may be decipherable by the system.”

It is positioned as almost an afterthought in the patent so I don’t think they are actually trying to implement “subvocalization sensing”. Well, at least not in the first iteration.  

New Magic Leap video shows impressive visuals, disappointing interface

We get another great example of Magic Leap’s hardware in action but is the interface what we should expect from the final product?

Earlier this week, Magic Leap CMO Brian Wallace spoke at a conference in China to Alibaba CMO Chris Tung.  Towards the end of the talk we were given another short video of Magic Leap in action.  There are some smartphone videos on youtube of the footage but The Verge managed to snag a good quality copy from the company itself.

 Brian Wallace in all his grainy glory!
Brian Wallace in all his grainy glory!

It depicts a user shopping for baby toys in a nursery.  We see the toys rendered in the real world just as nicely as if they were actually there.  The imagery is great but I am starting to worry about the software.  I can not say I was blown away by the interface depicted.  We see a grid of icons without context for what they are.  They appear floating in a good but relatively arbitrary position. There appears to be no way to get to any other apps or UI elements.  This is similar to every other video we have seen depicting Magic Leap.

Because of this, I am convinced what we are seeing is nothing like the actual interface Magic Leap will have.  These are complete mock ups that do not reflect the reality of the product.  They built it simply to show off a few use cases to give an idea of what the product will do.  For my part, I am going to largely ignore the software issues and chalk it up to a demo.  I imagine we won’t know what the true interface looks like until the product is unveiled.  That is going to be an exciting day. 

How Magic Leap Works – From Field of View to GPU

Magic Leap has not been forthcoming with details about how their technology works.  From what little we know, it is a truly novel system with capabilities far beyond the off the shelf components consumers are accustomed to.  Putting together patents and presentations we can get a pretty good picture of how the technology works and makes some educated guess on the specifications, such as field of view and resolution, we can expect to see.

Magic Leap has not been forthcoming with details about how their technology works.  From what little we know, it is a truly novel system with capabilities far beyond the off the shelf components consumers are accustomed to.  You can’t blame Magic Leap for wanting to keep it a secret.  There are many companies sniffing at their heels trying to emulate what has got people so excited. It sounds like the exact technology that Apple wishes it had.  A potentially revolutionary new thing that shows signs of true innovation.  It feels like it is what Microsoft was striving for with Hololens but haven’t quite got there yet.  It takes inspiration from Google Glass but is clearly gone generations beyond it.  So what is it and how does it work?  I have gone through talks, patents, job applications and the background of people working at the company to try to find out.

The Broad Strokes

 An example of the size of glasses we might expect. Now where to put the cameras... Note: This is not an actual image of a Magic Leap product.
An example of the size of glasses we might expect. Now where to put the cameras… Note: This is not an actual image of a Magic Leap product.

Before we dive into too much detail, let’s get an overview of what this will be. Simply put, Magic Leap is building a device capable of manifesting objects into a person’s field of view with a level of realism beyond what we have seen so far from other similar devices. Magic Leap will come in two parts: a pair of glasses and small pocket projector/compute unit, think phone sized rectangle without a screen. The pocket unit will be connected via cable to the glasses.  The glasses will be similar in size and design to glasses worn by people today, though they may be a bit chunkier than we are used to. The small size of the headset is the most fundamental piece of this product.  It means it will be socially acceptable to wear in public and gives it the potential to have the utility and accessibility of a smartphone.  

The Pocket Projector and Compute Unit

 This is a battery pack but the Magic Leap pocket unit could look similar.  
This is a battery pack but the Magic Leap pocket unit could look similar.  

The major trick Magic Leap has been able to pull off is to remove most of the required hardware from the glasses and put it in a separate unit.  Hololens, as a counter example, does an impressive job of keeping the size down but they can only do so much as all the components are placed in the headgear.  So what will the pocket unit contain.  Likely the following: 

  • Battery

    This will take a similar amount of power as a modern smartphone and potentially more depending on use. If this is to replace the smartphone it will need a seriously beefy battery. My guess would be at least 5000mAh.

  • CPU/GPU:

    These will likely be the latest generation mobile CPUs. I am guessing they will source them from Qualcomm. Luckily, they will save on the need to have high end graphics processing as MR only requires the rendering of some components and not the entire scene. This eliminates the issue VR has with intense graphic processing.

  • RAM

    Similar requirements to a smartphone. I imagine we get 3 – 4 GB.

  • Custom chips for SLAM processing

    This will be required for placing virtual objects in the real world. They might be building this chip in house or perhaps Movidius chips or some equivalent.

  • 4G/Wifi/Bluetooth connection
  • SIM card
  • GPS Chip
  • Camera

    There needs to be a number of cameras on the glasses but this doesn’t mean that the pocket device won’t have a camera. The requirements of the cameras on the headset to do SLAM are very different than the requirements for a good consumer camera. Given size restrictions on the glasses they may forgo a high powered camera on the headset and put it on the pocket device. This would have the added benefit of alleviating privacy concerns as you won’t be able to take photographs with the glasses alone.

  • Laser Projector

    This is the meat of their innovation. Moving the projection system off of the glasses and on to the pocket unit affords a significant size saving for the final product. The projected light is produced on the pocket device and then sent up to the headset via a fiberoptic cable. Later in the article we will go into more detail on how this works

The Glasses

After stuffing as much as we can in the pocket component of the product, what is left for the glasses? It will need to fit the following:

  • IMU

    This will be a traditional accelerometer, gyroscope  and compass. 
     

  • Headphones

    Perhaps they will use bone conducting headphones as seen with google glass.  This goes with their philosophy of working with your body instead of against it.  Bone conducting has the advantage that you can hear the outside world as well as the played audio.
     

  • Microphone
  • Optics
  • Cameras 

The optics and the Cameras are the most interesting components so let’s look at those more closely.

Optics

As we can see from patent applications, the optics used by Magic Leap afford considerable size advantages when compared to more traditional projection systems such as that found in Hololens or Google Glass.  The image above shows that the light source is separated from the main headset and it is how we can surmise that the pocket device will drive the light.  Secondly, it shows a lens system that is very small.  The image is clearly not meant to be to scale but must represent the approximate sizes of the components involved.  The only component we have actually seen is the photonic chip. Comparing components 5, 6, 7 and 8 (above) to the width of the chip and we start to see the sizes involved.  

So what is going on here? How have they shrunk the optics so much while also claiming to achieve a light field display, high resolution and impressive field of view?  The answer is two fold: The fiber scanning display and the photonic lightfield chip.

Fiber Scanning Display

The fiber scanning display is a completely novel display system that has not been used in a consumer product before.  Much of what we know of this invention comes from a patent application dating back to 2013.  The application is from some time ago so we can expect some of the details related to the performance of the system to have changed but the broad idea likely still holds true. The system uses an actuating fiber optic cable to scan out images much larger than the aperture of the fiber itself.  It works in a similar way to old style tube TV.  Instead of scanning electrons, this is scanning the light itself.  By using a piezoelectric actuator to achieve this scanning, one can maintain scan rates on the order of 10s of kHz.  We don’t get a framerate that high though because it takes multiple passes (the patent has an example of 250 cycles) to produce a full frame.  This changes how we think about resolution.  With this technology, resolution is dependant on fiber scan rate,  the minimum spot size the fiber can focus to (this will define pixel pitch), the number of scans to generate a full frame and the desired frame rate.  Depending on how well they have optimised since this patent was filed we can expect resolution much higher than any consumer system seen today.  

 Packing a number of fiber scanning units together to increase the size of the display.  Each tube is 1mm across.
Packing a number of fiber scanning units together to increase the size of the display.  Each tube is 1mm across.

While resolution and framerate are crucial to creating realistic holograms, field of view is equally important.  Related to this, there interesting paragraph in the background information section of the patent.

“The Field of view (FOV) of the head mounted display (HMD) may be determined by the microdisplay image size together with the viewing optics. The human visual system has a total FOV of about 200° horizontal by 130° horizontal (sic), but most HMDs provide on the order of 40° FOV.  …. An angular resolution of about 50-60 arc-seconds is a threshold for 20/20 visual acuity performance, and it is determined by the pixel density of the microdisplay. To best match the capabilities of the average human visual system, an HMD should provide 20/20 visual acuity over a 40° by 40° FOV, so at an angular resolution of 50 arc-seconds this equates to about 8 megapixels (Mpx) . To increase this to a desired 120° by 80° FOV would require nearly 50 Mpx.”

This puts two things in to place.  The first is that consumer displays are almost an order of magnitude behind what they need to be to increase FOV.  This makes it clear why hololens is struggling so much to produce a large FOV.  Second, it shows Magic Leap’s aspirations.  They want to produce a field of view of 120° by 80°.  This would be a larger field of view than the oculus rift and at far greater resolution.  So have they achieved this?  It is hard to say but the patent does give us some numbers to work with bearing in mind these are over 3 years old and they have likely improved the technology even more in the mean time.  

Pixel pitch is the distance from the center of one pixel to the center of the next and is a limiting factor in resolution.  Traditional microdisplays, like those used by hololens, have a pixel pitch of about 4-5 microns.  This limits the resolution of these displays and thereby also limits the field of view that can be produced.  The patent indicates that a scanning fiber display can produces a pixel pitch of 0.6 microns which is an order of magnitude improvement.  

So what resolution does that produce? They quote a resolution of 4375 x 2300 in one section of the patent but I don’t think that tells the whole story.  This is the example given for a naive approach before they start discussing multi-core fibers improving this further.  I believe the resolution is much higher than that.   This is crucial if we want a large field of view.  

Finally given the stated aspiration of a 120° field of view this line is of particular note:

“The above described technologies facilitate an ultra-high resolution display that supports a large FOV in a head-mounted or other near-to-eye display configuration.”

I think this all but confirms we will have a FOV that is, at minimum, far greater than 40° and I don’t think it is crazy to think it is approaching the stated goal of 120°.  If I was a gambling man, I’d put my money on 90°.

Photonic Lightfield Chip

When I first heard Rony Abovitz call his lens a “photonic lightfield chip”, I groaned.  Not another crazy name for something that already exists.  It is called a lens Rony! But the more I researched it the more it became apparent that it is in fact much more than a simple lens.  So, How does it work and why is it far more interesting than a simple lens? Let’s talk about diffractive optical elements.  

 An example of a Diffractive Optical Element
An example of a Diffractive Optical Element

Diffractive Optical Elements (DOEs) can be thought of as very thin “lenses” that provide beam shaping, beam splitting and diffusing or homogenizing.  Magic leap utilized linear diffractive grating with circular lens to split the beam wave front and produce beams with desired focus.  That is to say it directs the light to your eyes and makes it look like it is in the correct focal plane.  But this is far easier said than done and it is by no means easy to say.  The patent document that I am pulling all of my information from is verbose to say the least.  

To build a light field, Magic Leap has set up a photonic chip with two separate components. One element (6 in the diagram) which takes the projected light and inserts it into the second element (1) which redirects the light into your eyes.

Both components make use of DOEs to do their job.  The main drawback of DOEs is that they are highly tuned to do one specific job.  They cannot operate on different wavelengths and they cannot change properties to allow for different focal points in real time.  To solve this, Magic Leap has layered a number of DOEs together into the larger lens-like component that are tuned to different wavelengths and focal planes.  These DOEs are extremely thin, they are on the same scales as the wavelength of the light they are manipulating so this doesn’t add much bulk to the apparatus.  Here is where the chip nature of this optical system comes in.  Magic Leap is able to turn on or off the different layers of DOEs.  By doing this they can change the path in which the light reaches your eyes.  This is how they change the focal point of the image and achieve a true light field.  As the patent says:

“For example, a first DOE in the set, when switched ON, may produce an image at an optical viewing distance of 1 meter for a viewer looking into the primary or emission face of the planar waveguide. A second DOE in the set, when switched ON, may produce an image at an optical viewing distance of 1.25 meters.”

It might seem that this is highly limiting as you would need a large number of layers to produce the full range of focal points but this isn’t the case.  Different combinations of DOEs in conjunction also produce different output states.  So it isn’t one focal plane per DOE, it is one per combination of DOEs.

 Changing the set of DOEs that are currently active changes the path in which the light exits the Photonic Lightfield Chip.  They will likely have more layers than the number depicted here but how many is anyones guess.
Changing the set of DOEs that are currently active changes the path in which the light exits the Photonic Lightfield Chip.  They will likely have more layers than the number depicted here but how many is anyones guess.

Finally, we see how Magic Leap manages to create black with light as they have claimed to be able to do in the past.  If we take a DOE on the outer edge of the lens and one on the inner edge we can use them to cancel out light similar to noise cancelling headphones.  From the patent:

“Such may be used to cancel light from the planar waveguides with respect to light from the background or real world, in some respects similar to noise canceling headphones.”

So, why is this a chip?  Well, a typical electron chip changes the flow of electrons based on certain conditions.  The Magic Leap photonic lightfield chip changes the pathways of photons based on certain parameters. Sounds like a chip to me.  

Where does this leave us?  We have a photonic lightfield chip and we have a high resolution projector but how do we actually create an image. This is done via composition.  The image is layered such that the different components are projected at different focal lengths on a subframe basis.  This means, in a single frame there are multiple passes to construct the entire frame, each focal plane being laid down individually. 

Cameras

Magic Leap is trying to accommodate 3 different goals with camera technology.  The first is the most obvious.  A camera to produce everyday pictures.  This is the most well understood of the camera technology they will be using and they will likely use a similar sensor to the latest in the smartphone market.  If this sensor lives on the glasses or if it lives on the pocket device is still up in the air but it will have a camera that is capable of taking decent pictures.  

There are two other use cases which are far more interesting. Magic Leap has repeatedly talked about the ability for the device to understand the world around it.  In one particular interview, it was mentioned that it will be able to recognise objects, such as a knife and fork. To be able to do this they will need an array of cameras.  As an example of a device that does this very well we can look at hololens.  Hololens contains an array of four environment sensing cameras that work in combination with a depth sensing camera.  We get further information about Magic Leap from the patent documents.  

 This diagram shows two components on the the left and right arms of the glasses.  The top is the left arm and the bottom is the right arm.
This diagram shows two components on the the left and right arms of the glasses.  The top is the left arm and the bottom is the right arm.

As we can see from the above diagram, we can expect two outward facing cameras, labeled as “world camara”.  That said, the text of the patent implies there could be more than two, stating “one or more right outward facing or world view cameras [per side]”.  At this point it is unknown how many cameras will be included on the system nor is it known how much Magic Leap is able to shrink these components.   We do know they will be on the glasses and they are vital to SLAM processing. 

The final use case for cameras can also be seen in the diagram above. At least two cameras will be pointed at your eyes.  This is used to track your gaze and vergence such that the focal point and the direction of view can be obtained.  It will also point an infrared LED at your eye to illuminate the eye for these cameras. This eye tracking will be critical to the user interface.  I imagine the question of “what are you looking at?” will be fundamental to how you interact with Magic Leap.  It will potentially be the main interactive component similar to that of a mouse.

Clearly there is no way for me to verify this information at this time but it all adds up to sound like the product Magic Leap is trying to produce.  Regardless if it turns out to be a consumer success or not, this is the first example of real innovation the tech industry has seen in some time.   I am extremely excited to see what happens next for them and looking forward to the shake up this will put on the industry in general.

“We are kind of like a baby Apple” says Abovitz at the Fortune Brainstorm Tech 2016 conference

In interview after interview one thing always rings clear and that is that the ambition of Magic Leap is perhaps its largest feature.  Abovitz claims they are kind of like a “baby Apple”, while Wallace states we won’t mind wearing magic leap “out in the world, everyday”.  To me, they are clearly going after the smartphone market and whether they succeed or fail, it sure will be exciting to watch.  


“Looking extremely real, very real. So much so I thought I could touch the character”

-Robert Iger, Chairman and CEO of Disney


These days, it seems every other week we get a new interview with folks at Magic Leap.  This time it is courtesy of Fortune Brainstorm Tech 2016 conference.  First off, Robert Iger, Chairman and CEO of Disney, talks briefly about the experience he has had with Magic Leap off the back of the Star Wars partnership.  He explains a demo where in tinkerbell flies around his head and right up to his face.  His reaction is another in a long string of enthusiastic responses we have come to expect from seeing the technology.  He claims that tinkerbell was so real he “thought [he] could touch the character”.  When asked to rate the experience given the context of all the amazing technology he sees as CEO of Disney, he says that it is easily a ten out of ten.  He adds that the most exciting part is to imagine what comes next, as in, what other experiences can be made with this technology.  Magic Leap and Disney have just entered a partnership so we can’t take his word as an objective observer but he did seem genuinely excited and his remarks line up with everything else we have heard about the tech.   You can see the full interview below, helpfully queued up to his thoughts on Magic Leap.

Next we have Rony Abovitz, the enigmatic CEO of Magic Leap and  Brian Wallace, Magic Leaps CMO.  Abovitz opens with his typical technobabble.  In fact, this time he brought along a strange video where what I assume are cosmic rays hit some eyeballs and a floating pegasus appears in a brain.  It is what we have come to expect from talks by Abovitz but he did say a few interesting things amongst the mostly nonsense.  

“It’s a lightweight headset”, claims Abovitz.  Specifically, it isn’t a pair of glasses.  The distinction in his mind is that glasses do one thing, where as Magic Leap will have “millions of apps”.  While we can only hope that app developers jump on board his reticence to call Magic Leap a pair of glasses is disturbing for those who hope for a glasses size form factor.  Perhaps he wants to curb expectations and ensure we aren’t picturing regular size glasses in our heads when we think about this product.   That said, later in the interview, Wallace mimes taking off magic leap by taking off his own glasses. 


“This is very real. We are not in research mode anymore.”
-Brian Wallace, CMO of Magic Leaps


Now the big question, when do we get to see all this? “Soon-ish”, says Abovitz once again not delivering an answer. They claim the Motorola factory they took over is currently running.  They are in the stages of debugging their high volume production line this summer.  At this point, Wallace who seems to be a little more practical with his verbiage, chimes in and explains that there are already external developers working with the technology and people in the office are wearing it all day for hours at a time.  It sounds as if he has heard the cynicism from the tech press and wants them to know that this is very real.  Hopefully, he can prove it soon by showing it more broadly to the public.  Given the number of people who have seen it (thousands, they claim) it is somewhat shocking that more leaks haven’t happened.  The closest we get to a date is from Wallace. When asked when this will be mainstream he claims it takes about 3 years for that to happen specifically citing the 2020s as aligning with that time frame.  This points to, at earliest, a 2017 release.  

One last thing of note is Abovitz feeling on Google Glass.  He respects them greatly for jumping in first and, while Magic Leap’s products are very different, Glass was important on the road map for this technology.  

In interview after interview one thing always rings clear and that is that the ambition of Magic Leap is perhaps its largest feature.  Abovitz claims they are kind of like a baby Apple, while Wallace states we won’t mind wearing magic leap out in the world, everyday.  To me, they are clearly going after the smartphone market and whether they succeed or fail, it sure will be exciting to watch.  

Magic Leap tells us to “Have Faith”. Should we?

Recently Graham Devine, Chief Creative Officer at Magic Leap, told an audience to “Have Faith” that that the Mixed Reality revolution was coming.  He then went on to outline all of the facets of our lives in which Mixed Reality will have an impact.  In short, he listed just about everything we currently do with technology.  Even the most zealous among us might have a crisis of faith given these broad aspirations.  From the Phantom videogame console to the more recent Theranos scandal, there are plenty of examples to give anyone a health dose of skepticism. So why should we have faith? What evidence do we have to take Devine at his word?

Recently Graham Devine, Chief Creative Officer at Magic Leap, told an audience to “Have Faith” that that the Mixed Reality revolution was coming.  He then went on to outline all of the facets of our lives in which Mixed Reality will have an impact.  In short, he listed just about everything we currently do with technology.  Even the most zealous among us might have a crisis of faith given these broad aspirations.  From the Phantom videogame console to the more recent Theranos scandal, there are plenty of examples to give anyone a health dose of skepticism. So why should we have faith? What evidence do we have to take Devine at his word?


“Faith! What a dirty monosyllable—Jill, why didn’t you mention that one when you were teaching me the short words that mustn’t be used in polite company?”

-Robert Heinlein, Stranger in a Strange Land


When looking for faith in Magic Leap, you cannot go to the source.  Magic Leap themselves have said many things about their company but very little of it is tangible.  From “technical” nonsense such as that from which this blog gets its namesake to broad outlines that give you nothing concrete to hang on to, you would be insane to have faith in Magic Leap from their own press.  Luckily, there are other sources to draw from.  

Investors

The investor story is arguable the only reason we talk about Magic Leap at all.  Their record breaking fundraising pushed them from an unknown company to the front page of every tech blog around the world.  1.39 Billion dollars on the back of a tech demo is unheard of. Yet, other companies have had similar (if not as grandiose) funding stories and still failed to deliver.  The industry is still reeling from the failure of Theranos to deliver a product.  Could Magic Leap follow the same course?

Of course they could but I would guess they cannot be pulling the wool over every ones eyes. The quality and magnitude of the investment coming in is unprecedented for a company at this stage of development. By contrast, a quick scan of the funding rounds Theranos has gone through shows much of their raise has been through private equity and, to a lesser extent, Walgreens.  I’m not sure how much experience Walgreens has in funding technology but they certainly aren’t the first name you think of in the tech investment space.  Magic Leap, on the other hand, has some of the biggest names in the industry backing them: Google, Qualcomm, Alibaba, Warner Brothers, JP Morgan among others. These are all well respected and intelligent organizations.  I could imagine a good demo getting them in the door for one or two of these companies but they seem to be able to raise from a wide variety of companies, across a range of industries.  

Recall, this is a business that has no brand to speak of, no production capability and an unfinished product.  They have been given 1.39 Billion dollars on the back of the potential of the technology alone.  The organisations backing Magic Leap would not thrown that kind of money around without seriously vetting the ability for Magic Leap to follow through on its promises.  For the money we are talking about, those promises must be extraordinary.

Staff

Smart people work at Magic Leap.  Sure, they have a chief futurist, but Neal Stephenson is far more appealing in that “position” than say, Will.I.Am. Let’s look at some of the people building Magic Leap.

  • Graham Devine: A highly respected video game developer brought in to lead the creative content creation on the platform.  
  • Gary Bradski:  One of the leaders in computer vision. He actually wrote the book on it.
  • Brian Schowengerdt: A PhD scientist from the University of Washington. His main research interest were in the functioning of the human visual system and perception. His work involves 3D displays, retinal scanned light displays, autostereoscopic displays, interaction with displays, adaptive displays, human factors, and human vision. 
  • Rony Abovitz: The CEO of Magic Leap.  He has successfully built and sold companies in the past.

This is just a small sampling but shows they have the leading experts in many technology categories fundamental to how the product will need to work.

Partnerships

Two of the largest and most protected intellectual properties in the world include Lord of the Rings and Star Wars.  Both had (or continue to have) huge productions behind them that pioneered new technology in the cinema space.  Both Weta workshops, who did the effects for Lord of the Rings and Lucasfilm, the people behind Star Wars, have committed to partnerships with Magic Leap.  These companies are high profile and pick partnerships carefully.  They would not hop on board with a new and unknown company unless they were convinced it was something special.  

There are more reasons than this to think having faith in Magic Leap might not be as crazy as it seems on the surface.  What little we know of the technology used points to a novel display mechanism that no one else in the industry is looking at.  We will have a post on theories around this technology soon but it does lend credence to the ambitions of Magic Leap.

 

Graham Devine tells us to “Have Faith” in Magic Leap

Graham Devine, Chief Creative Officer at Magic Leap, gave a talk at the Games For Change Festival in New York recently telling folks to “Have faith” that the mixed reality revolution is coming. 

Graham Devine, Chief Creative Officer at Magic Leap, consistently has an air of excitement when he gives talks about his beloved company.  Yet he always has the caveat of not being able to show us anything tangible.  The other day, he gave a talk at the Games For Change Festival in New York telling folks to “Have faith” that the mixed reality revolution is coming.  Considering what we know of the company, this is a poignant statement.  To be excited for the future Graham is laying out, you really do need faith in what he is saying because he sure isn’t going to show it to you.

He went on to layout his 4 pillars of mixed reality: Games, Media, Communication and Information.  This outlines just about everything we do with computers once again showing the broad aspirations of magic leap.  Let’s talk about these Pillars

Games


“[The game industry] is stuck [at Mario 64] … Sometimes we give you guns now”


According to Devine, gaming went through a distinct change when Super Mario 64 was released.  It showed the world how to create a 3D game.  Since that time, not that much has changed in mainstream gaming.  Well, he claims there is one change, “Sometimes we give you guns now”.  In terms of raw game mechanics, and discounting advances in art and story, I think he is broadly correct about this. Devine went on to say Mixed Reality games are going to be a similar sea change to Super Mario 64.  It will be very interesting to see what games are launched with this product with bold claims like this.  Those are the games that will define the category.  

Media

Why would we want Netflix in mixed reality?  This is the central question Devine uses to think about media in this context.  Devine makes the bold claim that in 10 years we won’t have televisions anymore.  We will watch Television through mixed reality instead of on a physical television.  

But that answer presupposed the success of mixed reality in the living room.  To get there, there needs to be a benefit today or whenever magic leap launches.  He sees this as having extra information available while watching.  Who is that actor? What song is playing? Where was this shot? With an MR viewing platform, that information can be provided while simultaneously projecting the video itself.  

Communication

Devine seems to imply a holoportation style communication platform on magic leap.  He mentions that using avatars to communicate is increasingly popular with the younger generations.  The Microsoft demo of Holoporation was incredible but it requires lots of gear to achieve.  I believe Devine was hinting at an avatar based version of this.  Where it isn’t you exactly as you are but a reasonable facsimile.  Nothing was confirmed on this front but it is exciting to speculate.  

Information

If you are 5 miles from your house and you realize you forgot your phone, would you go back for it?  I would.  Devine would too.  He wants Magic Leap to be so useful, have such valuable information access, that it fits into this category.  As a counter example, he notes that the Apple Watch is not one of these products, noting that he might not even go into his living room for it.  Yes, the Apple watch is mostly uselss, but is Magic Leap going to so important that we treat it like our phones?  He believes so and is striving for that goal.  

 

If there is one take away from this talk, and all other talks we have seen from the company, it is that they are highly ambitious.  They want to take the market that smartphone now hold.  Is that possible? Who knows. But Graham Devines wants you to have faith.

You can watch the video for yourself here and here.  Sorry for the terrible periscope link but I could not find it on youtube.

Sis Wei has some nice notes on the presentation as well.

Magic Leap and Lucasfilm team up showing strong support for Magic Leap’s mysterious technology

The excitement and skepticism around Magic Leap seems to grow each day.  It is hard to know if Magic Leap is going to change the world or dissolve into vapour.  Well, chalk one up for the world changers.  Magic Leap and Lucasfilm’s ILMxLAB recently announced a collaboration at the 2016 Wired Business Conference showing a new level of confidence in Magic Leap technology.  Aside from the funding, this is perhaps the the most telling signal that Magic Leap has a truly interesting product on their hands. 

The excitement and skepticism around Magic Leap seems to grow each day.  It is hard to know if Magic Leap is going to change the world or dissolve into vapour.  Well, chalk one up for the world changers.  Magic Leap and Lucasfilm’s ILMxLAB recently announced a collaboration at the 2016 Wired Business Conference showing a new level of confidence in Magic Leap technology.  Aside from the funding, this is perhaps the the most telling signal that Magic Leap has a truly interesting product on their hands. 

ILMxLAB was launched in June of 2015 in an effort to bring Lucasfilms creative to MR experiences.  They have show impressive demos in the past particularly with the ‘holocinema’ shown at this years sundance festival as seen below.  

The partnership is not surprising as Magic Leap technology fits well with the aims of ILMxLAB but it is interesting they did not choose to partner with Hololens considering that it is already out in the world in the form of dev kits.  Further, the partnership seems to be fairly deep.  Not only are they putting Star Wars content on the Magic Leap platform but they are setting up offices that will be jointly staffed by people from both organizations.


“We are Building a collaborative laboratory right in San Francisco, right adjacent to [ILMxLabs] offices where we will be able to work, create and co-invent the technology and creative that will power the future”


This laboratory solidifies how serious this relationship is.  It does not appear to be a simple PR partnership but a sincere effort to bring content to the Magic Leap platform.  This could be a great boon for the launch of the Magic Leap product.  Seeing a familiar and respected property on a product from a company no one will have heard of lends a credibility to Magic Leap it will sorely need.  Google Glass may have ushered MR technology along and in that sense it was a success but it poisoned the well in the process.  It is likely that the sentiment imposed by Google Glass will put a further burden on Magic Leap.  Hopefully, this partnership is the first step towards healing that public perception.
 

Talking to the GPU in your head

Magic Leap is a strange company.  There is really none other like it.   Between the eccentricities of the CEO to the enormous funding it has received, it is a unique snowflake.  Rony Abovitz, the aforementioned CEO, claims that they have found a way to talk directly to the “GPU in your brain”.  He isn’t joking or being hyperbolic. He seems to believe the words that are coming out of his mouth.  Is he crazy? Almost certainly.  But is he really crazy?

This is what I want to explore on this site.  Is this pie eyed company worth our attention or is it full of lunatics?  Perhaps both statements are true or perhaps we have missed the point entirely.

In future posts, I am going to put together the leading theories of what the product they are going to sell actually is.  I am going to dive into the minds of the investors to understand why this product is so valuable.  I am going to follow the latest news on what magic leap is currently up to.  

I hope to turn this site into the main portal in which to discover news and information about Magic Leap. 

If you are interested then join me to help write this blog. Leave a note in the comment section below.