The inspirational power of Rony Abovitz at Harvard Hubweek

Rony Abovitz arrived to the talk by remote control telepresence bot to join Jon Hirschtick on stage. The ‘uniqueness’ of the appearance wasn’t limited to just the use of telepresence, though. The event highlighted both a longstanding friendship and a nascent partnership between the two CEOs and their companies. OnShape is a leader in the Computer-Aided Design field with their Cloud platform for collaborative 3D design, and Jon Hirschtick himself was a founder and CEO of SolidWorks, one of the leading modeling and design tools in multiple industries. It was quite a special event to see two people with nearly a twenty year friendship, both pioneers and leaders in their fields, talk to each other about the future.

David Harrison got a chance to see Rony Abovitz speak “live” at harvard hubweek yesterday.  By “live”, it was really via a telepresence robot but that is close enough to the real deal. He wrote of his experience here  as well as uploading a video of the talk.  He has graciously allowed me to repost both his thoughts and his video which can be found below. Thanks David!

My first impression of the event was that it was empty, but just five minutes later the lobby was packed as everyone started to show up. I was just way too early, it seems. The atmosphere was even more lively and enthusiastic then. I met a few people who were ‘looking into’ AR and VR for the first time, many startups and students, too. The demo hall had many local Boston-area companies, and the Vive was on display everywhere. HTC were there with a few demos (theBlu, for one) and NVidia brought their Funhouse with them. One of the more impressive things on display was a bouncing 360-degree video camera sphere for use in tactical scenarios, and it was paired with a 360-degree VR viewer. It’s easy to see, though, why Rony Abovitz had to start the chat off with a discussion of the definition of VR, AR and MR.

Rony Abovitz arrived by remote control telepresence bot to join Jon Hirschtick on stage. The ‘uniqueness’ of the appearance wasn’t limited to just the use of telepresence, though. The event highlighted both a longstanding friendship and a nascent partnership between the two CEOs and their companies. OnShape is a leader in the Computer-Aided Design field with their Cloud platform for collaborative 3D design, and Jon Hirschtick himself was a founder and CEO of SolidWorks, one of the leading modeling and design tools in multiple industries. It was quite a special event to see two people with nearly a twenty year friendship, both pioneers and leaders in their fields, talk to each other about the future.

Jon Hirschtick opened with praise for Rony Abovitz’s successes in his last company and with Magic Leap. He shared that he was overcome by a feeling that Rony would go on to do great things, even twenty years ago and especially now at Magic Leap. He says of Magic Leap:


“I truly believe that what Magic Leap is doing has the potential to improve the way every product on earth gets designed, manufactured and produced.”


His full comments are in the video, and you should watch them. As he was asked about the current generation of devices, Rony Abovitz responds with a ten to fifteen year plan that works with the current generation of technology to enable even more growth later on. He shared a vision of a cross-platform, cross-technology world, where smartphone users in developing countries can still serve as contributors to projects made for Leap, and the proliferation of the last generation of devices in the rest of the world will let people’s of poorly-developed countries reap investment and employment globally, despite not having access to Magic Leap. This vision and plan falls right in line with most industry analysis, where slowing or stagnant smartphone growth in the first-world is pushing the industry towards producing budget handsets for poorer countries where there are still many billions of people to connect to the internet and global economy. The challenge after that then is to connect them to Leap as those older devices become truly obviated.


It’s one thing to read and analyze and over-analyze (and yes, write and post), but to really see the trust, faith and admiration that Rony Abovitz has inspired in someone else was something else entirely


OnShape, and other industries, are looking at new opportunities with Magic Leap (a smooth transition), not a disruptive revolution, despite the obvious superiority of Magic Leap. It’s an impressive vision, and the confidence and trust in Rony Abovitz to execute it was on full display today and it was a little contagious. Throughout the talk, I couldn’t help but be moved by the same feeling. It’s one thing to read and analyze and over-analyze (and yes, write and post), but to really see the trust, faith and admiration that Rony Abovitz has inspired in someone else was something else entirely. The plan sounds like a pie-in-the-sky idea, but it’s OnShape that grounds it, already producing collaborative CAD applications for devices operating in 2D and now partnering early with Magic Leap to make it happen in the natural, spatial way Magic Leap enables. It’s a promising partnership and a direct execution of that plan, and it’s easy to see where that trust comes from.

I had a chance after the presentation to talk to Shaun, an original employee from the time when Magic Leap had only 90 employees. He gave me as good an impression of someone who’s seen the future as anyone could, and he showed that same trust in the Magic Leap vision as I asked him about things he couldn’t talk to me about. Working in software and hardware at Magic Leap, he couldn’t answer a lot of my questions with regards to Human Factors and interaction, either, but he confirmed to me that while the robot shooting game video wasn’t shot through Leap at the time, it is real and he has played it (and it’s really cool, and more than robots). He mentioned something about the ML audio system, but I don’t want to get him in trouble if he wasn’t meant to.

On a side note, Magic Leap reads the /r/magicleap subreddit. As I approached Shawn to introduce myself, he said, “I already know who you are.” I can’t imagine the shade of red I went! He had some great advice on where to start, and Unity and C, C# and Python are all great, but their job postings are the best source of information for figuring out what’s going on inside. Bootcamps and demonstrations, while discussed in the presentation, aren’t in place yet, but that will open the floodgates for development.

On a personal note, I was most impressed with the presentation. The content was visionary, but as I mentioned above, the real life demonstration of the trust and belief in Rony Abovitz and Magic Leap by Jon Hirschtick was most powerful. The quiet from Magic Leap has bred in me, at least, a wild speculative nature that ignored the people who believe in them, their vision and technology. There’s a thousand of them who call Magic Leap their employer, and dozens more in partners and investors that are betting the future of their businesses (at least a portion of it) on Magic Leap and investing in Rony Abovitz. That kind of impression is not one left easily, and I have to say I came out with a small divot myself. Listening to Shawn talk about Magic Leap (what he could, at least), I got the impression, too, that there’s a lot of brilliant people at Magic Leap working on everything I can imagine from sensors to chips to a platform. I’ve not been very fair on Magic Leap lately (partly because of the deafening silence!), but I think this event was made even more impactful by highlighting the people who share Magic Leap’s vision and have trust and confidence in the execution of that vision. I suppose seeing is believing. Now that’s a great motto…

For more from David, he can be found making insightful posts on reddit under the user David_Harrison.

Magic Leaps privacy predicament

Privacy always seems to be top of mind in the tech industry. It is a touchy subject and it is hard to have a nuanced conversation about.  There are vocal people on both sides of the privacy argument and separating economics and emotions from reality can be challenging.  Further complicating matters is that we, the public, constantly change what we are comfortable with while companies are constantly redefining what privacy even means.  Yet I would argue that all the ink spilled on the subject to date is but precursor to what will happen if Magic Leap is successful.  The potential for, and necessity of, data collection by Magic Leap will far exceed what any company has done in the past.  Magic Leap with have to tread carefully as it brings us into their future.

Privacy always seems to be top of mind in the tech industry. It is a touchy subject and it is hard to have a nuanced conversation about.  There are vocal people on both sides of the privacy argument and separating economics and emotions from reality can be challenging.  Further complicating matters is that we, the public, constantly change what we are comfortable with while companies are constantly redefining what privacy even means.  Yet I would argue that all the ink spilled on the subject to date is but precursor to what will happen if Magic Leap is successful.  The potential for, and necessity of, data collection by Magic Leap will far exceed what any company has done in the past.  Magic Leap with have to tread carefully as it brings us into their future.

Google Glass: a case study 

Luckily for Magic Leap, Google has already tested the waters. We have all heard the hullabaloo about privacy in regards to Google Glass.  Many attribute the Camera on Google Glass as the reason for Glass never became a consumer product. While it was an easy thing to point at, and certainly didn’t help matters, Glass had problems beyond the camera.  First and foremost was utility.  You couldn’t actually do all that much with Glass. Of the things you could do most were accomplished more efficiently through your phone.  When you couple that with an exclusive purchasing program and exorbitant price tag, you create an exclusionary item whose main attribute is to aggressively point out how rich and connected you are. Of course people are going to rebel against that anyway they can. 

For Magic Leap, they need to ensure that they don’t create an elitist product.  If only a select few “leapholes” can either afford to buy or are able to buy the product, I predict you get a similar backlash. It will not be possible to discreetly wear Magic Leap meaning you will stick out like a rich, snobby sore thumb while wearing the device. In this case it will be easy to point at privacy as part of the evil that is Magic Leap.  On the other hand, if it does not get the reputation for elitism, people will be far more forgiving of other issues. 

The main way to avoid this problem is for Magic Leap to create a product that is so compelling that people are willing to accept it into their lives and, literally, on to their face.  That is a big ask. If it is delightful to use and provides genuine utility, the issues Glass faced will be side stepped. 

But what about that camera?  Magic Leap will have some sort of recording device always on while under operation. They need it to do the simultaneous location and mapping required to project holograms. There is a good chance that information will be streamed to the cloud so that the world models you build can be passed to other users for shared MR experiences.  Is this going to be a problem?  

I’m not sure.  I am sure the tech press is going to talk about it ad nauseam but I am not sure the consumer will care.  We have seen repeatedly throughout the history of technology that privacy issues are often put aside when the utility provided is so great it outweighs the privacy issues presented.  When the first phone camera hit the market there were a litany of stories proclaiming the end of privacy. Consumers didn’t seem to mind though. Having a camera on the thing you take everywhere with you was just too useful. People were willing to give up a rather nebulous claim that our privacy is being violated to get the utility that a camera on a phone provides. Over time this has become so completely normalized it is now somewhat odd to not have a camera on you. And those nebulous privacy concerns? No one seems to care at all anymore. They have moved on to the next privacy issue.

What I’m leading to is that most people do not care about privacy.  Don’t get me wrong, they care about clear cut privacy invasion cases. If some guy starts taking photos through your bedroom window you are certainly going to care about that.  But if there is some distant issue that could become a privacy problem one day in the future somehow, then people simply do not care or at least will accept it once it has become normalised. 

Snapchat

Speaking of normalised. Snapchat are making a pair of glasses with a camera on it. I don’t expect these to be particularly successful but this is great news for Magic Leap.  The road to normalising a new technology that could be perceived as invasive is rocky and requires people to have lived with the tech for some time. These glasses will help them along that road and can further teach them lessons on how to present their product to the public. Who knows, by the time Magic Leap launches, it might be part of a whole stable of products with face mounted cameras.  We certainly put Gopros on our head from time to time. Magic Leap might get to sidestep the issue entirely though I don’t think this is likely. 

It is a fine line that Magic Leap will have to walk and there is certainly going to backlash against their products. I don’t imagine a single review of their first release not mentioning the privacy issue. How they manage this is based largely on how good the product actually is and if their PR team is up to the challenge. But I challenge those that think it is insurmountable. While some care about maintaining the privacy status quo.  Most, do not. They just want a cool and useful new toy.

Magic Leap field of view “far superior to hololens”

According to a former blackhat hacker that goes by Gummo, the field of view of Magic Leap is “far superior to hololens”.  While this should not come as a surprise it is nice to hear from someone who claims to have tried the technology.  It is also interesting to see who Magic Leap is showing this technology to. It is clear Magic Leap is concerned about security and have shown this through their sole acquisition of Northbit security company.  The fact they are giving demos and likely working with security experts such as Gummo bodes well for the security of the platform.

According to a former blackhat hacker that goes by Gummo, the field of view of Magic Leap is “far superior to hololens”.  While this should not come as a surprise it is nice to hear from someone who claims to have tried the technology.  It is also interesting to see who Magic Leap is showing this technology to. It is clear Magic Leap is concerned about security and have shown this through their sole acquisition of Northbit security company.  The fact they are giving demos and likely working with security experts such as Gummo bodes well for the security of the platform.

Kevin Spacey and Magic Leap: A quest for content

Looks like Kevin Spacey is a Magic Leap fan.  In a since deleted tweet from a Magic Leap employee, we see President Underwood sporting a Magic Leap branded phone case.  What does this mean? What revelations have been exposed? Is Usual Suspects 2 confirmed as a Magic Leap exclusive!?

Looks like Kevin Spacey is a Magic Leap fan.  In a since deleted tweet from a Magic Leap employee, we see President Underwood sporting a Magic Leap branded phone case.  What does this mean? What revelations have been exposed? Is Usual Suspects 2 confirmed as a Magic Leap exclusive!?

No, clearly it means basically nothing. All we may suppose is that Spacey has seen a Magic Leap demo and found it so compelling he was willing to brand his phone with the logo. Spacey has long been a big proponent of technology so this is hardly surprising. 

It does bring up the conversation about content.  Magic Leap knows that for its product to be a success it needs to bring compelling content to the platform.  This is just a further example of Magic Leap reaching out to content creators. Given House of Cards predilection to display interesting mobile games, perhaps next season we will see President Underwood sporting a Magic Leap headset.  

The field of view question

Many readers took issue with my claim of Magic Leap providing a 90° field of view and rightly so.  It is the most speculative statement in a speculative article and there is a good chance it is grossly wrong. It is particularly hard to reconcile with the quote stating that it would require 50 Mpx to produce a high field of view display.  50 Mpx is absurdly large and is far beyond anything we are capable of today.  So how can I stand by my statement? By reading more patents of course.

Not long ago I published my thoughts on how I think Magic Leap will work based on a number of sources but primarily on this patent.  In the previous article, I make some bold claims about resolution and field of view that, while based on information from that patent, is largely speculative.  To be honest, the whole article is largely speculative considering patents do not equal products.  As one reader put it, I built my description of the technology “by reading patent applications and other tea leaves”.  While admittedly flawed, these documents are the best information we have to make guesses at the coming products out of Magic Leap so I’m diving in again to see if I can put more weight behind some of these claims.  

Many readers took issue with my claim of Magic Leap providing a 90° field of view and rightly so.  It is the most speculative statement in a speculative article and there is a good chance it is grossly wrong. It is particularly hard to reconcile with the previous quote stating that it would require 50 Mpx to produce a high field of view display.  50 Mpx is absurdly large and is far beyond anything we are capable of today.  So how can I stand by my statement? By reading more patents of course.

Why is field of view hard?

This patent describes in further detail many different configurations Magic Leap might employ to create their head mounted display.  In particular it has lengthy sections on the difficulties of chromatic aberration, exit pupil and, you guessed it, field of view.  

Why is field of view so challenging? This Magic Leap patent gives us some insight into the problems involved.  From the patent: “In a conventional waveguide approach wherein total internal reflection is relied upon for light propagation, the field of view is restricted because there is only a certain angular range of rays propagating through the waveguide (others may escape out)”. So light bounces around in a waveguide at a low angle (think of a glancing blow off of the edge of the glass) and when it achieves a high angle it escapes and we can see it.  Towards the edge of the waveguide you need a low angle to be able to aim the light at the eye.  That low angle will cause reflection and not the desired transmission so it cannot escape and cannot contribute to the field of view.

 The top image shows total internal reflection happening.  All the angles hit by the light beam are low so they are reflected within the waveguide.  The second image has a mirror to redirect the light beam to hit the waveguide at a high angle.  This allows it to exit and such that we can see it.  The bottom image shows light escaping the waveguide and aimed at an eye.  The light near the center of the waveguide has a high enough angle to escape and hit the eye.  The light near the edge of the waveguide, the would-be dotted lines if it could escape, needs too low an angle to be aimed at the eye.  It cannot escape and continues to reflect instead of contributing to the field of view.  This is a major limiting factor for field of view. 
The top image shows total internal reflection happening.  All the angles hit by the light beam are low so they are reflected within the waveguide.  The second image has a mirror to redirect the light beam to hit the waveguide at a high angle.  This allows it to exit and such that we can see it.  The bottom image shows light escaping the waveguide and aimed at an eye.  The light near the center of the waveguide has a high enough angle to escape and hit the eye.  The light near the edge of the waveguide, the would-be dotted lines if it could escape, needs too low an angle to be aimed at the eye.  It cannot escape and continues to reflect instead of contributing to the field of view.  This is a major limiting factor for field of view. 

Given this fundamental constraint, increasing field of view is challenging but possible.  The patent outlines a number of methods to increase field of view and I honestly don’t fully comprehend most of them.  I will try to relate what I can decipher and give you hints at the other solutions but I don’t think I can give you a complete description.

Tiling

To overcome the issue of field of view, Magic Leap appears to be employing a tiling approach. The patent mentions many methods which may be used to tile images to generate a larger field of view.  From the patent, “a narrow field of view sub-image which is tiled with other narrow field of view sub-images presented by the other reflective surfaces to form a composite wide field of view image” and “wherein 6 sub-images are to be presented to the eye frame-sequential to form a large tiled image”.  Keep that number 6 in mind for later as I think it is telling. The methods for tiling range from using a number of reflective surfaces that can be turned on and off (similar to the DOEs explained in my previous article) to using “freeform” optics in ways that I can’t fully comprehend.  Tiling is conceptually quite friendly as you can imagine using 6 different scanning projectors to provide 6 different sections of the images that take different optical paths to your eye allowing an expansion of field of view. 

Freeform Optics


At no time since the science of optical system development was created in 1885 has the industry been so technologically dynamic.


The patent has an entire section on the use of freeform optics.  Freeform optics appear to be any sort of optical interface that does not conform to simple shapes such as spheres, parabolas or cylinders.  While this sounds fairly simple it means the surface geometry must be carefully calculated and manufactured.  Optics people seem very excited about the prospects of freeform optics stating “At no time since the science of optical system development was created in 1885 has the industry been so technologically dynamic.”. For Magic Leap, this means they can produce specially constructed lens arrangements to generate a larger field of view.  As the patent states: “With freeform lenses, rather than having a standard planar reflector, one has a combined reflector and lens with power, which means that the curved lens geometry determines the field of view … it is possible for a freeform arrangement to realize a significantly larger field of view for a given set of optical requirements”.   It appears that these optics don’t make use of total internal reflection (or at least don’t use it exclusively) so they can avoid the fundamental limitation discussed above.  

The patent goes on to indicate a combination of tiling and freeform optics may be used.  In particular, it references 2 different tiles but goes on to show an image with 6 potential freeform wedges driven by 6 different displays.  

Unfortunately, the science of freeform optics appears to be complex and hard to digest.  The surface structure is defined by numerically solving complex differentials and then fitting high order polynomials to the solution.  I wish I understood the concepts more thoroughly but regardless of how it works it does appear to be causing a revolution in the optics space and will give Magic Leap a much needed tool in generating a wide field of view.  

Okay, time to wildly speculate

In a previous patent we get this quote: “To best match the capabilities of the average human visual system, an HMD should provide 20/20 visual acuity over a 40° by 40° FOV, so at an angular resolution of 50 arc-seconds this equates to about 8 megapixels (Mpx). To increase this to a desired 120° by 80° FOV would require nearly 50 Mpx”.  The patent then ignores this statement and goes on to describe a system that provides 8 Mpx across a 40° field of view. This has always seemed odd to me.  The start of the patent explicitly says that a small field of view is bad and that a desired field of view is 120° by 80° with nearly 50 Mpx.  Why say this? Why pick those particular numbers especially if you can’t achieve them?  

I think there is a chance they have achieved them. If you were to tile the 8 Mpx display described in a 3 by 2 configuration for a total of six tiles you would get a resolution of 48 Mpx and a FOV of 120° by 80°.  The scanning fiber display itself takes up the space of ~1cm in height so it would only take 3cm on either side of the lens.  This is well within the size constraints of a typical pair of glasses. The idea is in the patent only it isn’t state expressly. It often describes using multiple fiber scanning displays in many distinct arrangements.

 So many scanning fibers.  So much resolution.  So much field of view. Is this at all possible? Probably not but a guy can dream right?
So many scanning fibers.  So much resolution.  So much field of view. Is this at all possible? Probably not but a guy can dream right?

There are other issues, such as no current hardware being able to drive such a display, but I think many of these problems can be overcome by the fact that most of the display will be dark.  Magic Leap does not have to push every pixel, only the ones required for the current scene.  Further, Magic Leap will certainly make heavy use of foveated rendering.  At any given time only 8 Mpx worth of pixels will likely be illuminated while the rest of the display is idle. Because of this I speculate Magic Leap will not be able to be used as a VR headset or it won’t be able to at a theoretical resolution of 48 Mpx.  It won’t be capable of filling the entire field of view with high resolution images.  It will always be limited to a subset of the scene or lower resolution. How much of the scene it is able to fill might be a crucial metric that we look at in future iterations of this sort of product.   

I know this is massively speculative and likely not true.  I know it even sounds a little bit crazy but it does line up with what we have heard from people who have tried Magic Leap.  Many have claimed it is not possible to see pixels or the dreaded “screen door effect”.  This implies a high resolution. The field of view is clearly under NDA. This means they are hiding it either because it is bad or because they want to surprise people with it. We have consistently heard that people are blown away by the technology and these are the same people who grumble about hololens due to poor field of view. I think this is evidence that they simply want to surprise us.

Further evidence is the reaction to my previous article where I postulated a 90 degree FOV.  

Andy Fouché is the head of PR at Magic Leap and Kevin Kelly is a wired editor who has personally tried Magic Leap.  I did say quite a bit in my previous article so they might have been ignoring my estimation on field of view but I believe it does lend credence to the theory.  

In any case, it is clear that Magic Leap is working to provide the highest resolution and field of view that it can. I don’t want to get peoples hopes up too high.  If I were you, I would expect something closer to 50 degree field of view and don’t let the hype train sweep you away to inevitable disappointment. That said, it is fun to speculate about achieving a 90+ degree field of view.  If they have produced this then they will have made a technical jump that is unprecedented in the industry of consumer electronics.  Unprecedented in the same way their funding is unprecedented. It is okay to get a little excited about that possibility.