Sunday Musings: Karl Speaks and Rony Tweets

Karl Guttag has posted his analysis of the Magic Leap One based on the photos and information that has been released thus far. The effort and time he has put into this is impressive. He managed to deduce the likely dimensions of the device and 3D print a reference design. If you are interested in a deep dive on this then I recommend reading the article. But don’t let his negativity bring you down.  Read on for my thoughts on this.

Karl Guttag has posted his analysis of the Magic Leap One based on the photos and information that has been released thus far. The effort and time he has put into this is impressive. He managed to deduce the likely dimensions of the device and 3D print a reference design. If you are interested in a deep dive on this then I recommend reading the article. 

That said, and as per usual with Karl, he peppers the piece with a lot of negativity. Karl made up his mind about Magic Leap some time ago and it is reflected in what he chooses to write about and spend his time on. He leans heavily on negative language and hyperbole in a piece where the results can be interpreted under many different lights. With that in mind, his analysis is impressive and worth your time.

Is the Real World view that bad?

 Not even Bowie can make this look cool... Okay, maybe he still looks a little cool.
Not even Bowie can make this look cool… Okay, maybe he still looks a little cool.

I suspect Karl is fairly accurate in his analysis due to the time and effort he has put in. However, I don’t agree with his conclusion. When Karl first proposed that the real world view is going to be limited I suggested a quick experiment that is easy to do for anyone. Simply make a pair of glasses with your hands. Karl tells me that “The hands doing an “OK” symbol sort of gives the idea“. So it isn’t nearly as robust an experiment as what Karl has done but certainly gives a good impression.

After trying this myself (and, yes I did look like a horses ass while doing it), I must say that while obviously more view of the world is better, the limitations that Magic Leap will have do not seem that bad. You do feel a bit hemmed in but if it provides a compelling experience then it is worth it. I might think twice crossing roads with this limited peripheral view but you really don’t need much peripheral vision in your living room.  And that leads me to Rony. 

While some might think this tweet is dismissive, I think it is a good thing to remember. No matter how much you research, it is hard to say anything definitive about a product unless you try it. It is equivalent to the “spec” wars we see in other areas of consumer technology. In video games, companies will tell you all about 4k resolution and the sheer horsepower that their consoles have. While the Nintendo Switch is comparatively terrible in this respect. Prior to launch, it would have been easy to be dismissive of the Switch’s 720p screen and mobile hardware. You could ask, why would the smart people at Nintendo make such a bad decision? But once you try it as a product you realize the tradeoffs are well balanced and made for good reason. You see that consumers don’t care about specs. They care if it provides a compelling experience. You see that sum is greater than the parts. 

For Magic Leap, we don’t know if these tradeoffs will be worth it. Perhaps the view will be too limiting, or there will be too many people that Magic Leap doesn’t quite fit. On the other hand, a restricted real world view will make the smaller digital field of view feel larger and less irritating. In this sense, it could improve the experience. One could also question if these tradeoffs were made to fit in the myriad of other features Magic Leap is reported to have. Features that move this away from a lab experiment and in to a product. Perhaps Magic Leap One, as a dev kit, will be a bit wanting but the first consumer release will get the balance right. I don’t know the answer to any of these questions. Importantly, neither does Karl.

So while Karl calls Magic Leap “terrible”, “visual garbage”, “poor”, having a “major design flaw”, “Obviously wrong”, “Bad” and so on, remember, he has a bit of a chip on his shoulder as is clear from his choice of language. He doesn’t know how good or bad Magic Leap will be as a product. I’d also caution Karl on calling out Rony as making a “logical fallacy“. Rony is making a good point about judging something before you have actually tried it. That kind of judgement means you are making a hasty generalization, a logical fallacy that should be avoided. 

In any case, Karl’s analysis is interesting. It does speak to the experience we will have with the Magic Leap One and worth your time. My thoughts here aren’t to say Karl is wrong. When we all get to try the device later this year we might find he was dead on. We just don’t know yet. So try to filter out the negativity, there is enough of that in the world already.

What is up with the Magic Leap website?

It looks like Magic Leap has more secrets than just their device. On the Magic Leap website there is a trail of floating leaves that follow you as you scroll down. One of these leaves is unique though.  What does it mean?

It looks like Magic Leap has more secrets than just their device. On the Magic Leap website there is a trail of floating leaves that follow you as you scroll down. One of these leaves is unique though. Have a look at the GIF below:

Did you see it? 

Towards the bottom, just left of the middle?

What’s up with that leaf?

In fact when you reload the page you get a different pattern. 

I have no idea what it means, but it certainly doesn’t look like an accident. Anyone out there got any ideas? There appear to be only 4 symbols repeated in different orders. Start sleuthing folks!

We only have half of the Magic Leap SDK

Recently, Magic Leap released its SDK to developers so they can start playing with what is possible on the Magic Leap platform. With these tools you can develop a full Magic Leap application to run on the device when it is eventually released. But the documentation and toolkit is incomplete. There is a huge section that is still yet to be released to the public. 

That is, the Lumin Runtime. And it could completely change how you think about apps.

Recently, Magic Leap released its SDK to developers so they can start playing with what is possible on the Magic Leap platform. With these tools you can develop a full Magic Leap application to run on the device when it is eventually released. But the documentation and toolkit is incomplete. There is a huge section that is still yet to be released to the public. 

That is, the Lumin Runtime. And it could completely change how you think about apps.

Making an App

When you go to make an app on Android or iOS you have to choose the tools you want to use to make that application. There are 3 main choices: A Web App, using the official toolkit, or going native.

These days, web apps are so powerful that in most cases this is a good choice for developers. But if they need a bit more, they can use the official toolkits provided by the platform. This is using Java or Kotlin in Android and Objective-C or Swift on iOS. Most Apps are developed this way. But some apps, mostly games, need direct access to the hardware and need to more carefully manage compute resources. Platforms expose a specific SDK for this. On Android this is called the NDK or native development kit. 

What Magic Leap has released is the equivalent of Androids NDK. It has yet to release its application toolkit. This is important as there is a good chance that most applications made for Magic Leap will be written using this toolkit and not the 3D engines already released.

Lumin Runtime vs. 3D Engines

The Lumin Runtime sits beside the 3D engines in the Lumin OS stack. The SDK for these 3D engines are released already but we only have one short section on what the lumin runtime will be in the docs. It seems this runtime is geared towards developers, such as web developers, who might not be as comfortable working with game engines or starting from a blank canvas. It is for simple applications. For example, making an email client from scratch in unity or unreal is quite a daunting proposition for most developers. The Lumin Runtime helps developers by giving them a framework to use. It will provide a common design language for developers and help them navigate the challenging task of developing applications in an unknown 3D space. 

Throw out the App Model

One other detail about the Lumin runtime is that it is built for multiple applications to run at the same time stating: “It is capable of running and rendering multiple native applications simultaneously.” This is not true for the 3D engines which appear to take over the whole device. 

Traditionally, on mobile hardware, applications are installed and to use them you have to open them up and run them. One at a time. On android you have some more flexibility with this but generally most applications stick to this model. I believe (watch me wildly speculate again!) that Magic Leap will be a bit different. You won’t click a grid of icons to open an app but an app will live in a physical place somewhere. Let’s say you have a word processing app. It will live at your desk. To use it you can just walk up to your desk and it is there. A television app will live on the wall. You install it like a real TV. You might stick a mapping application to your front door so you can grab it on the way out. Perhaps have a recipe app beside your stove. The point is that you won’t need to scroll through your app drawer and find an icon to click on, the app will just be where you left it in physical space. This is all speculation on my part but I think it makes sense. 

The documentation says that the runtime will “facilitate application sharing out of the box, allowing application developers to collaborate with multiple users.” So if I install a TV app on the wall, everyone will see it. It is persistent and as part of the world as a real television. I could “hang” digital art on the wall and everyone who comes to my house would see that art through their own set of ML glasses. 

This is a radically new way to think about applications and computing in general. It is fundamentally tied to physical space and multi users. It truly is augmenting reality and I think one of the more exciting aspects of what Magic Leap is doing. 

How Magic Leap Works: SDK Edition

Some time ago I wrote about how I thought Magic Leap might work based primarily on patents that were published by the company. While patents do give some indication of what the company is working on, they do not reveal what features a product will have. At the time, that list of features didn’t exist. 

It does now. 

Some time ago I wrote about how I thought Magic Leap might work based primarily on patents that were published by the company. While patents do give some indication of what the company is working on, they do not reveal what features a product will have. At the time, that list of features didn’t exist. 

It does now. 

  • Audio: Stereo audio output and voice microphone recording are supported.
  • Camera: Capture still images and videos from the color camera.
  • Dispatch: Allows apps to open URLs using the Magic Leap Browser or other apps.
  • Eye Tracking: Ability to retrieve fixation point position and eye centers. Blinks can also be detected.
  • Graphics: OpenGL ES, Desktop, and Vulkan rendering paths.
  • Hand Gestures & Key Point Tracking: Recognize the user’s hand poses (gestures) and track the position of identifiable points on hands such as the tip of the index fingers.
  • Head Tracking: Headpose is tracked in full six degrees of freedom (DOF).
  • Image Tracking: Track the position and orientation of specified image targets in the user’s environment.
  • Input (Control / MLMA Support): Retrieve either 3 DOF (orientation) or full 6 DOF (position and orientation) from the Magic Leap Control. Detect button and touchpad presses and the analog trigger. Trigger values range from 0 to 1. A range of touchpad gestures are also supported, as are haptic vibration and LED ring feedback. This interface works seamlessly with both physical Magic Leap Controls and the Magic Leap Mobile App.
  • Light Tracking: Provides information (luminance, global color temperature) about the ambient light of the user’s environment.
  • Media Codec: Low-level, hardware-accelerated media encoding and decoding.
  • Media Player: Simple, straightforward media playback interface.
  • Meshing: Converts the world’s depth data into a connected triangle mesh that can be used for occlusion and physics.
  • Music Service: Supports connecting and listening to a streaming music service.
  • Occlusion: An interface for feeding depth data to the Magic Leap platform for hardware occlusion.
  • Planes: Recognize planar surfaces in the user’s environment for placing content. This includes semantic tagging for ceilings, floors, and walls.
  • Raycast: Fire a ray and get the point of intersection with the world’s depth data.
  • Secure Storage:Save data from your app to the device’s encrypted storage.

From this list, and other information in the developer documentation, we can make some better guesses at what the Magic Leap One will be capable of doing and what it will struggle with.

Eye Tracking

Depending on how well it works, eye tracking is going to be a big part of using Magic Leap. Without a screen to touch or a mouse to move, selecting things in Mixed Reality has always been challenging. Gestures are not as accurate as you might like and using headpose means a lot of head moving. Being able to just look at something and then either blink or do a small gesture will be a revolution in this space. As a real world example you can see what Eyefluence (a company recently purchased by Google) has done in this space. We don’t really know how it works but they don’t even need blinks to confirm actions. If this level of eye tracking is in the Magic Leap One then it will be heavily used for user interaction

Hand Gestures

Magic Leap One will support 8 hand gestures. This is notably more than Hololens. These gestures in concert with eye tracking will form the basis of interaction with magic leap. At least for applications that don’t use the controller

Control

The Magic Leap controller is fascinating. It has the basics you might expect: touchpad, home button, trigger, etc. But it also features 6dof tracking. We do not know how accurate this tracking will be but to provide orientation as well as position is quite challenging using accelerometers alone. As we have written about in the past, there are indications Magic Leap is using magnetic tracking to facilitate this. The protrusions that stick out the side of the lightwear might well house the magnetic coils necessary for this tracking. If this is the case, we may have tracking accuracy comparable to the Vive but without the need for clunky external cameras or lighthouses. 

Another thing revealed in the docs but not outlined in detail is the existence of a Magic Leap mobile application. Presumable for Android and iOS, this application can be used as a controller for Magic Leap. The documentation implies that it will transfer touch inputs from the phone to Magic Leap likely working like the trackpad on the controller. I think this might point to the controller being sold separately as you can use your smartphone if you don’t want to spend the extra money. It will be interesting to see what the mobile app will look like.

Occlusion/Depth Mapping

Occlusion is supported in Magic Leap directly in the API. Occlusion is a notoriously hard problem that no one has quite got right before in a consumer device. It requires accurate depth mapping that both project tango and hololens struggled with. That said, there are good examples of occlusion working in those devices for very specific circumstances. I suspect Magic Leap is doing some amount of guesswork, potentially using machine learning, to make better guesses at hard edges for occlusion. 

The documentation points to Magic Leap using IR based depth sensors similar to those used in Project Tango and Hololens. This means they will struggle to work in bright sunshine and on dark materials. 

The documentation makes it clear that this will not work for dynamic, moving objects. It states:

The environment is expected to mostly be static and change (such as objects moving or people walking in front of the device) only happens occasionally. For small changes, the reconstruction should slowly update over several seconds to add new objects or remove moved objects. Environments with significant or continuous change may lead to holes or incorrect geometry.

This means occlusion won’t work if someone steps into your field of view. Digital objects will appear in front of them. Hopefully, in future iterations this can be improved. 

Image Tracking

Image tracking is built directly into the Magic Leap SDK.  This allows Magic Leap to identify a given image and track its position in the real world. This isn’t a feature I was expecting to see at the API level. I can envision an application that takes artwork on your wall and bring it to life. Below is an example of this, though not running on Magic Leap. Warning, the audio in this video is intense.

ARCore/ARKit Parity

The SDK also includes features such as plane detection and light tracking. These are sort of table stakes for an AR toolkit at this point. 

Lumin OS

Magic Leap will run a custom operating system called Lumin OS. Lumin is based off of Android but heavily modified to support the requirements of spatial computing. This is good news for developers as it means the platform will be familiar for anyone with experience working with android or working with linux in general. It also means that Magic Leap is not trying to reinvent the wheel as it will leverage the years of work that has gone into the android open source project. 

One thing we can see in the listed features above is something called Dispatches. This sounds like a renaming of Androids Intent system. This is the system that allows you to select certain apps to use for different actions, like selecting a default browser. While it remains to be seen if Magic Leap will allow users to select default applications or if they will force users into their in house apps, this is a good indication that we might get more choice on this platform than some other notable platforms.

How well does it work?

The documentation gives us a very good idea of what to expect from Magic Leap. We pretty much know what this thing is going to be at this point and on the surface it looks amazing. No device has had this range of features in such a small form factor before. But the question that this does not answer is how well does it work. Is the controller tracking accurate? How is the image fidelity? Is eye tracking accurate? What kind of performance can we expect from the GPU/CPU? You can have all the features in the world, but if they perform poorly it won’t matter. We’ll have to wait for the device to come out later this year to find out.