DalyBulge: Hololens

"There was also a shark floating around outside in the lobby, and I walked over to the window and looked out at the shark, and the shark swam in the air down the lobby hallway on the opposite side of the glass, and the frame of the window hid the shark as it would if a real person walked down the lobby hallway. I shot scorpions coming out of a hole in the wall, It genuinely feels like it's really there and happening. As you walk around the room everything adjusts to your position, point, location."

These are some of the ways I have been trying to describe hololens to my friends. "How was it?" they ask. And I begin trying to describe to them this experience which isn't really like anything else. It's a bit difficult. Both explaining is difficult and capturing worthy video footage is difficult. It is only really beneficial if you wear hololens headsets; then it might make more sense.

As can be seen in this video, placing a camera to view the lens doesn't really capture the immersion that you feel when wearing the headset. People who don't really understand, will simply dismiss hololens based on this type of footage

I have been in London for a couple of months now, everything is starting to settle. My multiple jobs are going fine, my life is under control, and I have started to realise that London is a good place, somewhat like a library of events that you can google search for anything you wish to do at any given time and in likelihood there will be an event to fit whatever it is you are hoping for. In my case, last week I found myself with some free time and I thought about filling it. Instead of searching for what is happening (What'sOn, Time Out etc), I soul searched a little bit and thought about what I would really like to do? Space flight? Fight tigers in a cage? hmmm. What has my technology experiences been missing? I know! let's google search "Hololens" + "London" and sure enough I was excited to find that there is indeed a company who specialise in delivering hololens seminars and demos right here. So I jumped on the bus, wondered briefly, if I am about to sign up for something akin to a Black Mirror episode, and headed to London's Banking district.

(On my way to the privately owned enclave called: City of London)

It's interesting, the City of London. Most of us these days see it not only as the land of the rich and plenty, but also it is more and more becoming realised as a privately owned piece of land within London itself, with it's own emblem, and even with it's own private police force. It gets weird. Mostly because the City of London doesn't answer people's questions as to how they pulled this whole thing off.

But anyway back to Hololens. following a little bit of phone navigation, I arrive at the entrance to a really nice hotdesking office in Devonshire square, greeted by two very knowledgeable guys from a company called Kazendi; James and Max. On first appearance they seem very like every other very happy, enthusiastic and knowledgeable technologists out there, still with plenty of fire, a bit too immersed and jaded by the technology they have been swimming deeply in for the past number of years. Often gets deep does technology development. My first reaction is that they need to go on holiday.

Hololens.

Hololens is known as a mixed reality headset that combines 3d graphics into what we see through the headset and our real world. It's a bit like how some people describe some psychedelic experiences. Things appearing on walls, miniature sharks swimming down hallways, weight lifters bench pressing on our conference tables. And indeed it is true. After experiencing a demo of hololens you find yourself questioning existence and reality deeply for a while. Out in the street, I find myself asking "Is that pigeon in front of me real? or a graphic? How can I tell? If I can't catch it?" This is a possibility for our futures. It all gets a little bit 'Plato's Cave'. I wonder if anyone has lost their minds using hololens. The more real the games become, will the brain also have difficulty letting go once the headset is put down?

Here are some examples of hololens, which although impressive, doesn't give you the true immersive feeling that you experience when actually wearing the headset

There are a couple more nice video examples here and here too.
well...

There are of course a number of reasons why I wanted to learn more about hololens. It's generally a bit vague as to what it is, and also what is under the hood. It's not that clear at first glance. You have to dig to find out. Of course I wanted to actually experience hololens, just like riding a theme park ride, that's nice and easy. But learning more about the hardware technology is more rewarding for me, and also learning about development. Where it is going and how it is done.

The technology

Hololens it seems, is (in my eyes) a custom FPGA chip system. Microsoft is calling it a HPU. It's a Microsoft product so they can of course call it what they like. They can make it exclusive to microsoft windows if they like, and they can make apps only available through their app store if they like. That's a closed environment. And that is freedom to choose. But anyway, the FPGA/HPU thing is important. While developing drone hardware I learned some things about the value of FPGA chips over x86 or ARM chips. FPGA chips are able to handle multiple hardware peripherals much faster (think multiple web cameras, 3D Depth cameras, Gyroscopes, LIDAR, SONAR, all these things all at once can quickly make smoke blow out of a good old x86 processor. FPGA's are seen as the solution. I see them being used more and more on DJI drones such as the Matrice and it's camera sensing unit. Self-driving vehicles are using them too. DJI seem to be using Altera Cyclone V chips, Microsoft hololens seems to be using custom/disguised Tensilica FPGA, but I can't be entirely certain on that one, just based on observation.

*Update video above

But that's okay, just as much as open source hardware is great too (I'm an advocate). I like open source lots, it doesn't often put food in my belly though, and earning a buck is a nice reward to have for your development work.

(Look at all that connected stuff)

Speaking to Max and James, they offered some details about the headset, that I wrote down in my notebook. It has:
3D Audio (multiple surround speakers), Multiple microphones (6 I believe, to help noise reduction and isolation), A Bayer-filtered RGB(G) lens projector that handles at 240fps in total, 60fps per colour), SLAM mapping system using depth cameras and IR cameras (think Kinect/ZED/Intel Realsense camera). Snazzy..

However, If you have any experience at all using these types of cameras you will know and understand that there are constraints to SLAM imaging, for example, matt black surfaces seem to affect the imaging results, as does using outdoors (think Sun's Infra-Red light reducing the devices IR capability). But that is not to say these things are anything from functional, it's just currently a problem when handing these types of things to a novice consumer with high expectations, if they feel the device is not to their requirements they will rubbish something fairly quickly. See what is happening with drones when a novice quickly crashes/destroys their £1500 purchase!* (Hint: they are quickly hitting the refund button, and leaving very sour critical reviews). I think device technologists are learning this quickly. Developers have greater skills and acceptance of a new technologies' limitations and don't really mind as such, they enjoy it even.

What all these sensors mean to us, is that the hololens is able to react to world co-ordinates (gps, gyroscope, accelerometer sensors), gaze input (our sight direction), voice input, and gesture input. I know from my time with hololens that developers want to expand the current gesture availability, and to have the choice to develop hardware peripherals. Imagine a cricket bat with haptic feedback that reacts to a hololens cricket bowler?

(Touch Gestures)

During my Demo Experience

I experienced all the great demos that are currently available. There was this weight lifter on the conference table telling me how I should press weights (far from my body type could ever deliver). He was about 12 inches tall. I could resize him to life-size or miniature using gesture control and adjusting resize handles just like you do in Photoshop.

Drop and gimme 10 Pussy!!!! (Or I think he said that to me)

I shot some robots (roboraid) and scorpions which scuttled along the walls. I had a flash back moment to the horror-show spider head in Black Mirror. It might be for the best right now that Microsoft limits it's developers from doing this. I have some concerns over the potential abuse of hololens technology in the future which I won't go into too much, but imagine peddling immersive snuff movies, or interactive child pornography. I'm not wanting to be a party-pooper for hololens and it's wondrous technology, but I'm fairly certain there are some nut-jobs already conceiving things like this. General, high-quality, wholesome pornography however, is going to be quite outstanding though. Imagine your all-time favorite porn star wandering around your room with a twinkle in her eye. I swear right now, that in ten years time you're going to wish you were 16 again.

(Zap!!!)

Developing for Hololens

How might I as a non-hololens owner, develop software for it? That's a decent question to ask. Microsoft has in all fairness, released its SDK (Software Development Kit) freely to anyone who wishes to begin developing for hololens regardless of whether you own a unit or don't. You can download the SDK here. The SDK also has pre-requisite minimum requirements for a usable developer system which can also be read here. You will also need to consider using Unity - Game Engine to help you make nice graphics. But I am wondering if the new QT 3D Studio will do the job too.

There is also a nice developers page linking to forums, samples, codes, and tools here.

If you are not in London, you can find hololens demo locations here: International Demo
However, if you are in London, and wish to give this experience a try (which I think you should just for an understanding of where the future is leading to) I suggest you give these guys a call, or take a look here for more details on the London Demo. Makes a change from the Opera, and you will learn much more about software development too.