Setting up an application environment for HoloLens

Since some time now I’m playing around with HoloLens. There are already some cool and nifty applications built which allow you to enjoy playing a game in your room, design holograms, create videos or learn from information programs like the human body for medical students and about dinosaurs for children. Playing around will give you tons of ideas. But how do you transform those ideas to an actual app?

This post (and some other ones in the future) will help you started developing applications for HoloLens. The first part is to explain to you where to start. What do I need to start developing an augment reality app? Building an augment reality app for the HoloLens device can be done in two ways.

1. Using Visual Studio – Create you mixed reality application for HoloLens by using DirectX11, SDK and a lot of coding
2. Using Unity with Visual Studio – Microsoft and Unity have been working together to deliver you the tools to create mixed reality applications for the HoloLens.

Since the second option gives you the tools to create mixed reality applications, it’s far the best choice to start with. Still you will need to consider that there is a learning curve to understand Unity.

HoloLens application

A HoloLens application is a Windows Universal App. It can be deployed to the HoloLens emulator or to the HoloLens device. The application can be added to the Microsoft App Store and made available to others.

Requirements

In principle, you will need to have the following:

1. A machine running Windows 10. Make sure your machine has at least an i5 core processor and 16GB of memory. Less than that will influence performance and shutting down one application to start another one. For example, running the HoloLens emulator together with Unity still open on an 8GB machine will not work.
2. Visual Studio 2015 Update 3 or higher
3. Microsoft HoloLens emulator. This will include the Microsoft HoloLens emulator but also the Microsoft HoloLens App Templates for in Visual Studio which are needed when you develop only with Visual Studio.
4. An actual HoloLens device is recommended. There are some things which are tricky or not possible with the HoloLens emulator. I will explain these differences in a separate blogpost.

If you want to use Unity you will be needing Unity for HoloLens. Unity allows you to create 3D objects and bind them to code using Visual Studio. You can get a free license for Unity which gives you enough to get started. The free license does not allow you to change the startup logo of your application and some other small things.

Make sure that you have the right version of Unity. Unity for HoloLens is currently in technical preview. Be aware of using this version and not the standard Unity application version. I have already seen with others that creating mixed reality applications with the wrong version will cause some strange errors and if your application shows up in HoloLens it is in a 2D frame.

HoloLens application

A HoloLens application is a Windows Universal App. It can be deployed to the HoloLens emulator or to the HoloLens device. The application can be added to the Microsoft App Store and made available to others.

Setting up an application using Visual Studio

The first option which is available is solely using Visual Studio as a development platform. Currently, there is only one template available when you have installed the HoloLens Emulator package. You will find the template under Visual C#àWindowsàUniversalàHolographic.

The template used SharpDX which is an open-source managed .NET wrapper for the DirectX API. After the project is created we have a Visual Studio solution containing some initial code for displaying a 3D box using rainbow colors which spins around. When you check the NuGet packages you will notice that there are multiple DLL references to SharpDX to support DirectX11.

And that all there is. Now you need to start developing your holographic objects in c# code.

Setting up an application using Unity for HoloLens

To create a HoloLens application using Unity you will need a special edition of the software. This software is still not in production. The current version is called Unity HoloLens 5.4.0f3-HTP. You can start with the free (Personal) version of Unity. The difference between the paid and free model is mostly in the startup logo of your application. You can’t change that to your own in the free version. Also, some components for your 3D objects are only available in the paid version. But there is plenty enough left to start with.

The first step is enabling the Unity software. Since we do not need the full (plus or pro) edition, select Unity Personal as your license.

In the next screen select the option I don’t use Unity in a professional capacity. When you want to distribute your app in a commercially way you need to choose another one.

After you have filled in credentials (you can get them free) you will receive a thank you screen.

So, let’s start working with Unity. Select the option New to start a new project. Some info regarding the new project is requested. It is important in this step to select 3D. If you don’t do that your holographic objects will be presented as two dimensional and that is not what we want. Press the Create project button to create the project.

At this time Unity is loading the project into the editor. You will notice a Hierarchy view at the left site which shows a new scene called Untitled. Save the scene as for example GazeDemo. That will change the name of the scene. Although we are using a special version of Unity to develop mixed reality applications for HoloLens, some settings need to be taken care of to get it to work.

First we need to change some of the settings of the Main Camera. Select Main Camera from the Hierarchy view.

Select the Inspector tab on the right of the screen. This will show the properties of the selected Main Camera. Make sure that the Position is set to 0,0,0. Secondly we need to set the Clear Flags to solid color and the background color to black (0,0,0). HoloLens uses the color black as transparency to create mixed reality. You could put in another color but then your application becomes more of a virtual reality application since the physical real world is almost not visible anymore to the user.

Now go to the main menu and select EditàProject SettingsàPlayer. At the same location of the Inspector view now the PlayerSettings view appears.

Select the Windows Store tab and go into the Other Settings category. Make sure you have Use 16-bit Depth Buffers turned on. Select Virtual Reality Supported and make sure you have added the Windows Holographic SDK.

Next you need to open the Publishing Settings category under the Windows Store tab. Make sure you have at least SpatialPerception selected. This allows the application to fully use the spatial mapping functionality of the HoloLens device.

A project made with Unity is by default empty. Running the application will not show anything then the startup logo. Let’s add a 3D object to the scene and see how it end up in the HoloLens.

To keep things simple for now we use a cylinder as 3D object. This will represent as a static holographic object in the HoloLens.

• Select the scene in the Hierarchy view
• Go to the main menu and select GameObjectà3D ObjectàCylinder

This will add the cylinder to the scene. The cylinder is by default not really nice and has a single color. To give it more color let’s create a Material and add it to the cylinder as a shader.

• Select the Assets category in the Project View
• Create a new folder by right mouse click àCreateàFolder
• Rename it to Materials
• Select the folder Materials
• Create a new material by right mouse click àCreateàMaterial
• Rename the material to TestMaterial

The next step is choosing some random color (in my case red) as material.

• Select the material TestMaterial
• Go to the Inspector view
• Change the Albedo color to some color you want
• Change the Emission color to some color you want

The result in the Materials folder will look like the following.

Now use the material TestMaterial for the cylinder. This is rather easy to do.

• Make sure that the contents of the folder Materials in the Project view is visible
• Select the cylinder object in the Hierarchy view
• Drag the material into the Inspector view of the cylinder

The result will be something similar to below.

And there you have it! Your cylinder has a shade based on red.

The second configuration we need to do is making sure that the cylinder is positioned before us in the scene. For that we need to go to the Inspector view of the cylinder and change the Position to 0, 0, 6. That means that the unit is on equal height as us positioned 6 units in front of us.

This will result in the following if you look at your scene designer.

Everything is in place and we can start building our application for the HoloLens. Choose from the main menu àFileàBuild Settings. Make sure you have the following settings selected before running the build.

• Select Windows Store as platform
• Select Universal 10 as SDK
• Select UWP Build Type as D3D
• Select the option Unity C# Projects

Make sure that by pressing the button Add Open scenes your scene is added and selected to the build.

Press the button Build to start the build. When this is your first time building your application, you get the option to select a build folder. Create one called App and press the Select Folder button.

When you have done this earlier you will need to reselect the same folder. The reason for this is actually simple. Because Unity generates a Visual Studio solution for you in this folder, you can change the code through Visual Studio. If you do not select the same folder you will lose those changes.

During the build a number of different steps are executed. A few of them are:

• Restoring nugget packages
• Building the scenes, assets
• Patching assemblies
• Compiling the code. Unity uses its own SDK libraries.
• Create a visual studio solution
• Copying player files

The end result is a Visual Studio solution. But be aware. As you can see during execution of the steps the code is compiled by Unity. Unity uses its own SDK libraries. Meaning that while you change code which does rebuild in Visual Studio, you always need to check if that code also compiles in Unity. There can be differences between them.

Select the folder and run the solution. This will start the solution in Visual Studio.

Running an application on a HoloLens device

To run your application on a HoloLens we need to connect Visual Studio to the HoloLens. Make sure that you choose in Visual Studio for x86, debug or release and Remote Machine. Because the HoloLens is a completely self-sustained Windows 10 device it is also a remote machine.

Run the application by deploying it or running it via the debugger.

If this is your first time and you are not paired yet, you will get a dialog for Remote Connections. Make sure that the HoloLens is on. Not wearing a HoloLens which is not connected to the micro USB will go into sleep mode. Wear it or make sure that it is turned on and connected to micro USB.

Select the HoloLens from the Auto Detect connections. If the HoloLens does not appear you will need to connect manually by using the IP address of the HoloLens. This IP address can be found in the HoloLens Device by opening the Control Panel àNetwork & Internet àAdvanced.

If you are not paired yet it will request a PIN code.

Use the HoloLens and open the Control Panel à Update & Security à Developer Mode à
Pair device. You will receive a PIN code. Enter the PIN code in the Visual Studio dialog. Press the OK button.

Result

The application will start to deploy to the HoloLens device. Depending on which method of application development you have used you will see different results.

In case of using only Visual Studio, the application opens in the HoloLens as a full immersive application and shows a rainbow-colored box spinning. The box is static and you are not able to interact with it since there is not yet a Gaze or gestures implemented.

In case of using Unity, the application opens in the HoloLens as a full immersive application showing the logo of Unity. After some seconds, it disappears and the cylinder will appear in front of you. Keep in mind that the cylinder is currently static and you are not able to interact with it since no Gaze or Gestures are implemented.