How to use the Device Portal API of HoloLens – Take a picture containing holograms and camera

The Windows Device Portal is your gateway to remotely control and manage your HoloLens device via a browser. It allows you to deploy apps,has a file explorer, virtual input, monitor and trace performance of your apps and more. All the stuff which is available in the Windows Device Portal is built on top of REST API’s. These API’s can be used to access data on your device as control your device. More information about that REST API van be found here

In this example i will demonstrate how to build an UWP which takes a photo via the HoloLens device and retrieves the image file to display the photo in your app. Th UWP has a very simple setup and i didn’t spend too much time on the design as you can see. It contains a single button “Take picture” and two fields which are filled in when the photo is taken. The first field shows the filename as generated by HoloLens when the photo was taken. The second field shows the picture. In my case a combination of camera and hologram. Don’t mind the episode of Luke Cage on Netflix 😉

Windows Device Portal

The Windows Device Portal is accessed though a web browser. Your HoloLens device needs to be connected via USB to your computer or on the same WiFi network.

• Connected via USB – Use http://127.0.0.1:10080/ to reach the portal
• Connected via WiFi – Use http://[ip address of Hololens]/ to reach the portal

When you enter the address to access the Windows Device Portal you are requested for a username and password.

If you want to connect via WiFi you will need to disable SSL on the HoloLens device when not using a certificate. In case of production make sure SSL is enabled and you have downloaded and implemented the certificate which you can retrieve from the HoloLens Device. The settings for both SSL and the certificate can be found through the Device Portal -> Security Tab at the top.

Connecting to Device Portal API

The HoloLens device needs to be “Online” to access the HoloLens via the Device Portal API. If it is not online it will fail to execute a call after you are connected. Make sure that at the top in the Windows Device Portal is says “ONLINE”.

At the home page of the Windows Device Portal you will find the Sleep settings. By default they are on 3 minutes. You can change this to a maximum of 30 minutes.

In our demo we have created a simple class containing some properties. We use the System.Net.Http.HttpClient class for calling the REST API calls. Secondly we have the HoloLensUrl. In this example the HoloLens device is connected via USB to the computer. We also have two properties which contain the REST API Calls for taking a photo and getting the file. I will explain the parameters later.

        HttpClient client = null;

        static string HoloLensUrl = "http://127.0.0.1:10080/";
        static string ApiTakePhoto = "api/holographic/mrc/photo?holo=true&pv=true";
        static string ApiGetFile = "api/holographic/mrc/file?filename={0}&op=stream";

In this class we have a method called Initialize() which is called from the constructor of the class. The method defines a HttpClientHandler which allows us to pass the credentials. These credentials are the same as the credentials of the Windows Device Portal in the browser. A new HttpClient is created with this handler.

        private void Initialize()
        {
            HttpClientHandler handler = new HttpClientHandler();
            handler.Credentials = new NetworkCredential("ameijers", "********");

            client = new HttpClient(handler);
        }

This same handler can be used to specify the HoloLens device certificate in case of production apps.

Taking a photo

The REST API call for taking a photo is api/holographic/mrc/photo?holo=true&pv=true. The parameter holo defines if the photo should capture the holograms and the parameter pv defines if the photo should capture the PV camera.

The call will return a Json payload. So we need an object to serialize the contents to it. For this call we need a class called Picture containing one property names PhotoFileName

    public class Picture
    {
        public string PhotoFileName { get; set; }
    }

The next function will take the photo via the HoloLens device. It calls the REST API call with a PostAsync. When the call StatusCode returns a OK, the result of the call is returned through the Content property. The result from the ReadAsStringAsync is deserialized by the NewtonJson library class JsonConvert into the Picture object.

        public async Task TakePhoto()
        { 
            HttpResponseMessage message = await client.PostAsync(HoloLensUrl + ApiTakePhoto, null);

            Picture picture = new Picture();

            if (message.StatusCode == HttpStatusCode.OK)
            {
                // content contains json payload with name of picture taken
                string content = await message.Content.ReadAsStringAsync();

                picture = JsonConvert.DeserializeObject(content);
            }

            return picture;
        }

Get the photo file as stream

To retrieve the photo from the HoloLens device we need to call another REST API call with the photo filename. That method does an Base64 encoding of the filename and retrieves it via api/holographic/mrc/file?filename={0}&op=stream. The parameter op specifies if it is returned as stream.

        private string Base64Encode(string plainText)
        {
            var plainTextBytes = System.Text.Encoding.UTF8.GetBytes(plainText);
            return System.Convert.ToBase64String(plainTextBytes);
        }

        public async Task GetFileAsStream(string fileName)
        {
            string url = string.Format(HoloLensUrl + ApiGetFile, Base64Encode(fileName));

            HttpResponseMessage message = await client.GetAsync(url, HttpCompletionOption.ResponseContentRead);

            return await message.Content.ReadAsStreamAsync();
        }

App implementation

Finally we need to call the methods to take the photo and get the photo based on the filename. We define a button click event. It should be asynchronous due to the calls we do are also asynchronous.  At first we call the method TakePhoto() to take the photo via the HoloLens device. As soon as that happens you will notice a white light appearing at the front of the HoloLens device. Than the filename of the photo is used to retrieve a stream to the photo file on the HoloLens device via the method GetFileAsStream(string filename).

        DevicePortalAPIClient client = new DevicePortalAPIClient();

        public MainPage()
        {
            this.InitializeComponent();
        }

        private async void btnTakePicture_Click(object sender, RoutedEventArgs e)
        {
            Picture picture = await client.TakePhoto();

            tbResult.Text = picture.PhotoFileName;

            Stream stream = await client.GetFileAsStream(picture.PhotoFileName);

            BitmapImage bitmapImage = new BitmapImage();
            bitmapImage.DecodePixelWidth = 600;
            await bitmapImage.SetSourceAsync(stream.AsRandomAccessStream());
            imgPicture.Source = bitmapImage;

        }

The stream is set as data source to the Image on the UWP app by using a BitmapImage class. Keep in mind that you need to specify a decoded pixel with or height to have the image appear in your app. You will need to have an IRandomAccessStream instance of the stream. That can be accomplished by using the method AsRandomAccessStream().

Conclusion

The Device Portal API has some interesting methods to control and manage your HoloLens device. But be aware some of the returning Json payloads are not described. That means you will need to figure them out yourselves. The blog post shows you an example which is a simple way of getting images containing holograms and the camera view. Getting this done through C# code is kind of difficult and the result depends on the used classes of the framework.