The gaze is one of the different forms of input, but also one of the most important ones, used with a HoloLens. The gaze is used for targeting holographic objects and your environment. The gaze helps you as developer to determine where the user is looking at and it allows you to predict what his/her attentions are. The position and orientation of the head is used to determine where the gaze is pointing at. The gaze is presented as a holographic object itself. You can use any type of visual gaze as you wish allowing you to create some nifty gaze cursors.

The purpose of this post is to explain how you implement a gaze which interacts with the physical environment around you. It expects that you have a project setup with Unity containing at least one scene containing a Main Camera and Directional Light. I will be using the default project setup I described in an earlier post.

Creating a gaze cursor object

As already mentioned you can use anything for a gaze. For now, we will keep things simple and use the 3D cube object you can create with Unity.

We will create a prefab for the gaze containing a 3D object, material to style the 3D object and code to handle the gaze. The prefab is actually a package containing everything needed to place a gaze cursor in your scene. Changes made in the prefab will reflect into the objects based on that prefab in your scene. Kind of inheritance with C#

  • Go to the Hierarchy view
  • Select the scene in the hierarchy. In our case the scene is called SceneStart
  • Go to menu GameObjectà3D ObjectàCube
  • Go to the Project view
  • Create a new folder named Gaze under the Assets category
  • Go into the new folder named Gaze
  • Drag the Cube from the Hierarchy view to the Gaze folder
  • Remove the Cube from the Hierarchy view
  • Right-mouse click àCreateàPrefab and name it GazeCursor
  • Right-mouse click àCreateàMaterial and name it GazeMaterial
  • Configure the material by changing the color in Albedo
  • Configure the Cube to disable the Box collider. If the collider of the cube is active, you will notice that the cube is coming towards you due to hitting itself continuously.
  • Drag the Cube from the Gaze folder into the Cube prefab
  • Drag the GazeMaterial from the Gaze folder into the Cube prefab

At this moment, we have a GazeCursor containing no code. But before we are going to add code we need an additional prefab for the spatial mapping of the environment.

Create the spatial mapping

The spatial mapping is needed to define the area of the physical environment. It uses a component called Spatial Mapping Collider to define the colliding areas of the environment. The following steps need to be executed.

  • Go to the Hierarchy view
  • Select the scene in the hierarchy. In our case the scene is called SceneStart
  • Go to menu GameObjectàCreate empty
  • Go to the Project view
  • Create a new folder named Spatial Mappiong under the Assets category
  • Go into the new folder named Spatial Mapping
  • Right-mouse click àCreateàPrefab and name it Spatial Mapping
  • Drag the empty GameObject from the Hierarchy view to into the prefab Spatial Mapping
  • Remove the empty GameObject from the Hierarchy view
  • Select the prefab Spatial Mapping
  • Click on the button Add Component in the Inspector view
  • Choose ARàSpatial Mapping Collider

Spatial mapping script

Now we are going to start with adding C# script to the Spatial Mapping prefab. This code will Initialize and configure the collider of the Spatial Mapping prefab. Due to the collider, we will be able to hit the GazeCursor to the physical environment.

  • Go to the Project view
  • Go into the folder Spatial Mapping
  • Right-mouse Click àCreateàC# Script
  • Rename it to SpatialMapping.cs
using UnityEngine;
using UnityEngine.VR.WSA;

public class SpatialMapping : MonoBehaviour
{
    public static SpatialMapping Instance { private set; get; }

    // this property is used by the GazeCursor as a property with the Raycast call
    [HideInInspector]
    public static int PhysicsRaycastMask;

    // The layer to use for spatial mapping collisions.
    private int physicsLayer = 31;

    // Creates/updates environment colliders to work with physics.
    private SpatialMappingCollider spatialMappingCollider;

    void Awake()
    {
        // create an instance of the class
        Instance = this;
    }

    // Use this for initialization
    void Start()
    {
        // Initialize and configure the collider
        spatialMappingCollider = gameObject.GetComponent<SpatialMappingCollider>();
        spatialMappingCollider.surfaceParent = this.gameObject;
        spatialMappingCollider.freezeUpdates = false;
        spatialMappingCollider.layer = physicsLayer;

        // define the mask
        PhysicsRaycastMask = 1 << physicsLayer;

        // set the object as active one
        gameObject.SetActive(true);
    }
}
  • Drag the SpatialMapping.cs file into the Inspector view of the Spatial Mapping prefab.

The class is by default derived from MonoBehaviour. It contains two methods called Awake() and Start(). These methods are called from Unity when the objects are instantiated. As you can see we initialize and configure the collider. Most of these configurations can also be set in Unity through the Inspector view. But in our case, we need the physical layer to define the PhysicsRaycastMask which is used in the code for the Gaze Cursor.

Gaze cursor script

The gaze cursor script handles the routine to check if the gaze cursor hits the physical environment defined by the HoloLens. We will be adding a C# script to the GazeCursor prefab.

  • Go to the Project view
  • Go into the folder Gaze Cursor
  • Right-mouse Click àCreateàC# Script
  • Rename it to GazeCursor.cs
using UnityEngine;

public class GazeCursor : MonoBehaviour
{
    private MeshRenderer meshRenderer;

    // Use this for initialization
    void Start()
    {
        // Grab the mesh renderer that's on the same object as this script.
        meshRenderer = this.gameObject.GetComponent<MeshRenderer>();
    }

    // Update is called once per frame
    void Update()
    {
        // Do a raycast into the world based on the user's
        // head position and orientation.
        var headPosition = Camera.main.transform.position;
        var gazeDirection = Camera.main.transform.forward;

        RaycastHit hitInfo;
        if (Physics.Raycast(headPosition, gazeDirection, out hitInfo, 30.0f, SpatialMapping.PhysicsRaycastMask))
        {
            // If the raycast hit a hologram...

            // Display the cursor mesh.
            meshRenderer.enabled = true;
            // Move the cursor to the point where the raycast hit.
            this.transform.position = hitInfo.point;
            // Rotate the cursor to hug the surface of the hologram.
            this.transform.rotation =
                Quaternion.FromToRotation(Vector3.up, hitInfo.normal);
        }
        else
        {
            // If the raycast did not hit a hologram, hide the cursor mesh.
            meshRenderer.enabled = false;
        }
    }
}

  • Drag the GazeCursor.cs file into the Inspector view of the Gaze Cursor prefab.

The script is similar as the code from the Microsoft examples of HoloLens. The difference is that we now extended the call of the Physics.Raycast method with an additional parameter. The method gets the PhysicsRaycastMask of the SpatialMapping class we created earlier. This allows us to identify if we hit the collider defined in the SpatialMapping prefab, hence hitting the physical environment.

Extending the scene

We have created a Gaze Cursor and Spatial Mapping for our scene. To use the objects, we need to add them both to your active scene. Drag both prefabs into the Hierarchy view as a child into that scene.

The result

And that’s all the work we had to do. Let’s see how it runs in the HoloLens. Build your application and let Unity create the Visual Studio solution. Open the solution and run the application on your HoloLens device. The result should be like the movie below.

Keep in mind that it will take a little bit of time for the Spatial Mapping Collider before it has mapped the environment. Therefor the Gaze Cursor will not appear until it is ready. In my case it takes approximately 4-5 seconds.

Previous articleCreate your own gaze which interacts with Holographic objects for HoloLens
Next articleMoving holographic objects by hand using the HoloLens
A professional which inspires, motivates and educates businesses on how to leverage emerging technologies using Virtual, Augmented and Mixed Reality in their journey to digital innovation. He has a strong background on SharePoint, Office 365 and Microsoft Azure and is working with machine learning, artificial intelligence and cognitive services. Primarily focused on manufacturing, construction, industry, logistics and maritime/offshore, his goal is helping businesses to achieve more on creating, improving, smartening and shortening of business processes by using services, apps and devices. Furthermore, he engages in speaking, blogging and is organizer of events like the Mixed Reality User Group, Mixed Reality Talk @ VR, SP&C and the Global AI/MR Bootcamp. With more than 20 years of IT Business experience, acting in different roles like solution architect, he believes the future is just starting! Highly experienced in all realities, HoloLens and other devices, User experiences on devices and connecting the world to the cloud will help him to shape that future!

LEAVE A REPLY

Please enter your comment!
Please enter your name here