Voice commands provide simple and flexible ways to interact with the environment. To enable it in our application, we utilize the speech input system in MRTK together with the SpeechInputHandler component. Different voice commands are specified in the MixedRealityToolkit object > Input > Speech settings. Detailed response functions are set in the SpeechInputHandler bounded to objects that handle the activities. For example, as the RosPublisher object handles robot-related commands, one SpeechInputHandler component is added there, and corresponding reacting functions are specified. To ensure that the voice recognition module works properly, a speech confirmation tooltip prefab is enabled. When a voice command is detected, a small box with the corresponding recognized command pops up in the view as shown in Figure 1.

When users operate the robot arm, sometime the view of the users would be blocked by the arm itself. To help the users have better views, a video live stream is added in Hololens, which is placed in an image plane, in front of the user, as shown in Figure 2. The video is captured by the camera which is mounted on the gripper of the robot arm. But since the arm will rotate according to the head's motion, if we just simply fix the orientation of the image plane, the video itself would also rotate, which is hard for the user to watch. To avoid this problem, we subscribe to the orientation angle of the gripper by the ROS topic joint_states, and also apply this orientation change to the image plane, this way, even if the camera is rotated, the video is always adjusted to make sure to keep the right angle.

The prefab for the help panel is from the MRTK Foundation package. On the panel, useful voice commands are listed as shown in Figure 3. The panel will pop up and disappear according to voice commands. Besides, it locates at position (-0.5f, 0.25f, 2.5f) relative to the camera position whenever it is enabled. It does not change position according to user movement. It gives users necessary prompts when they interact with the Spot robot using HoloLens 2.