MRLink: Robot Teleoperation for Complex Manipulation Tasks Through Virtual Reality

Interacting with robots in complex environments—especially when it comes to performing delicate manipulation tasks—has often been limited by the interfaces available. Traditional remote-control systems can be cumbersome and don’t offer the precision or flexibility required for more intricate tasks. This is where our project steps in: an innovative solution that enables remote manipulation of control panels and other objects using robots through virtual reality. By combining the power of advanced robotics with cutting-edge virtual reality technology, we have developed a system that provides a natural and intuitive way to control a mobile robotic platform. This interaction is made possible through a Meta Quest 3 device, offering an immersive and intuitive way to interact with the environment remotely.

The core motivation behind this project is to provide a new, more intuitive control interface for robots—one that enables users to engage in complex manipulation tasks with ease. By leveraging real-time hand tracking or haptic gloves, coupled with continuous visual feedback in VR, users gain the ability to interact with robots in a way that would be incredibly challenging with traditional remote-control systems. This system allows for more precise control, enhanced navigation, and improved perception, all of which are crucial when interacting with robots in dynamic environments.

This project aims to bridge the gap between human dexterity and robotic precision, enabling operators to perform complex tasks in distant or hazardous locations without being physically present. Whether for industrial maintenance, hazardous environment exploration, or research applications, this technology has the potential to revolutionize remote operations.

The success of this project relies on the powerful combination of three cutting-edge robotic systems: the Boston Dynamics Spot robot, the Kinova Gen3 robotic arm, and the Allegro Hand V4. Each of these robots plays a critical role in achieving our goal of enabling remote manipulation via virtual reality, offering unique capabilities that work seamlessly together.

Boston Dynamics Spot Robot

The Boston Dynamics Spot [1] is a versatile, agile mobile robot designed to navigate complex environments. Known for its ability to handle rough terrain, Spot can walk, climb stairs, and traverse obstacles with impressive stability. This robot serves as the base of the system, moving around environments while carrying useful payloads. Spot’s advanced mobility allows it to reach places that would be otherwise difficult or dangerous for humans to access. The robot’s control system ensures it avoids collisions with obstacles in real time, maintaining a high level of safety as it navigates while performing tasks.

Kinova Gen3 Robotic Arm

The Kinova Gen3 [2] robotic arm is a lightweight, highly flexible arm designed to provide advanced manipulation capabilities. With seven degrees of freedom, the Gen3 arm can reach a wide range of positions and orientations, mimicking the dexterity of the human arm. This robot is ideal for performing delicate tasks that require precision and agility.

Allegro Hand V4

The Allegro Hand V4 [3] is a highly advanced robotic hand designed to provide lifelike dexterity. Equipped with 16 actuators, it is capable of performing intricate movements. The hand’s flexibility and precision make it ideal for handling a wide variety of objects, from small tools to more fragile materials. This ability to control the hand in detail brings an unparalleled level of interaction to the system, allowing for sophisticated object manipulation in a remote environment.

Controlling the robot

The system allows users to control the Spot robot using the Meta Quest controller. While the user has direct control over its movement, the robot retains local autonomy, meaning it will still avoid obstacles and navigate safely. This ensures a balance between manual control and intelligent obstacle avoidance, preventing accidental collisions.

In addition to movement control, the user can also switch between different operating modes of the robot, such as:

• Walk Mode – Enables the robot to move around the environment.
• Stand Mode – Allows the robot to pause and look around without walking.
• Sit Mode – The robot rests in place, which will also improve stability when operating the robotic arm.

Furthermore, the user can adjust specific movement parameters using intuitive sliders in the VR interface, such as:

• Trunk Height – Adjusts the robot’s body height relative to the ground for different terrain conditions.
• Step Distance – Modifies the length of each step, allowing for finer control over movement and stability.

Controlling the robotic arm

The robotic arm can be operated in multiple ways, providing flexibility based on the task at hand:

1. Direct Control Mode – The user has full control over the arm’s position and movement, with minimal safety constraints. This mode is ideal for precise manipulations where full dexterity is required.
2. Protected Mode – In this mode, the system actively prevents self-collisions and ensures safe operation. This is particularly useful in scenarios where there is a high risk of damaging the robot or the environment.
3. Planned Mode – This upcoming feature will allow the user to plan movements in advance, visualizing trajectories and outcomes before executing them. By simulating the movement beforehand, operators can avoid mistakes and optimize their actions.

Controlling the robotic hand

We can control the robotic hand using two primary control methods:

1. Predefined poses – Users can select predefined hand positions, such as pointing a finger or completely opening the hand, to simplify interactions with objects.
2. Real-time hand tracking – The system can track the user’s hand movement in real-time, translating it directly to the robotic hand for more complex manipulations. This method allows for intricate gestures and fine motor control, making remote interactions feel more natural.

To bring this project to life, we leverage several advanced technologies and frameworks:

Evergine

Evergine [4] is our powerful in-house component-based industrial engine built on .NET. Designed to be multiplatform, it provides the foundation for this VR interaction system. Evergine enables real-time rendering, physics simulations, and smooth VR experiences, making it a crucial part of this project.

It’s built with modern rendering techniques in mind, supporting graphics through forward and deferred rendering pipelines. Its compatibility with Windows, Linux, Android, iOS, and WebAssembly allows us to deploy applications across a wide array of platforms efficiently. With built-in support for AR and VR through OpenXR, Evergine is especially well-suited for immersive environments.

Evergine also offers robust extensibility through its addon system, allowing developers to easily enhance the engine’s capabilities. We continuously expand its functionality with new features delivered as addons, ensuring Evergine stays at the forefront of industry needs. Recent additions include specialized tools like the DICOM renderer for medical imaging and a Gaussian Splatting renderer for advanced visualization techniques. This flexible addon architecture ensures that users can tailor Evergine to their specific requirements, making it adaptable to a wide range of applications.

Evergine Studio, the engine’s visual editor, further enhances productivity by providing a user-friendly interface for managing scenes, assets, and materials. These features make Evergine an ideal choice for creating high-performance, real-time industrial and interactive applications.

MRTK

MRTK [5] is an Evergine package that accelerates the development of cross-platform XR applications by providing a rich collection of foundational building blocks. This addon includes essential components such as pointers, buttons, interactable elements, and input system integrations that make it easier to design intuitive and responsive XR user interfaces. It abstracts platform-specific details, allowing developers to focus on user experience rather than hardware compatibility.

The package supports a variety of input modalities, including gaze and controllers, enabling natural and immersive interaction across devices. Additionally, it offers visual feedback mechanisms that enrich XR applications. If you’re targeting AR or VR headsets, MRTK for Evergine helps streamline development with reusable components and a consistent interaction model.

XRV

XRV [6] is a specialized library we utilize while developing custom XR experiences. It provides a collection of essential functionalities, including floating windows, hand menus, and voice commands, using a module-based architecture to allow for easy extensibility. With XRV, we enhance user interaction within the virtual environment, making the experience more intuitive and user-friendly.

ROS2

ROS2 [7] is a set of software libraries and tools that serve as the backbone of this robotic application. It offers essential components such as drivers, state-of-the-art algorithms, and robust developer tools, allowing us to efficiently integrate and control all the robots in the solution. Using its node-based architecture, we can deploy different components to separate machines in order to optimize resource usage across all hardware.

While the system already provides impressive capabilities, we are continuously working on new features to improve functionality and usability:

• Tactile feedback – By incorporating touch sensors and a haptic glove, we aim to give users a sense of touch when interacting with remote objects, enhancing realism and precision.
• Object scanning – We plan to implement scanning features that allow the system to generate digital replicas of remote objects and display them within the virtual environment. This will provide better situational awareness and improved interaction.
• Symbiotic control – Using hand tracking, we intend to introduce a mode where the robotic arm moves in a coordinated manner with the user’s hand, creating a more natural and fluid control scheme.

This project represents a leap forward in the field of remote robotics and virtual reality. By combining cutting-edge VR technology with intelligent robotic systems, we are making remote manipulation more intuitive, precise, and accessible. As we continue to enhance this system with new features and capabilities, we look forward to unlocking even greater possibilities for industrial, research, and hazardous environment applications. The future of remote interaction is here, and we are excited to be a part of this revolution.

• 1] https://bostondynamics.com/products/spot/
• [2] https://www.kinovarobotics.com/product/gen3-robots
• [3] https://www.allegrohand.com/v4
• [4] https://evergine.com
• [5] https://github.com/EvergineTeam/MixedRealityToolkit
• [6] https://github.com/EvergineTeam/XRV
• [7] https://docs.ros.org/

This project, file MIG-20221067, has received funding from CDTI – Centro de Desarrollo Tecnológico y la Innovación – in the call for projects “Misiones CDTI” 2022 and from the European’s Union NextGeneration EU Programme – The Recovery and Resilience Mechanism.