Elena Canorea
Communications Lead
Interacting with robots in complex environments—especially when it comes to performing delicate manipulation tasks—has often been limited by the interfaces available. Traditional remote-control systems can be cumbersome and don’t offer the precision or flexibility required for more intricate tasks. This is where our project steps in: an innovative solution that enables remote manipulation of control panels and other objects using robots through virtual reality. By combining the power of advanced robotics with cutting-edge virtual reality technology, we have developed a system that provides a natural and intuitive way to control a mobile robotic platform. This interaction is made possible through a Meta Quest 3 device, offering an immersive and intuitive way to interact with the environment remotely.
The core motivation behind this project is to provide a new, more intuitive control interface for robots—one that enables users to engage in complex manipulation tasks with ease. By leveraging real-time hand tracking or haptic gloves, coupled with continuous visual feedback in VR, users gain the ability to interact with robots in a way that would be incredibly challenging with traditional remote-control systems. This system allows for more precise control, enhanced navigation, and improved perception, all of which are crucial when interacting with robots in dynamic environments.
This project aims to bridge the gap between human dexterity and robotic precision, enabling operators to perform complex tasks in distant or hazardous locations without being physically present. Whether for industrial maintenance, hazardous environment exploration, or research applications, this technology has the potential to revolutionize remote operations.
The success of this project relies on the powerful combination of three cutting-edge robotic systems: the Boston Dynamics Spot robot, the Kinova Gen3 robotic arm, and the Allegro Hand V4. Each of these robots plays a critical role in achieving our goal of enabling remote manipulation via virtual reality, offering unique capabilities that work seamlessly together.
The Boston Dynamics Spot [1] is a versatile, agile mobile robot designed to navigate complex environments. Known for its ability to handle rough terrain, Spot can walk, climb stairs, and traverse obstacles with impressive stability. This robot serves as the base of the system, moving around environments while carrying useful payloads. Spot’s advanced mobility allows it to reach places that would be otherwise difficult or dangerous for humans to access. The robot’s control system ensures it avoids collisions with obstacles in real time, maintaining a high level of safety as it navigates while performing tasks.
The Kinova Gen3 [2] robotic arm is a lightweight, highly flexible arm designed to provide advanced manipulation capabilities. With seven degrees of freedom, the Gen3 arm can reach a wide range of positions and orientations, mimicking the dexterity of the human arm. This robot is ideal for performing delicate tasks that require precision and agility.
The Allegro Hand V4 [3] is a highly advanced robotic hand designed to provide lifelike dexterity. Equipped with 16 actuators, it is capable of performing intricate movements. The hand’s flexibility and precision make it ideal for handling a wide variety of objects, from small tools to more fragile materials. This ability to control the hand in detail brings an unparalleled level of interaction to the system, allowing for sophisticated object manipulation in a remote environment.
The system allows users to control the Spot robot using the Meta Quest controller. While the user has direct control over its movement, the robot retains local autonomy, meaning it will still avoid obstacles and navigate safely. This ensures a balance between manual control and intelligent obstacle avoidance, preventing accidental collisions.
In addition to movement control, the user can also switch between different operating modes of the robot, such as:
Furthermore, the user can adjust specific movement parameters using intuitive sliders in the VR interface, such as:
The robotic arm can be operated in multiple ways, providing flexibility based on the task at hand:
We can control the robotic hand using two primary control methods:
To bring this project to life, we leverage several advanced technologies and frameworks:
Evergine [4] is our powerful in-house component-based industrial engine built on .NET. Designed to be multiplatform, it provides the foundation for this VR interaction system. Evergine enables real-time rendering, physics simulations, and smooth VR experiences, making it a crucial part of this project.
It’s built with modern rendering techniques in mind, supporting graphics through forward and deferred rendering pipelines. Its compatibility with Windows, Linux, Android, iOS, and WebAssembly allows us to deploy applications across a wide array of platforms efficiently. With built-in support for AR and VR through OpenXR, Evergine is especially well-suited for immersive environments.
Evergine also offers robust extensibility through its addon system, allowing developers to easily enhance the engine’s capabilities. We continuously expand its functionality with new features delivered as addons, ensuring Evergine stays at the forefront of industry needs. Recent additions include specialized tools like the DICOM renderer for medical imaging and a Gaussian Splatting renderer for advanced visualization techniques. This flexible addon architecture ensures that users can tailor Evergine to their specific requirements, making it adaptable to a wide range of applications.
Evergine Studio, the engine’s visual editor, further enhances productivity by providing a user-friendly interface for managing scenes, assets, and materials. These features make Evergine an ideal choice for creating high-performance, real-time industrial and interactive applications.
MRTK [5] is an Evergine package that accelerates the development of cross-platform XR applications by providing a rich collection of foundational building blocks. This addon includes essential components such as pointers, buttons, interactable elements, and input system integrations that make it easier to design intuitive and responsive XR user interfaces. It abstracts platform-specific details, allowing developers to focus on user experience rather than hardware compatibility.
The package supports a variety of input modalities, including gaze and controllers, enabling natural and immersive interaction across devices. Additionally, it offers visual feedback mechanisms that enrich XR applications. If you’re targeting AR or VR headsets, MRTK for Evergine helps streamline development with reusable components and a consistent interaction model.
XRV [6] is a specialized library we utilize while developing custom XR experiences. It provides a collection of essential functionalities, including floating windows, hand menus, and voice commands, using a module-based architecture to allow for easy extensibility. With XRV, we enhance user interaction within the virtual environment, making the experience more intuitive and user-friendly.
ROS2 [7] is a set of software libraries and tools that serve as the backbone of this robotic application. It offers essential components such as drivers, state-of-the-art algorithms, and robust developer tools, allowing us to efficiently integrate and control all the robots in the solution. Using its node-based architecture, we can deploy different components to separate machines in order to optimize resource usage across all hardware.
While the system already provides impressive capabilities, we are continuously working on new features to improve functionality and usability:
This project represents a leap forward in the field of remote robotics and virtual reality. By combining cutting-edge VR technology with intelligent robotic systems, we are making remote manipulation more intuitive, precise, and accessible. As we continue to enhance this system with new features and capabilities, we look forward to unlocking even greater possibilities for industrial, research, and hazardous environment applications. The future of remote interaction is here, and we are excited to be a part of this revolution.
Elena Canorea
Communications Lead
Cookie | Duration | Description |
---|---|---|
__cfduid | 1 year | The cookie is used by cdn services like CloudFare to identify individual clients behind a shared IP address and apply security settings on a per-client basis. It does not correspond to any user ID in the web application and does not store any personally identifiable information. |
__cfduid | 29 days 23 hours 59 minutes | The cookie is used by cdn services like CloudFare to identify individual clients behind a shared IP address and apply security settings on a per-client basis. It does not correspond to any user ID in the web application and does not store any personally identifiable information. |
__cfduid | 1 year | The cookie is used by cdn services like CloudFare to identify individual clients behind a shared IP address and apply security settings on a per-client basis. It does not correspond to any user ID in the web application and does not store any personally identifiable information. |
__cfduid | 29 days 23 hours 59 minutes | The cookie is used by cdn services like CloudFare to identify individual clients behind a shared IP address and apply security settings on a per-client basis. It does not correspond to any user ID in the web application and does not store any personally identifiable information. |
_ga | 1 year | This cookie is installed by Google Analytics. The cookie is used to calculate visitor, session, campaign data and keep track of site usage for the site's analytics report. The cookies store information anonymously and assign a randomly generated number to identify unique visitors. |
_ga | 1 year | This cookie is installed by Google Analytics. The cookie is used to calculate visitor, session, campaign data and keep track of site usage for the site's analytics report. The cookies store information anonymously and assign a randomly generated number to identify unique visitors. |
_ga | 1 year | This cookie is installed by Google Analytics. The cookie is used to calculate visitor, session, campaign data and keep track of site usage for the site's analytics report. The cookies store information anonymously and assign a randomly generated number to identify unique visitors. |
_ga | 1 year | This cookie is installed by Google Analytics. The cookie is used to calculate visitor, session, campaign data and keep track of site usage for the site's analytics report. The cookies store information anonymously and assign a randomly generated number to identify unique visitors. |
_gat_UA-326213-2 | 1 year | No description |
_gat_UA-326213-2 | 1 year | No description |
_gat_UA-326213-2 | 1 year | No description |
_gat_UA-326213-2 | 1 year | No description |
_gid | 1 year | This cookie is installed by Google Analytics. The cookie is used to store information of how visitors use a website and helps in creating an analytics report of how the wbsite is doing. The data collected including the number visitors, the source where they have come from, and the pages viisted in an anonymous form. |
_gid | 1 year | This cookie is installed by Google Analytics. The cookie is used to store information of how visitors use a website and helps in creating an analytics report of how the wbsite is doing. The data collected including the number visitors, the source where they have come from, and the pages viisted in an anonymous form. |
_gid | 1 year | This cookie is installed by Google Analytics. The cookie is used to store information of how visitors use a website and helps in creating an analytics report of how the wbsite is doing. The data collected including the number visitors, the source where they have come from, and the pages viisted in an anonymous form. |
_gid | 1 year | This cookie is installed by Google Analytics. The cookie is used to store information of how visitors use a website and helps in creating an analytics report of how the wbsite is doing. The data collected including the number visitors, the source where they have come from, and the pages viisted in an anonymous form. |
attributionCookie | session | No description |
cookielawinfo-checkbox-analytics | 1 year | Set by the GDPR Cookie Consent plugin, this cookie is used to record the user consent for the cookies in the "Analytics" category . |
cookielawinfo-checkbox-necessary | 1 year | This cookie is set by GDPR Cookie Consent plugin. The cookies is used to store the user consent for the cookies in the category "Necessary". |
cookielawinfo-checkbox-necessary | 11 months | This cookie is set by GDPR Cookie Consent plugin. The cookies is used to store the user consent for the cookies in the category "Necessary". |
cookielawinfo-checkbox-necessary | 11 months | This cookie is set by GDPR Cookie Consent plugin. The cookies is used to store the user consent for the cookies in the category "Necessary". |
cookielawinfo-checkbox-necessary | 1 year | This cookie is set by GDPR Cookie Consent plugin. The cookies is used to store the user consent for the cookies in the category "Necessary". |
cookielawinfo-checkbox-non-necessary | 11 months | This cookie is set by GDPR Cookie Consent plugin. The cookies is used to store the user consent for the cookies in the category "Non Necessary". |
cookielawinfo-checkbox-non-necessary | 11 months | This cookie is set by GDPR Cookie Consent plugin. The cookies is used to store the user consent for the cookies in the category "Non Necessary". |
cookielawinfo-checkbox-non-necessary | 11 months | This cookie is set by GDPR Cookie Consent plugin. The cookies is used to store the user consent for the cookies in the category "Non Necessary". |
cookielawinfo-checkbox-non-necessary | 1 year | This cookie is set by GDPR Cookie Consent plugin. The cookies is used to store the user consent for the cookies in the category "Non Necessary". |
cookielawinfo-checkbox-performance | 1 year | Set by the GDPR Cookie Consent plugin, this cookie is used to store the user consent for cookies in the category "Performance". |
cppro-ft | 1 year | No description |
cppro-ft | 7 years 1 months 12 days 23 hours 59 minutes | No description |
cppro-ft | 7 years 1 months 12 days 23 hours 59 minutes | No description |
cppro-ft | 1 year | No description |
cppro-ft-style | 1 year | No description |
cppro-ft-style | 1 year | No description |
cppro-ft-style | session | No description |
cppro-ft-style | session | No description |
cppro-ft-style-temp | 23 hours 59 minutes | No description |
cppro-ft-style-temp | 23 hours 59 minutes | No description |
cppro-ft-style-temp | 23 hours 59 minutes | No description |
cppro-ft-style-temp | 1 year | No description |
i18n | 10 years | No description available. |
IE-jwt | 62 years 6 months 9 days 9 hours | No description |
IE-LANG_CODE | 62 years 6 months 9 days 9 hours | No description |
IE-set_country | 62 years 6 months 9 days 9 hours | No description |
JSESSIONID | session | The JSESSIONID cookie is used by New Relic to store a session identifier so that New Relic can monitor session counts for an application. |
viewed_cookie_policy | 11 months | The cookie is set by the GDPR Cookie Consent plugin and is used to store whether or not user has consented to the use of cookies. It does not store any personal data. |
viewed_cookie_policy | 1 year | The cookie is set by the GDPR Cookie Consent plugin and is used to store whether or not user has consented to the use of cookies. It does not store any personal data. |
viewed_cookie_policy | 1 year | The cookie is set by the GDPR Cookie Consent plugin and is used to store whether or not user has consented to the use of cookies. It does not store any personal data. |
viewed_cookie_policy | 11 months | The cookie is set by the GDPR Cookie Consent plugin and is used to store whether or not user has consented to the use of cookies. It does not store any personal data. |
VISITOR_INFO1_LIVE | 5 months 27 days | A cookie set by YouTube to measure bandwidth that determines whether the user gets the new or old player interface. |
wmc | 9 years 11 months 30 days 11 hours 59 minutes | No description |
Cookie | Duration | Description |
---|---|---|
__cf_bm | 30 minutes | This cookie, set by Cloudflare, is used to support Cloudflare Bot Management. |
sp_landing | 1 day | The sp_landing is set by Spotify to implement audio content from Spotify on the website and also registers information on user interaction related to the audio content. |
sp_t | 1 year | The sp_t cookie is set by Spotify to implement audio content from Spotify on the website and also registers information on user interaction related to the audio content. |
Cookie | Duration | Description |
---|---|---|
_hjAbsoluteSessionInProgress | 1 year | No description |
_hjAbsoluteSessionInProgress | 1 year | No description |
_hjAbsoluteSessionInProgress | 1 year | No description |
_hjAbsoluteSessionInProgress | 1 year | No description |
_hjFirstSeen | 29 minutes | No description |
_hjFirstSeen | 29 minutes | No description |
_hjFirstSeen | 29 minutes | No description |
_hjFirstSeen | 1 year | No description |
_hjid | 11 months 29 days 23 hours 59 minutes | This cookie is set by Hotjar. This cookie is set when the customer first lands on a page with the Hotjar script. It is used to persist the random user ID, unique to that site on the browser. This ensures that behavior in subsequent visits to the same site will be attributed to the same user ID. |
_hjid | 11 months 29 days 23 hours 59 minutes | This cookie is set by Hotjar. This cookie is set when the customer first lands on a page with the Hotjar script. It is used to persist the random user ID, unique to that site on the browser. This ensures that behavior in subsequent visits to the same site will be attributed to the same user ID. |
_hjid | 1 year | This cookie is set by Hotjar. This cookie is set when the customer first lands on a page with the Hotjar script. It is used to persist the random user ID, unique to that site on the browser. This ensures that behavior in subsequent visits to the same site will be attributed to the same user ID. |
_hjid | 1 year | This cookie is set by Hotjar. This cookie is set when the customer first lands on a page with the Hotjar script. It is used to persist the random user ID, unique to that site on the browser. This ensures that behavior in subsequent visits to the same site will be attributed to the same user ID. |
_hjIncludedInPageviewSample | 1 year | No description |
_hjIncludedInPageviewSample | 1 year | No description |
_hjIncludedInPageviewSample | 1 year | No description |
_hjIncludedInPageviewSample | 1 year | No description |
_hjSession_1776154 | session | No description |
_hjSessionUser_1776154 | session | No description |
_hjTLDTest | 1 year | No description |
_hjTLDTest | 1 year | No description |
_hjTLDTest | session | No description |
_hjTLDTest | session | No description |
_lfa_test_cookie_stored | past | No description |
Cookie | Duration | Description |
---|---|---|
loglevel | never | No description available. |
prism_90878714 | 1 month | No description |
redirectFacebook | 2 minutes | No description |
YSC | session | YSC cookie is set by Youtube and is used to track the views of embedded videos on Youtube pages. |
yt-remote-connected-devices | never | YouTube sets this cookie to store the video preferences of the user using embedded YouTube video. |
yt-remote-device-id | never | YouTube sets this cookie to store the video preferences of the user using embedded YouTube video. |
yt.innertube::nextId | never | This cookie, set by YouTube, registers a unique ID to store data on what videos from YouTube the user has seen. |
yt.innertube::requests | never | This cookie, set by YouTube, registers a unique ID to store data on what videos from YouTube the user has seen. |