Research Agenda

Motivation and context

The European Commission is currently defining the scope of the next research and innovation framework programme Horizon Europe for the budget period ranging from 2021 to 2027 with a total budget of 100 billion Euros. The framework programme will be organised in three pillars, namely Excellent Science, Global Challenges, and European Industrial Competitiveness, as well as Innovative Europe (see Figure 1).

Figure 1: Preliminary Structure of Horizon Europe

In Pillar 2, six thematic clusters are defined:

• Health;
• Culture, Creativity and Inclusive Society;
• Civil Security for Society;
• Digital, Industry and Space;
• Climate, Energy and Mobility;
• Food, Bioeconomy, Natural Resources, Agriculture and Environment.

The aim of this document is to identify the gaps, needs, challenges and vision in the domain of XR technologies in order to shape the next framework programme with a significant amount of budget for this domain.

As from the currently available structure of Horizon Europe, three areas can be identified, which relate to future research and innovation for XR technologies. There is a need for new research infrastructures as set in Pillar 1. Secondly, XR technologies are considered as an axis spanning across all six clusters described in Pillar 2. Finally, it also has the potential to serve as a base for the innovation ecosystem as presented in Pillar 3.

Current state and challenges of XR technologies

The impact of Covid-19 on XR in Europe

The Covid-19 pandemic has unleashed adverse impact on the global business. The lockdown in almost all countries severely affected the ways companies do business and operate. Even though the crisis is persisting, it will likely generate long-term trends resulting from increased need to digitalise businesses and processes to face the Covid-19 restrictions (travel restrictions, remote work, distancing, etc.). To understand the impact of the Covid-19 pandemic on XR businesses, XR4ALL conducted a survey in May 2020, which received 134 responses from different types of stakeholders.

Asked about how they would scale the general impact of the Coronavirus outbreak on their business or businesses in their network/portfolio, respondents gave diverse answers ranging from significant negative impact to significant positive impact. Figure 2 shows that only 24.6% had a significant negative impact and for the remaining respondents, 44.8% indicated that they had partially negative impact and 19.4% had rather partial positive impact and 8.2% significant positive impact. Thus, in the XR industry, the impact on XR businesses is not too severe, as more than 75% of the respondents stated, the impact was not significantly negative and for 27.6% of the respondents, the impact is rather positive. The coronavirus pandemic is perceived to provide a boost for the XR industry.

Figure 2: Impact of the Covid-19 outbreak on business

For those companies, who perceived an impact of Covid-19 on their business or on companies in their network responded as follows (see Figure 3). The majority mentioned a decrease in revenues, profit and cash flow (47.0%). 38.1% had a more difficult access to funding and capital. However, a third of the respondents mentioned an increased interest in provided products and solutions.

Figure 3: Impact of Covid-19 on the business or the companies in the network/portfolio

The participants in the survey were also asked, how they or companies in their network take measures to counter the crisis. As presented in Figure 4, the most prominent measure was cutting costs (41.0%), followed by renegotiating salaries (33.6%), lay-off employees (33.6%), freezing hire (33.6%) and reduction of investment (32.8%). R&D activities were reduced by 26.1% and contracts with suppliers were negotiated or stopped by 20.1% of the respondents.

Figure 4: Measures taken to counter the crisis

Asked about if they see any new opportunities for XR technologies as a result of the Covid-19 crisis, as indicated in Figure 5, almost all respondents see new opportunities. The majority (38.8%) sees a lot of opportunities, and a big portion (27.6%) sees opportunities and knows how to take advantage of them while 34.3% of the respondents sees opportunities but is unsure how to take advantage of them.

Figure 5: You see any new opportunities for XR technologies as a result of the Covid-19 crisis?

Looking further at these opportunities, as shown in Figure 6, respondents expect flourishing interest in XR technologies (69.4%), an increasing customers demand for XR solutions (56%), an increased user-base (46.3%) and more public funding for R&D support (44.8%).

Figure 6: Which opportunities they expect will have an impact on the XR market

The Covid-19 pandemic is impacting businesses around the world and bringing a number of challenges :

• Social distancing measures restrict our ability to have face-to-face interactions or restrict group activities to limited groups size;
• Travel restrictions and key staff working from home restrict the ability to conduct business, manage effective team operations, and provide expertise locally;
• Fewer on-site staff due to illness, self-isolation, costs and other restrictions limits the ability to continue operations “as before”;
• Lack of classroom and hands-on training and learning makes it difficult to teach students, up-skill new staff or train existing staff on products and processes;
• Disrupted supply chains require more flexible manufacturing and sourcing processes to help ensure continuity of production;
• The reluctance among workers to touch surfaces and objects that may have been touched by others.

A further question asked about which XR industries will best benefit from available opportunities (Figure 7), Education comes in top position (81.3%), followed by Healthcare (61.2%), Enterprise (52.2%), Events (49.3%) and Games/Entertainment (45.5%). This is consistent with the feedback received in the survey 2019 and different polls.

Figure 7: Which XR industries will best benefit from available opportunities?

Clearly, this opens opportunities for new or enhanced XR solutions and tools that can play an effective role in solving a number of these challenges and offering new opportunities in the industries highlighted in Figure 7.

Finally, we asked for the XR technologies that can help develop solutions that can best benefit from opportunities opening for the crisis. As shown in the result diagram in Figure 8, the most relevant technologies can be clustered as follows:

• VR headsets (55.2%) and AR headsets (41.8%) are considered as quite important;
• Another highly relevant cluster of technologies is related to 3D capture and modelling such as 3D capture and modelling of static objects and scenes (6-DoF) (40.3%), 3D capture of volumetric video (6DoF) (38.8%), and 360-degree video (3-DoF) (38.8%);
• Thirdly, Human body tracking (gestures, pose, face, emotions …) is also considered important by 44.0% of the respondents together with Sensing (speech, tactile, bio …) (29.1%);
• Artificial intelligence and XR authoring tools received both 38.8%.

Figure 8: Which XR technologies can help develop solutions to benefit from opportunities through Covid-19? (multiple choices)

The conclusion of this survey can be summarized as follows:

• The impact of Covid-19 on XR businesses is not too severe, as more than 75% of the respondents stated, the impact was not significantly negative and for 27.6%, the impact is rather positive;
• For those, which were impacted, the decrease in revenues and difficulties in receiving funding and capital were mentioned with priority.
• A majority of the respondents see lot or at least a few opportunities as a result of the crisis, which is mainly caused by flourishing interest in XR technologies and increasing customer sales and demand for XR solutions.
• In terms of XR industries that benefit from opportunities, education, healthcare and enterprises are considered as the most relevant domains.
• An important result belongs to the most relevant technologies that benefit from opportunities. Headset technology, 3D capture and modelling, human body tracking and sensing as well as AI and authoring tools were ranked top. This is in line with the conclusions drawn in the next sections.

Major progress

General trends

XR technologies will enter all fields of our daily life, either private or in the corporate setting. The most relevant domains are: repair, maintenance, immersive training, inspection and quality assurance, design, and assembly. However, the entertainment and film industry are entering this domain as well and VR becomes a standard in entertainment and education. For the above mentioned domains, new business models are possible. For remote assistance scenarios, XR technologies lead to increased efficiency, time saving with regards to travel and shorter time for repair and maintenance. Multi-modal and multi-user interaction scenarios are available, which broadens the applicability in a wide range of domains. These changes as well as real-time and low-latency streaming will further create interest for the digital transformation, especially for employees.

The XR domain has received a big push from major players, a.k.a. GAFAM (Google, Amazon, Facebook, Apple, and Microsoft) in terms of marketing. Furthermore, China started taking the technological lead, especially with its investments in Nanchang, where the government setup a centre for XR technologies. In 2018, an annual event was launched called World Conference on VR Industry with an online follow in 2019 and in 2020 . This event aims to be the world-leading event in the industrial VR domain. In 2018, more than 2000 participants took part, where about 300 leading experts from the US, Europe and Japan were invited as speakers.

Pokemon GO can be considered as the first AR killer app, which attracted millions of users. It served also as a showcase for the maturity of basic AR technology and mass deployment. Many other useful AR applications for mobile devices have reached the market. Due to the improvement in quantity and quality of content, a significant push to consumers can be recognised. New VR setups are emerging and they are appropriate for common users and daily use, and not anymore limited to entertainment centres.

XR technology

XR hardware is more affordable and can also be used more easily by technology users and consumers. The possibility of mass production of these devices opens up new opportunities for services. Especially for XR headsets, the devices have greatly evolved and are becoming more and more efficient with a better form factor. Magic Leap, Oculus Quest, HoloLens 2, Varjo XR1 and NReal provide a wider field of view and much better rendering quality. Most of the headsets are tether-free and allow for easier installation and use. Hybrid optical see-through (OST)[1] as well as video see-through (VST)[2] devices are now on the market.

Overall, the ratio between quality and price has improved and XR headsets become an accessible technology for consumers.

With regard to capturing, 360-degree cameras are no longer expensive and provide relatively good quality. The process to capture and generate 360-degree video has been automated, and several tools are available to generate related content. 360-degree video has now become a standard.

More complex approaches for the generation of 3D video content for XR applications, such as volumetric capture, have left the state of pure research topic and turned into commercial volumetric video capture stages and production workflows. Leading companies and institutions such as Microsoft, 4D-Views, Fraunhofer HHI and Volucap already provide professional studios for the creation of volumetric assets.

XR authoring tools are another important topic, as they are more and more integrated. The same can be said for 3D rendering engines such as Unity and Unreal as they become more and more compliant with XR technology and tools used by the movie industry.

Significant advances have been achieved in the field of spatial computing, inside-out-tracking, and especially hand and gesture tracking. In particular, due to the progress of AI and machine learning, tasks like marker-less tracking, scene understanding, object recognition, reconstruction, and classification are becoming more and more robust. SDKs such as ARKit by Apple for iOS and ARCore by Google for Android are available and provide easy to use APIs for mobile devices and offer calibration free optical-see-through HMDs with remarkable registration and positioning capabilities.

Some first prototypes on 5G technology and edge computing are available. Expanding bandwidth and faster internet connection allow to run XR applications in the cloud. The AR cloud will serve as a future base for localisation and interactive mapping of virtual content. This is a base technology for many use case and customer facing applications. The Open Spatial Computing Platform from Open AR Cloud are already under tests in three European cities. Technologies such as Web XR are reducing complexity and make solutions easily accessible. Given the right infrastructure nearly every service can be delivered to the client.

Major challenges

General challenges

One important challenge is to encourage the user to explore deeper and to experience high-quality immersive content. Hence, there is a need for significant investment in high quality, consumer-ready content in order to take the industry from a technology driven development mode to a user and application driven development, where users’ needs and expectations are the main driver.

In the entertainment sector, there is a need to define guidelines and tools for improving storytelling in XR. A new film language grammar for 360 video has to be developed in order to create appealing and attractive content. Synthetic content generation approaches need to be aligned with user-centred design principles.

In the education and learning sector, there is more insight and feedback needed on the impact of the use of XR technology. This holds for memorisation but also for the return on investment with respect to money and time.

Perception in XR is a real barrier today. The phenomenology of the problem has not been fully understood so far. Some troubles and issues are identified to represent the problems in a phenomenological way related to motion-sickness, distance perception, and other biases that are not even discovered yet. The user gets tired quickly, but the reasons for this are not fully understood, because there is no model of XR perception available yet. After deeper understanding of perception in XR, technological solutions need to be developed to solve these perception issues. This is clearly one of the major issues of XR. A real technological breakthrough is necessary for the perception comfort.

Interoperability is one of the prerequisites for wide-scale adoption of the technology and for a healthy ecosystem with a diverse range of technology providers.

The following three obstacles can be identified:

1. Technological obstacles such as different 3D representations, incompatible definitions of markers and natural features, great divergence in computational power between platforms and lack of a common, comprehensive interaction model;
2. Human factors obstacles e.g. multiple terms and contexts for the same object, general lack of knowledge at all management levels and usability, which is limited by heavy and obtrusive I/O devices;
3. Obstacles from physical world properties (mainly related to Augmented Reality) such as diverse value ranges, refresh rates and precision levels; definition of clear limits for analogue data, ignoring small changes of data and lack of standards defining the position and orientation of objects.

Industry-wide, there is awareness of the hardware challenges that are limiting market adoption. This also includes complex home installation requirements limiting the B2C and C2C mass markets.

In addition, the industry relies too heavily on a few suppliers to develop complete XR solutions and the provisioning of industrial grade XR applications for the three major platforms (Apple app store, Google play store and Windows market place) requires a lot of effort. Therefore, a multi-user, cross-device and cross-platform development is probably the biggest challenge at the moment. While the supported features in the different XR toolkits gets more similar, and thus somewhat facilitates the use of cross-platform frameworks, it is often necessary to optimise for a particular platform due to performance constraints. In addition, today’s XR toolkits still lack support for Diminished Reality (DR), so that supporting DR requires interfering with the native rendering pipeline of the target platform.

The lack of open interfaces is currently the biggest obstacle for interoperability, which hinders the use of a mix of technologies from different providers to combine them and build technical enablers. This also holds for standards and specifications of data formats. A cross-sectorial exchange of data and common data bases and data formats are required. Even more, common semantic concepts across sectors would support fast and easy development of XR applications, e.g. between automation and building sector or the logistic and building sector.

Another challenge is the lack of a digital 3D representation of the real world and associated content and services, sometimes referred to as the “AR Cloud”, which users can easily access and store digital objects in. At the moment, all vendors provide their own digital mapping and relocalisation methods, so there is no interoperability.

Interoperability is strongly related to the availability of authoring tools, which reduce the barriers to fast and successful experience development. There are well-established tools available such as Unity, Unreal and several platforms are supported by these. However, there is a lack of tools for easy development of customised solutions. Hence, such tools, which do not require software engineering or dedicated programming skills need to be made widely available.

Another challenge is to build an open ecosystem such as an XR market place. This would allow access to technologies of different stakeholders (corporates, SMEs, start-ups) in Europe. This market place could be dedicated not only to technologies, but also to XR applications as well as talents and jobs. This approach may be considered as a European initiative competing with SteamVR and the platform related app stores as VivePort and Oculus, but with a much broader offering targeting the industry, SMEs and academia.

In addition, a common platform offering trust and data security is required as data sovereignty becomes more and more important. Current discussions around GAIA-X[3] show that responsibility for privacy of personal data by individual companies is a need. In this respect, joint efforts with other organisations developing digital services for e.g. IoT and BIM are required.

The roll-out of 5G will lead to a big push for the XR industry. The processing for XR experiences can then be moved to the cloud enabling the use of low-capability devices with less power consumption. This will then have immediate impact on the form factor and reduce costs. Depending on the application, the computation can be shifted from the local device to a remote or edge server. On the other hand, this will allow more complex processing such as the use of AI and Deep Learning based algorithms. However, 5G will first become commercially mainstreamed in big Asian cities, therefore Europe and US are in danger of falling behind. Furthermore, an outsourcing of complex computer vision algorithms to the cloud is not straightforward and will take a lot of further developments to allow real-time and low-latency processing support for interactive streaming services through the cloud.

XR technology

The acceptability of XR devices for the mass market will be a major challenge. Concerning head-mounted displays, the following issues persist:

• Field of view still limited;
• Lack of user friendly design and operation;
• Lack of more aesthetic devices;
• Need for light weight devices;
• Limited resolution;
• Need for fully wireless 6DOF tracking capability;
• Power consumption is too high.

In the industrial sector, AR headsets need to become more robust to enable their use in more severe environmental conditions. This holds especially for outdoor applications such as the building sector, where 3D geometry is registered and augmented. Furthermore, AR headsets need to become suitable for being used during longer periods of time in the office and at work in general.

Tracking of body parts is still a challenge. Currently, a convincing self-representation of the user in VR is not yet available. Animated hand models are used to represent the user’s hands. A full body representation ideally with its real shape and texture is still a matter of research and not yet available in existing applications.

A major challenge is the development of convincing haptic interfaces for the mass market. There is a need that haptics systems provide users with a real feeling of immersion well-adapted to XR content in many application domains.

In the long term, AI will be the leading-part in devices and applications. The trend is focused on combining machine learning with 3D scene analysis and re-localisation, where Europe has a strong role in research. The main issue is to keep very good students and researchers in Europe. Another major challenge is the availability and the access to high volumes of high-quality data. As Europe cannot compete with Google or Facebook on data collection and production, new approaches for machine learning with less data are required, for which excellent knowledge and promising approaches exist in Europe.

Short and long-term vision in XR

This section summarises feedback from the community collected through a wide range of activities: an online survey, over 20 expert interviews, round table discussions at the XR4ALL luncheon at VR Days 2019 in Amsterdam and workshops run as part of the ‘Engineering XR for the future’ online event co-organised by ETSI, The AREA and XR4ALL.

Short-term vision

General

The community is in general very enthusiastic about the future of XR and believes in an acceleration of adoption of XR technologies over the next 3 to 5 years for a wide range of applications and services. The general view is that B2B market will initially be the most booming, but with the reduction of cost of hardware, consumers market should quickly follow.

XR technology

The deployment of 5G across the globe will greatly improve connectivity by providing wider bandwidth, higher bitrate and lower latency. XR applications requiring ultra-reliable and ultra-fast connectivity will benefit from the roll-out of 5G. The roll-out of 5G will also support the development of lightweight XR wearable devices with limited resources as some of the processing will be moved to the edge or to the cloud.

XR Clouds are expected to be deployed at different scale within 3 to 5 years. XR Cloud can be defined as a persistent 3D digital twin of the real world, continuously updated in real-time, offering users a shared XR experience. XR Cloud will support services providing the 3D mapping of the real environment surrounding the user and services providing accurate localization of the device using this 3D mapping.

The form factor of headsets will improve and their price will decrease by at least 50%. More computation will be moved to the cloud and devices will require less computational resources and therefore smaller batteries. This will have a significant impact on the accessibility and adoption of this new technology by professionals and consumers alike.

Better rendering devices can also be expected with light-field display technology and possibly even holographic displays providing correct eye accommodation and vergence.

Current sensors on mobile devices will be much more exploited and used for new XR applications. Significant advances will be achieved in several areas such as 3D reconstruction, encoding, streaming and novel rendering techniques. AI and Deep Learning algorithms might run in devices with limited capacity in order to support novel and more robust processing enabling better mixing of digital elements with the physical world. Hand tracking and gesture recognition will be generalized and will be more accurate and robust. Advances in AI-based scene understanding tools will facilitate the interaction of AR applications with their environment, as well as the creation of augmented scenes. However, due to computational requirements advanced AI tools might still be used as services rather than on device in the near future.

Attempts in exploring additional senses like smell and touch will be emphasized. However, this will be mainly on the research level and for prototype demonstrators. For smell rendering the technology readiness level will not achieve more than TRL 4-5, while for triggering the touch sense with haptic devices, a TRL of 7-8 will be reached.

Inside-out localisation solutions will become more and more accurate and robust for dynamic and large environments firstly thanks to the improvement of vision processing enhanced by machine learning approaches and secondly thanks to a better fusion of multi-modal sensors. There will be better methods for 3D scene understanding mostly applied to AR, resulting in a better integration of the virtual elements into the real environment.

The digitalisation of users through avatars can provide more efficient, interactive, realistic, immersive experiences with a better embodiment feeling. This is a topic of intense research, which will lead to tremendous improvement over the coming 3-5 years window. The creation of very realistic dynamic 3D representations of humans will likely require the use of volumetric video or photogrammetry techniques.

Applications

In the short term, considering the next 3-5 years, XR technologies will initially be implemented for daily use in enterprises. There will be an increase in the implementation of use cases with relevant focus and measurable Return-on-Investment (RoI). With advances in tracking and registration techniques, XR use cases will include outdoor usage e.g. on construction sites. XR design becomes part of Human Computer Interaction (HCI) in theory and practice. The challenge will be to develop content for XR application based on automatic or semi-automatic solutions. Therefore, the creation of authoring tools as addressed in ICT-Call-55 will further support the development of solutions in this respect.

The development of applications will further increase for the B2B market. The reason is still the significant costs for hardware. This is the market currently addressed by big companies, because there are significant needs and opportunities in the industrial sector such as manufacturing, maintenance, quality inspection, design and repair.

As presented in D4.1 Landscape Report, US and Asian device manufacturers have the lead and will also be leading the operation of XR B2C platforms. However, Europe is considered to be in second position from 2023 onwards with regards to AR/VR revenue.

The market reach of 5G will enable cloud-based streaming of XR solutions. Especially in upcoming social XR applications, 3D video communication will be possible. This means that classical video conferencing might be replaced with XR conferencing in the not too distant future.

Social VR seems to be poised to see a significant development. A better term for “social VR” might be “collaborative telepresence”. Some applications are communication within families and between friends, and collaborative work between people working on a common project in industry. Social VR implies the use of avatars, which can be either life-like or magic/fantasy-like.

XR technologies will also enter more and more the educational area including teaching of soft skills. Consumer applications will also become more main stream in the tourism, entertainment and real estate domains with virtual sightseeing.

Long-term vision

The trends identified in the 3-5 years window will continue with an increase in use of XR technologies across all domains both in professional and personal environments. In the long-term, considering the next 5-10 years, new types of displays and glasses will emerge possibly based on light field or holographic technologies. It is expected that these devices will be more comfortable to wear with a larger field of view, better eye accommodation and lightweight. Compute intense processing will be remote in the edge or in the cloud due to the availability of ultra-high speed and highly reliable connectivity. Furthermore, there will be more aesthetic and cheaper devices increasing adoption for consumer applications. It is expected that eventually everybody will have its own dedicated device, compared to a shared one in a short term, very similar to the way telephony evolved. XR devices could replace both smartphones and desktop computers as XR will offer very efficient interfaces between human minds and information. Therefore, Europe needs to invest in XR hardware capabilities to make sure that it is not left out as current leaders are America and Asia based.

In view of the investments made by industrial players, augmented reality (AR) and mixed reality (MR) should expand rapidly. The design of XR applications will be the new way of user experience (UX) design.

More and more verticals will use XR technology, in addition to the current fields of medicine, industry 4.0 and tourism applications. Overall, XR technologies will become a new standard similar to video nowadays.

Another vision is real-time communication in 3D (social VR). This will enable many services and applications such as remote assistance in companies, communication in healthcare e.g. communication between patients and hospital, distant learning in education and remote social interaction. Furthermore, this will support distributed and more flexible working environments applicable to many domains e.g. engineering or the healthcare sector. Such new approaches towards telepresence will have a great impact on carbon reduction by replacing travelling with distant XR communication. It can also improve work productivity while staying at home and meeting many people.

The methods for 3D scene understanding will be even more efficient and will make it possible to analyse not only the user's environment, but also his/her activity. There is a lot of progress in computer vision and machine learning reported at conferences, but much further progress is needed. Machine learning is really a fundamental research domain also for immersive and interactive technologies. Europe needs to pursue research in this area in order to remain competitive as it cannot compete with GAFAM on data collection and production. Thus, methods for learning with scarce data, data generation methods and semi-supervised/unsupervised learning methods need to be researched, in order to enable European companies to deploy AI-based solution without being dependent on datasets collected in third countries.

Most of the computation required by XR systems (e.g. spatial computing or rendering) will be distributed on cloud infrastructures thanks to the large-scale deployment of mobile networks offering very high bandwidth and very low latency. Digital twins of the real world (XRCloud) will be available on a global scale. These could be controlled by a few players, but it is expected that an open and sovereign alternative of an XRCloud will emerge in Europe. With respect to the tech sector in Europe with its innovative small and medium sized players, it will be able to keep up if it works towards building an open ecosystem, which is not operated by GAFAM and complies with the laws and values of the EU. Hence, new regulations will need to be established to ensure the privacy of personal and professional data. The associated tools will be proposed by trustworthy players to comply with these regulations. Green solutions will have to be implemented to reduce the energy balance of cloud computation dedicated to XR applications.

Ethical dimension of XR

Women in XR

Context

The XR market is growing quickly, with the global immersive technology market expected to reach $570 billion by 2025 according to Allied Market Research . As for gender balance in this industry, according to a statement by one of the invites of the EC luncheon at VR Days, one of the biggest market opportunities in XR lies in fact with female entrepreneurs, as they could influence the way products are built, how companies are run, and how experiences are delivered to an audience.

At the luncheon, it was also noted that the employment market for immersive technologies might be set to become heavily unbalanced in gender terms. This was confirmed to be the consequence of the fact that the immersive tech workforce is seeded from both the technology sector and the creative sector. Both of these sectors were faced by a huge gender diversity problem according to European Women’s Audio-visual Network (EWA), just 1 out of 5 film directors are women. According to a Techcrunch headline from 2017, only 17% of tech founders are female .

The lack of diverse teams in immersive technologies is already causing problems. According to an article by Fast Company and re-confirmed by The Economist , women are more likely to experience motion sickness from XR devices, as well as fuzzy lenses and ill-fitting headsets. This is due to the exclusion of women from the design process, and it creates a vicious cycle where there are further barriers for women to access this innovative technology. A study from January 2020 by Design Interactive Inc. & Lockheed Martin Corporate found that interpupillary distance (IPD) non-fit was found to be the primary driver of gender differences in cyber sickness, with motion sickness susceptibility identified as a secondary driver From a broader diversity point-of-view, a recent workshop by MIT confirmed that the popular stand-alone HMD Oculus Go was clearly not designed for black hair .

The participants at the EC luncheon raised that some initiatives to change the dominance of white men in the decision-making process across VR/AR verticals are set up. For example, the American Women in XR Venture Fund (WXR) is dedicated to this mission , as well as Women in Immersive Tech Europe , counting more than 2000 members.

Concrete actions

At the EC luncheon at VR Days, the following suggestions were articulated by the attendees:

• Horizon Europe should explicitly include the gender balance in future call for proposals. This will advise consortia to implement strategies that address gender balance;
• The EC, national and regional funding schemes must implement funding schemes that support female start-ups;
• To achieve gender equity, a strict quota should be established. This has been successfully applied by the organisation running Venice VR in terms of female content creators, but this requirement should be considered as well for panels, submissions to workshops and conferences and funding events;
• To attract women in general to work, research and develop in the tech industry and XR specifically, which can be achieved through dedicated teaching programmes, upskill initiatives and other supportive activities;
• Mentoring programs should be established to further support women in XR. This can already be put in place within the XR4ALL framework;
• As part of the quality review process, hardware manufacturer and software providers should be requested to confirm whether or not women were involved during the design and review process. A new quality label or certification process has to be established.

Security and privacy of personal data

Context

Immersive/XR technologies provide a new set of issues and challenges concerning the “security and privacy of personal data”, where this should be understood as meaning both the security of personal data and the privacy of these data. On the one hand, “security” refers to the fact that our personal data are protected and not accessible by anyone without our expressed authorisation. On the other hand, “privacy” refers to the fact that we have full control on the release, or not, of our personal data.

Nowadays already, but even more in the future, headset or glasses have the capability of sensing and recording a multitude of data, which can be classified in user data and surrounding scene data:

• Information about the user such as:
  • Its position;
  • The gaze direction (via eye-tracking);
  • The speech;
  • The movement and gestures;
  • The emotional state;
  • Biological signals (EEG, ECG, …);
  • Objects, the user is looking at (via gaze tracking), either in the virtual world or in the real world
• Information about the world around us, which are objects, places and persons, acquired via:
  • Visual information (images or videos) by semantic image/video analysis, face recognition, gait analysis, and identification;
  • Sounds and speech via semantic sound classification, speech recognition, and speaker recognition/identification;
  • Geo-spatial information.

Related to privacy of data, the existence of these data about ourselves may mean that the employers may have a more precise understanding of our work cycle and habits, as well as about our emotional state. This is a phenomenon that one can already see in the case of drowsiness monitoring of professional drivers. They are not too eager that their employer knows about their state of alertness/drowsiness throughout their entire work day. Another pertinent example is related to job interviews. It has been reported that some companies are already using VR during job interviews to gather data and disqualify people on the basis of some unannounced algorithmic process, which is furthermore problematic due to AI bias, as well as other ethical reasons.

An important aspect is identification, which is already possible e.g. from the movement of a person (e.g. gait analysis), herby referring to the term “kinetic fingerprint” . In addition, once data about our physiology has been collected, such as EEG and ECG, it becomes possible to draw conclusions about our emotional state at any point in time and, certainly, our personality. Even, when AI/ML is involved, it is already possible to 'guess' highly intimate information about the user with even limited data: one study revealed that attributes such as sexual orientation, ethnicity, religion, political views, intelligence and gender can be inferred from publicly accessible digital records such as Facebook likes.

As for data tracking, the most recently launched Oculus Quest 2 turns out to be quite a controversial device: It requires a Facebook login to work. This move allows Facebook to immediately identify the device owner, but also have a better understanding of how the owner uses the device (based on biometric data tracking). The VR community reacted with a huge backlash. On top of all that new data becoming available to Facebook, a recent report of Stanford University confirms that the privacy policies of both, Oculus and HTC, the two most popular VR headset manufacturers in 2020, permit to share any de-identified data . Also, that they are able to reliably identify individuals after only a five-minute session in a standard consumer VR headset. Hence de-identified biometric data can be shared with third parties and they, in-turn, “could use it to figure out who the viewer is, predict their habits, understand their vulnerabilities and create marketing profiles intent on grabbing their attention with a new level of granularity”. Although there are many interesting and ‘good’ use cases for biometric data, the society is rightly worried the data can be abused. Furthermore, the accessibility to user data by Facebook is incompatible with regulations and rules in Europe e.g. GDPR, but especially in the healthcare sector due to privacy issues. The huge amount of personal data in therapy, pre- and post-operative applications has to be on the devices and it must be assured that these data are not transferred to some server operated by device manufacturers or unauthorized platform operators. Although governments are collecting data about their population, the private actors are currently dominating the world of data. Companies have no moral obligations vis-a-vis the users and personal data are transferred to third parties, notably advertisers. Even the introduction of explicit confirmation by the user to allow cookies on web sites, did not really change the situation. Users are annoyed by the check boxes and click “OK” just to proceed quickly.

Security of data becomes even more crucial in the case of spatial computing and the 3D digital representation of the real world and associated content and services (sometimes referred to as the “AR Cloud”), where users can easily access and store digital objects. At the moment, every vendor is providing his own digital mapping and relocalisation and there is no interoperability. Today, this means that when adopting XR, a user must choose one vendor’s technology and only use content created for or by that vendor’s platform. One objective of future development must be to enable users to choose the devices and software options of their choice, without having to choose the content or even more critical, to give up or narrow requirements on safety and privacy. Therefore, the devices and the technologies for content creation and delivery need to be separated from the place and content of experience delivery. There is a strong need for either extending existing or creating new systems to protect identity and privacy of users of AR Cloud services. The management of identity credentials and authorisation deployed across different systems must be assured. Interestingly, Facebook is addressing these aspects in its project ARIA at RealityLabs .

The question “Who is the owner of data?” must be answered and therefore, the EU needs to be pro-active rather than reactive. Data privacy should be an integral part of EC research, because Europe has a tradition of strength in the protection of the privacy for the general society, which led to the General Data Protection Regulation 2016/679 (GDPR). It is a regulation in EU law on data protection and privacy in the European Union and the European Economic Area. It also addresses the transfer of personal data outside the EU and European Economic Areas (EEA). This initiative signified an important breakthrough, but there are still challenges. As the first proposal for GDPR was released in 2012, the regulation entered into force in Mai 2016. Therefore, the regulation was not necessarily easy to introduce, due to the consensus requirement. In terms of security and privacy needs resulting from XR technologies, there is immediate action required given the time delays and challenges.

Concrete actions

The following concrete actions are recommended:

• Extend existing technical protections to XR technology;
• Extend existing regulations (including GDPR) to XR technology;
• Make sure that the upcoming spatial computing and AR cloud in Europe remains in the control of Europe, and especially not in the control of China and/or the USA;
• For security, it is suggested that one should embed the XR tools into traditional, well-known web browsers, and use the same, well-proven security technologies for browsers;
• For privacy, one should develop algorithms, primarily in the spatial domain, that will sense the user's circumstances, and automatically comply with privacy policies.

Social relations, behaviour and addiction

Context

In a sense, the impact of immersive/XR technologies on social relations, people behaviour, and people addiction will be very similar to what we know today as a result of a variety of tools such as email, Skype (and similar), and the various social networks. Of course, in the case of VR in particular, where people are shut-off from the real world and disappear into a virtual world, there will be new effects. Some of these effects have already been identified and studied in some depth.

On the one hand, the above mentioned tools are very useful to communicate with colleagues at work, with family members, and with close friends, when used appropriately and in moderation.

On the other hand, one can see the strong negative impact of these tools on face-to-face interactions. Anyone with an awareness of their surroundings will have noticed that virtually all people (in some age range) on a train or bus have their eyes looking down at the screen of their phone (when not working on a computer). This leads to missed opportunities for communicating with real people in the real world. In many cases, this reaches the point of being rude and impolite. The same lack of good manners happens when someone opens/looks at his/her phone every few minutes, even though they are travelling with another person and could use the time for direct interaction. Still on the negative side, people, especially young ones, worry more about their friends on Facebook and similar social networks, and the number of “likes” on Facebook, than about the real people, possibly yet unknown to them, that are around them, e.g. in a crowd or in a train. There is another addictive aspect of smartphones. Studies are beginning to show links between smartphone usage and increased levels of anxiety and depression, poor sleep quality, and increased risk of car injury or death .

One can predict that the same will happen for the general public with XR, which is, after all, a new powerful means of communication with other people, now through the use of avatars, whether life-like or magic-like. Of course, there are new dimensions with XR, since the people can meet in landscapes that can possibly be completely magic-like, and that they will never encounter in the real world.

Just as people will have the possibility of meeting in a possibly fantastic world, they will also have the possibility to meet with people that they will never, ever meet in the real world, and, importantly, to commit acts that they would never be able to commit in the real world. Of course, this is already true, to some extent, with videogames, whether they use VR or not.

In fact, most of the points raised above in a rather informal way, have progressively been analysed and categorised by experts. The paper by J.S. Spiegel gives a detailed account of the risks of VR. In the section on “Bodily neglect”, he says: “VR technology also raises serious concerns related to users’ personal neglect of their own actual bodies and physical environments.” Risks in this area are already evident in those who spend excessively long periods on social media . And there have been numerous reports in recent years of severe illnesses and fatalities due to video gamers neglecting their own physical well- being.

While reports of extreme self- and child-neglect are horrifying, they are not too surprising given the addictive nature of video games. And such incidents are likely to be increasingly common with the widespread use of VR. As Michael Cranford notes , “The greater our degree of participation, the more engrossed we become in the virtual world, and the less conscious of the real one”. One should also note that the above-mentioned papers also discuss the psychological dangers for some people of being exposed to specific situations, especially when these involve atrocities.

There is a real ethical issue at the application level since we can see that the effects of virtual incarnation are quite powerful. Real study in the design of applications that involve realistic avatars will be necessary since the effects of this realistic incarnation can persist in the real world for a more or less long time. We must be very vigilant about the experiences that users live in VR, and we must be aware of these psychological effects to propose applications that do not harm people.

The risk of addiction may not be higher for XR than for video games. Addiction to games does not concern so many people (~3% of gamers are addicts). Addiction belongs to the use of different media in general, not just XR technologies. Hence, addiction must be observed with vigilance and actions taken to reduce.

Concrete actions

The following concrete actions are recommended:

• One should examine carefully all existing publications on social relations, behaviour, and addiction in the context of XR, as well as on related topics, such as the physiological and psychological consequences of XR and extend these studies as necessary;
• The possible arrival of spatial-computing and the AR cloud will bring an interesting set of new issues that will need to be studied in a similar way.
• The outcome of the studies should be used to create a guidance for XR aware product and service generation. Furthermore, a regulated XR approved quality label could be an overall outcome.

Inclusion and fragile population

Context

Immersive technologies can offer a lot of advantages to people with disabilities in particular with cognitive deficiencies. XR technologies have the potential to support users with their healthy senses. Several applications such as serious games for medical purposes, training for phobias, and training for social behaviour are already in development. Some interesting applications have been developed for people with autism spectrum disorder.

XR technologies can also be very useful for socially isolated people and for distant learning and training. It will give access to expert training to people who may otherwise not be able to travel to attend in-person training.

However, current prices for headsets do not allow easy access to the technology and widespread use. This holds not only for people with disabilities, but for the majority of users who would be interested in this technology, but who cannot afford the current costs. In 2018, 16.8% of the population in the EU-27 were living at-a-risk-of poverty and 31.5% were unable to face unexpected expenses . This situation can be expressed by the term “Digital gap between Socio-Economic Groups”.

Beside costs of devices, there are additional aspects that limit the access for a wider audience:

• In a longer term, access to high bandwidth and low latency networks will be required to experience novel XR technologies;
• With regards to content creation, the content must be of good quality, diverse and inclusive to ensure it addresses the needs and desires of a wide audience;
• Usability and ease of use is a barrier to adoption of XR technologies.

However, there are further opportunities with regards to inclusion, especially for enterprise and industrial XR. These technologies will allow people to be more flexible and efficient in their jobs and to benefit from remote expert assistance, resulting in reduced expenses and time economies for the employers.

Concrete actions

In order to (1) spread XR technologies to a wider audience, to (2) increase the acceptance and the recognition of XR applications, and to (3) reduce the barriers to adopt this advanced technology, the following actions can be formulated:

• Provide a large range of XR products, some low cost, and other at a higher price with the latest technologies;
• In order to overcome the high costs of devices, the required infrastructure can be provided through public creative labs, educational frameworks, seminars, workshops, through public libraries and youth clubs. This could be provided and supported by the EC. This should also be set up in a way that is more inclusive to women, by making the technology attractive. A widespread availability will finally demystify this new technology;
• Content creation, usability and acceptability of XR technologies must be researched much deeper and cross disciplinary education and research is required. Therefore, people with diverse academic and disciplinary backgrounds must be included and brought together. Research funding should be accessible to researchers without a traditional computer science background. This will create interest in XR technologies outside of tech contexts.

Standardisation

Overview

Virtual reality technologies, created in the 1950s, have experienced a new craze due to the significant increase in sensing and processing capabilities. Addressing entertainment for the mass market, major IT actors have invested billions of dollars through merging, acquisition and innovation development, leading to a monopolistic situation where only a few players dominate the market, leaving little opportunity for smaller players. Unfortunately, virtual reality technologies have not been widely adopted in the same way as smartphones, and have faced a vicious circle: few users, few markets, few investments in content production, and so few users. Standards aiming at facilitating the porting of content to any VR platform has therefore proven to be a necessity to open up the ecosystem, and thus initiatives such as OpenXR (Khronos Group) and WebXR have emerged to improve interoperability and break this vicious circle.

Most device browsers across desktop, mobile and VR headsets have integrated the WebXR standard in the last version of their browser, including Apple/Safari. Facebook has invested a lot to make the latest Oculus Quest 2 browser the highest performing browser in a headset to date. The issue here though is that Facebook is leading and other HMD manufacturers don’t have the resources to follow this incredible achievement.

The augmented reality technologies that have emerged more recently are following the same path, but with a much higher level of investment. As AR is still at an early stage of adoption, some large players impose their AR ecosystem and are not very supportive of the development of standards. Nevertheless, some initiatives to standardise AR technologies are beginning to emerge.

Challenges

First, VR and AR systems involve many technologies such as sensors, computer vision processing, audio processing, 3D rendering, display, content authoring, data management, cloud, etc. Second, VR and AR systems and services address many specific domains (e.g. manufacturing, entertainment, construction and health). Specifying a global framework with low-level standardised interfaces (protocols, data formats or API) compliant with systems specific to the different applications domain addressed by AR is very challenging.

Moreover, developing a standard has no value if it is not adopted by the important stakeholders. Getting them involved in standardisation consortia is crucial for standard adoption. But the involvement of major players will only be achieved if the standards bring a benefit to their business model. Standardisation then becomes a tricky exercise seeking compromises to benefit the entire value chain, where a fair balance is essential to the interests of each actor.

Opportunities

The analysis gives way to a rather disappointing reality, where no European actor is present in the top 10 actors of mixed reality technologies, and most European actors have currently set itself this objective for the next five years. There is one exception, which is Lynx VR, based in Paris that offers a standalone headset combining VR/AR in a single device. Hence, it is difficult for European companies to gain a significant influence in standardisation bodies.

However, Europe has some strengths, which offer opportunities: First, many European research laboratories are internationally recognised. They are the source of many technologies implemented in current XR systems, they are very active in top international XR conferences (e.g. IEEE ICCV, IEEE ECCV, IEEE CVPR, IEEE VR or IEEE ISMAR) and hold many chair positions.

Secondly, Europe is one of the regions where the expected revenue relative to XR technologies is the most important for the next five years according to Digi Capital reports. And technology providers are not the only one to weigh in the balance; end-users and content providers can act as a counterweight in the power balance, provided that their actions are coordinated to have a greater impact.

The latter is one of the main objectives of the Industry Specification Group “Augmented Reality Framework” at ETSI: to define a framework for the interoperability of AR components, systems and services. The group aims to identify industry requirements, specify some standardised interfaces, and publish them or to submit industry requirements in a coordinated manner to other international standards.

Although XR hardware sales represent a significant part of revenue, many other areas are also valuable such as cloud services and connectivity, many industrial applications, in which European companies have a global weight (e.g. car industry, mechanical engineering), content authoring, domain-specific implementation, etc. And, in most of this domain, European players have many opportunities as they have great expertise in key elements such as telecom infrastructure, data, cultural specificities, creativity, etc.

Finally, data protection and privacy should be a priority for standardisation for which European actors should mobilise in a coordinated way. Indeed, XR technologies will increasingly rely on cloud technologies, and much data will be hosted on the servers of XR service providers. Knowing that XR systems scan the XR user's environment on a very large scale (also called XRCloud), it becomes essential to ensure the confidentiality of the data used by these systems. This is a matter for European sovereignty, and standardisation is the key to preserve it in this technology area.

Recommendations

The following recommendation can be given:

• Encourage the involvement of the scientific community in standardisation consortia, promote XR standards and recommend their adoption to accelerate the transfer of research results from European laboratories to industry and production;
• Encourage European standardisation initiatives involving strong European players to act in a coordinated manner to weigh in international standardisation consortia;
• Make data protection and privacy related to XR technologies a main European priority for the coming years (and not only in standardisation);
• Oblige headset manufacturers that wish to enter Europe to include a WebXR enabled browser. This will avoid the monopoly of the headset store and data tracking in the browser needs to follow the European GDPR guidelines;
• Push towards a standard for the exchange of localisation data, which is required to establish an open and accessible XR cloud.

EC policy and funding

Overview

By analysing the XR landscape, there is a severe fragmentation in policies and technology development and a lack of funding in Europe. While XR technologies may still not be quite mainstream, this industry with hundreds of companies, tens of thousands of employees, and over $4 billion in investments to date is set for a bright future, although further financing remains key for its success. Especially for start-ups and early stage companies, there is a need to improve funding opportunities and increase its values, which holds for both public and private funding.

Opportunities

Various surveys about the future development of XR technologies show that there are several opportunities for Europe to play an important role in the future, although the focus will not be in the domain of HMDs for the consumer mass market, which is occupied by large US and Asian companies (Facebook, Google, Microsoft, HTC). Concerning the overall revenue in XR technologies, the enquiry in sees Europe in 2023 even at second position of worldwide revenue regions (25%) after Asia (51%) followed by North America (17%). In a study about the VR and AR ecosystem in Europe in 2016/2017 , Ecorys identified the potential for Europe when playing out its strengths, namely building on its creativity, skills, and cultural diversity.

European innovation in AR and VR is largely driven by SMEs and start-ups. Several acquisitions of small companies by big US players demonstrated this in the past.

One major challenge is still the hardware like AR glasses and VR headsets. The acceptability of the mass-market and the suitability of whole day use requires a significant progress in technology development in the coming years. However, the main technology developers and manufacturers for XR devices are from outside Europe showing an aggressive marketing strategy. One example is Facebook, which wiped out nearly all competitors from the market. The current pricing of Oculus devices is much cheaper than the XR2 chipset, it uses. However, Europe has the opportunity to compete with non-European companies in the areas of microelectronics and optics e.g. Carl Zeiss and Bosch. Europe also consists of strong research capacities represented e.g. by DFKI and Fraunhofer in Germany and INRIA, IRT in France. There are also excellent companies in Switzerland that are coming out of ETHZ and EPFL. Beside the XR devices market, there are several other topics, where European research, SMEs and industry play an important role e.g. application development, content creation and content authoring, user experience and many more. Here, an increase of funding and investment will strengthen the European position and foster the XR community.

Especially, in view of privacy and security of data, a European answer must be given to the data policy of the major industry players in the device market. The need for a Facebook account to run the new Oculus Quest will hinder many companies and application domains using this device. Hence, a support for European SMEs, start-ups and industry becomes even more important.

Horizon Europe, the next research and innovation framework programme with an ambitious budget of €100 billion for research and innovation, is another opportunity for Europe to position XR as a foundational technology accelerating the digital transformation of industry and services and providing new opportunities to innovate, create new ventures and to invest in technologies and applications that will deeply change the existing business models. As seen in section #Motivation and context of this document, three areas of Horizon Europe can be identified, which relate to future research and innovation of XR technologies. There is a need for new research infrastructures as set in Pillar 1. Secondly, XR technologies are considered as an axis spanning across all six clusters described in Pillar 2. Finally, it also has the potential to serve as a base for the innovation ecosystem as presented in Pillar 3.

General recommendations

Based on the feedback received by the community, several recommendations can be derived that target the role of the EC:

• Although promised, the next framework programme Horizon Europe should have much less administration and shorter period between proposal submission and grant agreement;
• Financial support for creative industries is required to re-tool, experiment and innovate in this new medium;
• In order to support gender balance in this innovative area, gender balance should be stated explicitly in the call text for future EC funding schemes and dedicated supportive programmes for women in XR tech industry should be established;
• The continuation of Open Source policy offers definitely a benefit for the stakeholders in the XR community and Europe in general, as Open Source must then be in line with EU guidelines in data-tracking and privacy. But, for an SME to provide an open-source offering, one needs again developers, time and resources, which should be funded by the EU with special funding and grant schemes;
• There is a need for more transdisciplinary collaborative research and more support for social sciences that are linked to projects working on technologies. More collaborative projects should be supported that consist of creatives, UX/UI designers and content creators on one hand, and technology developers on other hand;
• A funding for corporates working together with SMEs and start-ups will help to accelerate the Technology Readiness Level (TRL) in a relevant environment;
• There needs to be more platforms and events, where information and best practices are shared. This will support the discussion about standardisation and help to fasten the go-to-market strategy. These platforms can be used for conducting research, to address specific questions and to collect requirements from the experts in the market on demand.

Major topics and actions to be addressed in future calls

Based on the major challenges listed in Sec. #Current state and challenges of XR technologies and the short and long-term vision in Sec. #Short and long-term vision in XR, a set of highly relevant topics and actions can be derived. The following list of topics does not presume an order of priority. It further complements the recommendations and concrete actions given in the previous sections.

XR sensing and feedback

Specific challenge

Future XR applications require ubiquitous sensing of and feedback to the user. The user should be able to feel, touch and interact with the virtual world in a truly immersive manner. This also includes the removal of controllers and to allow for a realistic interaction purely with its hands, body posture, eyes, speech etc. However, accuracy and robustness are still an issue in the tracking and sensing domain. On the other hand, the sensation of presence in extended reality applications is lacking the necessary feedback beside audio-visual cues. Especially, interacting with virtual objects requires a convincing sense of touch. This will increase the feeling of tele-presence and improve the interaction due to immediate feedback. However, the availability of mature haptic feedback systems is not yet there. Forced feedback systems provide a reasonable sense of touch, but they are highly specialised and expensive. Haptic feedback devices based on ultrasound are in a more mature state and available as consumer-like products, but the sense of haptic is quite limited. Moreover, risk assessment of such an approach is not yet fully performed, because strong and uncontrolled ultrasound sources may harm other senses like vision and hearing. Beside vision, hearing and touch, other senses like smell and taste shall be investigated. Therefore, fundamental research towards easy-to-use devices with convincing sense of touch supporting a wide range of materials, surfaces and structures is required.

Scope

Research and development is suggested in the following areas:

• Development of fast, reliable and accurate sensing technology of human behaviour (touch, point, grab,…);
• Research and development in generalised technical solutions, which are independent of the scenario and environment (indoor, outdoor);
• Development of easy to use and to learn systems, and being able to support user specific capabilities to increase robustness;
• Completely new approaches for haptic feedback;
• Research in ultrasound including its implication on health;
• Development of gloves that include sensing as well as haptic feedback;
• Research on thermal and radiation imaging;
• Exploring additional senses like smell and taste;
• Research in non-invasive biometric sensors (e.g. electrocardiogram, electrodermal activity, electroencephalography, electromyography, or photoplethysmogram) for estimating user state (e.g. cognitive load, emotion, or sickness) and Brain Computer Interaction (BCI) in general;
• Auto-adaptive XR scenarios according to user feedback.

Expected impact

Investment in and funding of sensing and haptic technology will

• Increase the acceptance and usability of future XR technology;
• Increase the number of applications and use cases XR technology can be used for;
• Strengthen the position of European SMEs already active in the market.

World capture and understanding

Specific challenge

Spaces and objects become increasingly instrumented with network-connected sensors and users in those spaces carry and wear more sophisticated sensors on their devices. More particularly, XR devices embed vision sensors able to capture continuously the environment of the user. Spatial computing technologies are now able to reconstruct a 3D digital twin of the real world based on vision sensors, and machine learning approaches can analyse this digital twin in real-time to extract highly valuable information (users’ activity recognition, semantic segmentation of the environment, tracking of objects, etc.). To address large-scale environment, or even global scale, spatial computing processing will have to be distributed in the cloud (edge and remote) relying on 5G connectivity offering very low latency, so that mapping of the real world will be crowd-sourced. Spatial computing and world understanding based on XR cloud technologies will become a key technology for Industry 4.0, construction, smart cities, health, protection of the territory and many other verticals. The main leaders in the XR market who invest hundreds of millions of dollars each year in technological development and acquisitions related to XR cloud are likely to impose a monopoly where no European players will be able to emerge, leading to a worrying situation for sovereignty and privacy.

Scope

The following actions will contribute to address the listed challenges:

• Develop a sovereign European platform for XR cloud;
• Establish European sovereignty in terms of datasets needed for XR applications;
• Encourage open alternatives to build an ecosystem with a diverse range of solution providers including smaller players, new entrants and academics;
• Develop cutting-edge technologies in spatial computing and scene understanding;
• Support standardisation activities to improve interoperability;
• Ensure anonymisation of the captured data for the protection of the privacy;
• Drive regulations for the future of XR cloud technologies to ensure ethics;
• Improve XR cloud sobriety in terms of ecological footprint;
• Explore use cases enriched by XR cloud technology.

Expected impact

The following impact is expected:

• European sovereignty on XR cloud technology;
• Promote European players working on XR cloud technology;
• Prepare regulations related to XR technology;
• Reduce ecological footprint of XR technology.

Social interaction and collaboration

Specific challenge

The current pandemic situation with Covid-19 shows a strong need for significant improvement of technologies that support social interaction across distances, remote collaboration and communication. During 2020, almost all scientific conferences, industrial fairs, workshops were transformed into online events. However, the current solutions for online events do not offer the same level of experience compared to physical events. Personal exchange of views, private discussions and establishing new personal relationships is rather difficult. The virtual conference applications are mainly based on artificial avatar representations of participants; therefore, non-verbal communication is impossible. On the other hand, it is clear that a complete fall back to physical meetings and events after the pandemic will not happen anymore. Companies and the research community recognised that online events offer a huge amount of time and cost savings. The impact on climate change by reducing the ecological footprint is also quite important. Therefore, a new way of meetings, so-called hybrid events needs to be researched, developed and established. Hybrid meetings means that real and online participants meet in a common meeting space, which is real and virtual at the same time, without recognising the boundary between real and virtual. All aspects of non-verbal communication as well as interaction are preserved in this real-virtual meeting space.

The recent FET-PROACTIVE call in Horizon 2020 (FETPROACT-EIC-07-2020) asked for projects dedicated to research and development of future technology-mediated social interaction with a planned funding of 12 million Euro leading to 3-4 projects. This funding initiative is focusing on cutting-edge high-risk research and innovation projects aiming at artificial intelligence for extended social interaction. However, the digital society asks for immediate solutions towards social XR applications and remote and distributed working.

Especially significantly improved real-time communication is required to enable and improve services and applications. In the industrial environment and in companies, training and remote assistance will improve workflows and knowledge transfer. In the health-care sector, immersive communication of patient with hospital, remote experts support during diagnostic and surgery and online education and training of medical staff need to be implemented in large scale. However, several important aspects of truly immersive communication are not yet mature enough and current XR-enabled solutions do not offer the feeling of social presence. There is a lack of representing humans in a natural and truly immersive way that enables to convey verbal and non-verbal cues of social interaction. With respect to inclusiveness, research and development is required to allow handicapped people to take part in the same way in novel technology-mediated social interaction. The integration of other senses like smell and touch is still in a prototypical stage, but will enhance the feeling of presence. Truly touchless interaction with people, objects and content is still limited in current XR applications.

Scope

The scope of this research topic covers the following areas:

• Seamless communication and interaction in real-virtual meeting spaces;
• A realistic representation of humans is mandatory. This requires a significant progress on volumetric video technology in various aspects, such as:
  • Automatic and fast volumetric capture and processing with focus on low-cost capture set-ups;
  • Definition of standardised data format including encoding schemes;
  • Authoring tools for modifying and manipulating volumetric assets in terms of geometry and texture.
• Real-time interaction and communication with a special focus on 3D capture of humans, remote and touchless interaction and steering of devices;
• Research on analysis, transmission and representation of non-verbal cues such as gaze, pointing, emotions, etc. This includes also the transfer of non-verbal cues across devices e.g. between tablet and VR;
• Research on how to transfer interactions and access virtual spaces using cross platforms.

Expected impact

The impact of this research topic is expected as follows:

• Reduction of travel and therefore reduction of carbon footprint;
• Increased user acceptance of new XR technologies due to realistic and truly immersive representation and interaction;
• Inclusion of handicapped people in future social interaction;
• Improved workflows through easy-to-use remote collaboration and training leading to reduction of costs and user satisfaction.

Open XR ecosystem and collaboration platform

Specific challenge

The current XR ecosystem is highly fragmented. A number of European, national and regional organisations and associations put tremendous effort on bringing stakeholders of the XR community together, to share ideas and knowledge; and to support product development and marketisation. For example, EuroXR follows a similar strategy compared to XR4ALL to foster the XR community, gather relevant stakeholders, establish connections, promote research excellence, and facilitate the structuring and VR/AR research integration in Europe.

Furthermore, a common marketplace for XR solutions, technologies and tools is not yet available. This makes it difficult for SMEs to advertise their products, solutions and services to a wider audience. The huge amount of XR applications especially from research projects and academia is not easily accessible. A similar situation can be observed for recruiting skilled experts. Especially for the XR industry, but also in research, specialised expertise is required to fulfil the needs of a dynamically changing technology domain. For SMEs, industry and academia, it is difficult to reach the right target group with their job offers. At the same time, developers and young talents cannot approach the relevant employers.

Another limitation to foster collaboration on a European level is a missing data platform that offers trust and security for European industry, SMEs, start-ups and academia. Open source code, publicly available test data sets, libraries of assets and 3D data are distributed among different platforms and not commonly accessible.

Scope

Actions to address the challenges for an open XR ecosystem and collaboration platform could include:

• Creating a publicly available marketplace, where all stakeholders can offer their applications and technologies. This marketplace should also act as a place for networking and information exchange between companies, SMEs start-ups and academia;
• Offering a central access point for employment exchange between companies and academia on one hand, and young talents, developers and creatives on the other hand, to improve the job-skills matching;
• Providing a commonly accessible data platform that fulfils the needs of GDPR and facilitates the exchange of data;
• Promote and support a Pan-European umbrella organisation for the XR community that manages and services a commonly accessible collaboration and data platform.

Expected impact

A commonly accessible XR ecosystem will support:

• Job opportunities for young talents and offering the required skilled experts to stakeholders;
• Widespread availability of products, tools and applications for the whole community to support knowledge transfer, marketisation and networking;
• Fostering of community building.

Support for start-ups and SMEs

Specific challenge

Horizon 2020 programmes are too complex for young start-ups looking for small amounts of funding needed to turn an existing technology into a product and reach markets. Low-risk funding schemes for start-ups providing support at short amount of time are not available. In addition to that, many start-ups require accelerator type support and funding to kick-off their product idea especially in the area of XR technologies.

For a successful support of SMEs, the right market placement, knowing the market and knowing the niche is essential. It is still difficult to find investors and VCs in Europe to get the necessary financial support compared to the American market. The current national and EC funding is far too slow to cope with the dynamic change of the technology and market. Time spans of 2-3 years before entering the market with a new product is too long to compete with companies working in other regions of the world where capital is more easily available.

Thus, universities, industry and customers need to collaborate in technology and business development acceleration programs, such as innovation hubs to accelerate the process and reduce time to market. It is necessary to identify the best use cases, the best talent and the real needs in the markets. More risky funding schemes are necessary because funding can be considered a statistical process. It can never be guaranteed that all funds invested will succeed. Therefore, greater trial and error-based approach to funding will help to identify the successful ideas and products sooner.

A good instrument that has proven its attractiveness through short evaluation period and immediate funding is the cascaded funding instrument initiated by XR4ALL. The Open Call for instance has attracted huge interest from a high number of teams to submit applications. The number of submissions for the three cut-off dates reads as follows:

• First cut-off date (29 November 2019): 90 applications submitted;
• Second cut-off date (26 June 2020): 138 applications submitted;
• Third cut-off date (16 November 2020): 158 applications submitted.

Although the maximum funding XR4ALL provides to the selected projects to produce a fully working solution is very limited (maximum €50,000), there is a strong appetite for small size funding with a simplified application procedure and shorter project duration. EU funded projects with cascaded funding and providing business support should therefore be encouraged.

From another side, an analysis of the funding rounds that have taken place since Q1 2015 shows that around 70% have been labelled as either a seed round, a grant, or an angel investment. This clearly confirms that the majority of the deals stay under €5 million, the investment amounts in start-ups such as Improbable (€443.7 million in May 2017) and Mindmaze (€98 million in September 2017) being huge and rare exceptions in Europe. With private investors in Europe more risk-averse compared to investors in North America or Asia, European companies are forced to either be a market leader in a niche market segment or be acquired by large foreign corporations. In general, the number of investors who exclusively invest in VR or AR is low. According to the analysis of funding rounds, a potential investment in VR/AR is merely the consequence of the fact that VR/AR is part of the value proposition the start-up offers (e.g. investing in healthcare start-ups, where VR is part of the service). Interviews conducted with investors by XR4ALL confirm that many investors are still reluctant to invest in XR as such but rather in applications in specific domains/verticals where there is a clear market. Private investors are in general very selective and usually look for qualified deal-flow. This creates a gap between innovative XR companies and investment opportunities.

XR4ALL’s XR Tech Venture Forum in Brussels on 13 December 2019 was only one step in this direction, but it has generated a lot of interest from XR start-ups. For the project pitches of start-ups to investors, more than 70 companies applied for participation, from which 32 have been selected and invited to pitch and meet with the participating investors. This shows a need to provide business support for XR start-ups to raise their awareness and readiness about their market and investment strategies and create matchmaking and networking opportunities for XR solution developers to meet with potential investors and partners.

Scope

Actions to increase investment opportunities for XR start-ups are needed and could include (but not limited to):

• Gaining a better understanding of the existing barriers/obstacles seen from the perspective of companies to enhance the access to funding and to provide insights into the most important financial needs companies have at different stages and times;
• Enhancing the awareness and readiness of innovative projects and solutions in XR about their market and investment strategies to make them more attractive for investors and potential partners;
• Creating awareness about financial instruments, policies and EU programs in other domains where XR could apply and making them sound to innovators to better benefit from these opportunities. Financial instruments could also leverage private finance (from Venture Capital and Business Angels) to encourage public-private partnerships and reduce the risks for private investors and crowd in more investments into the industry;
• Raising awareness about the potential of XR Technologies towards EU financial instruments (including the European Innovation Council - EIC, the instruments managed by the EIB & EIF) to be considered as enabling technologies worthy to support at the same level such as AI, Blockchain, etc. This will create new opportunities for XR solutions developers;
• Raising awareness and visibility of investment ready innovative projects and solutions in XR with the broader investment community (financial intermediaries, private investors, National promotional banks, EU financial instruments such as the European Innovation Council - EIC). This could support investors and technology companies to bridge the existing knowledge gap and thus to reduce the information asymmetry between innovators and investors;
• Funding schemes (cascaded funding) dedicated to start-ups in XR need to be supported and fulfil the following aspects: (1) Smaller grants; (2) Faster and easier application process; (3) Shorter review period; (4) Faster provision of funding combined with acceleration services;
• Empowering XR clusters and innovation agencies and building their capacity to better support XR start-ups in their ecosystem and make them benefit from available funding opportunities.

Expected impact

The consideration of the proposed actions listed above will lead to:

• Increased number of XR start-ups ready for market and investment;
• Increased participation of XR start-ups in funding programmes in other domains where XR could solve challenges;
• Raise capacity of XR clusters and innovation agencies to better support start-ups in their ecosystem;
• XR start-ups funded, accelerated with improved market and investment strategies.

Conclusion

This document provides a summary of the major challenges and the vision of XR technologies both for the world and for Europe at the time of writing in 2020.

Several instruments were implemented during the last 18 months to gather the opinions and views from the XR community in Europe. Two online surveys, expert interviews, a luncheon at VRDays 2019 in Amsterdam and online sessions of the ‘Engineering XR for the future’ event led to a total of more than 380 inputs from XR experts.

These experts cover the whole spectrum of stakeholders in this domain, such as leading experts from industry and academia, investors, representatives from start-ups, creators/artists, and policy-makers.

In addition to the challenges and the vision in the XR technology domain, a strong focus in this document is on ethical aspects namely gender balance, ethics and privacy, social relations and inclusion.

Beside detailed discussions, the report also provides concrete recommendations on how to approach the issues raised. An important area is standardisation, where Europe has significant potential to influence the development of XR technologies.

Finally, the exchange with the XR community also led to a constructive discussion on current EC policy and funding strategies. Several recommendations are given to adapt the funding to the current needs of the XR community. The authors believe that this document provides valuable input for successful future discussions and brainstorming within the European Commission and within the XR community about how to best position the European R&D community and industry in the competitive, worldwide field of immersive/XR and interactive technologies and how to shape the next framework programme Horizon Europe.

---

[1] Optical see-through (OST) refers to devices, where the real environment is seen through a transparent screen and virtual objects are overlaid onto the display.

[2] Video see-through (VST) refers to devices that capture the real world with a camera and virtual elements are rendered on the captured image.

[3] GAIA-X is a project initiated by Europe for Europe. Its aim is to develop common requirements for a European data infrastructure. Therefore openness, transparency and the ability to connect to other European countries are central to GAIA-X. Representatives from seven European countries are currently involved in the project.

Authors

Name	Organisation	Country
Oliver Schreer	Fraunhofer HHI	Germany
Ivanka Pelivan	Fraunhofer HHI	Germany
Ralf Schäfer	Fraunhofer HHI	Germany
Youssef Sabbah	Europe Unlimited S.A.	Belgium
Jérôme Royan	b<>com	France
Muriel Deschanel	b<>com	France
Jacques Verly	Image & 3D Europe	Belgium
Alain Gallez	Image & 3D Europe	Belgium
Sylvain Grain	Image & 3D Europe	Belgium
Alexandra Gérard	Image & 3D Europe	Belgium
Leen Segers	LucidWeb	Belgium
Maelle Quevillard	LucidWeb	Belgium
Jan Plüger	advisXR	Germany