10 Scientific Problems in Virtual Reality

Virtual Reality was one of the 14 Grand Challenges identified as awaiting engineering solutions for the 21st century announced in 2008 by the U.S. National Academy of Engineering (NAE http://www.nae.edu/). Here, I explore 10 related open VR challenges, with hoped-for potential breakthroughs promising to advance VR techniques and applications.

VR is today being applied in multiple contexts, including training, exercise, engineering design, and entertainment, while also serving as a research tool in such fields as neuroscience⁶ and psychology, as explored in Michael Heim’s pioneering 1993 book Metaphysics of Virtual Reality (http://www.mheim.com/books.html).² More recently, scholars have described the Internet itself as representing a virtual world modeling its real-world counterpart.⁵ The relationship between VR and its application fields is, in terms of expression and validation, like the relationship between mathematics and physics, while VR is attracting attention from a growing number of governments and science/engineering communities. Along with the NAE Committee on Engineering 14 Grand Challenges (http://www.engineeringchallenges.org/), the Chinese government’s 2006 report Development Plan Outline for Medium- and Long-Term Science and Technology Development (2006–2020) (http://www.gov.cn/jrzg/2006-02/09/content_183787.htm) and the Japanese government’s 2007 report Innovation 2025 (http://www.cao.go.jp/innovation/index.html) both included VR as a priority technology worthy of development.

VR has likewise emerged as an important research area for many Chinese universities and research institutes,⁷ ranging from theoretical foundations to technology innovation, system development, and practical applications. For example, Zhejiang University in Hangzhou and Tsinghua University in Beijing are known for realistic modeling and rendering; Peking University in Beijing focuses on computer vision and human-machine interaction; the Beijing Institute of Technology in Beijing emphasizes head-mounted displays; and the Institute of Computing Technology of Chinese Academy of Sciences in Beijing has made significant progress in crowd simulation.

Since 1995, I have directed the State Key Laboratory of Virtual Reality Technology and Systems of China (http://www.skvrlab.net) in Beijing, focusing on VR. Over that time, we have proposed many new methods and technologies in distributed virtual environments, realistic modeling and rendering, and augmented reality, including two major VR software infrastructures: BH_RTI for distributed virtual environments (http://www.hlarti.com) and BH_GRAPH for real-time 3D rendering (http://www.skvrlab.net:8080/soft). This work has produced several influential VR application systems, including the 2008 Beijing Olympic Games Opening Ceremony Simulation Platform and the Beijing 08 Digital Museum. We also designed a training-and-scenario-simulation system for a military review of China’s 2009 60th anniversary celebration, with more than 8,000 soldiers, based on visualized immersion technologies to deliver scenario preview, modification, and decision-making functions. Meanwhile, our work advancing theory in related fundamental disciplines may help produce further breakthroughs in VR; for example, electrorheological fluids⁴ could change the existing method of haptic rendering.

Back to Top

Most Important Problems

My research and practical experience have led me to identify what I consider the 10 most important VR problems inhibiting development of VR technology and applications. A breakthrough in any of them could significantly advance development of VR techniques and applications. My aim here is to outline them, motivate discussion/solutions, and ultimately help deliver solutions.

1. Digital models. Is it possible to (digitally) model any given object in the physical world? The problem of whether an object can be digitized is related to such factors as the developer’s understanding of that world, the complexity and computability of a particular model, and a typical user’s expectation of the resemblance of the model to its real counterpart. For example, experienced developers easily model rigid objects with great precision but find it extremely difficult to model fluids; fluid simulation equations are still far from reflecting a fluid’s natural variability. Moreover, there is simply no knowing whether human thinking will ever be modeled. Is it possible to establish a “modelability” theory along the lines of the theory of computability? What even is the relationship between modelability and computability? Clearly, answers require further investigation.
2. Complexity. How can the complexity of an object/model be evaluated? Is it possible to measure the complexity of real objects, as well as their virtual counterparts? Is there a uniform scale of such measurement? Countless real objects cannot be completely modeled due to multiple related complex factors. As a practical matter, scientists simplify these factors to create less-than-perfect models; for example, modeling ocean-wave dynamics is notably difficult, while a simplified conceptual wave is relatively easy. Moreover, no realistic weather model is totally consistent with the real weather, but meteorologists regularly produce simplified meteorological models for weather forecasting. Thus the complexities of real objects and related models have a direct relationship with modelability; for example, the number of triangles, vertices, or voxels is commonly used by VR developers as a measure of the complexities of geometry models but are far from sufficient for the computational complexities of physical laws and behavior models. Identifying new measures for model complexity also requires further research.
3. Model credibility. What factors influence model credibility? Can that credibility be measured? By simplifying real-world mechanisms, scientists today create models that previously could not be created. Most current digital 3D models require simplification, but to what extent does a simplified model reflect reality? Full immersion for a user in a model of a complex scene is not always necessary¹; related issues include how to anticipate how user expectations might affect VR application credibility and how to balance VR-application modelability and usability.
4. Model similarity. How is model similarity measured? Measuring similarity is a fundamental problem in model recognition and retrieval, with existing methods far from satisfactory. The difficulty mainly involves two aspects of model creation: no universally accepted theory of the binary relation of similarity (such as the axiomatic definition of properties) and no unified treatment due to model complexity and diverse classification. This leads to two questions: Is it possible to establish an axiomatic definition and a theory of similarities? And how might similarity measurement and metrics be made to cover different model classifications?
5. Image quality. What standards are available for evaluating image similarity and image quality? As in model similarity, image similarity represents a fundamental problem in image recognition and retrieval, and is thus important in VR. Humans generally recognize similarity between images and degree of image distortion, but, for computer vision, such recognition remains unsolved; for example, there is no universal efficient measure of image similarity or universal way to evaluate image clarity and distortion.
6. Real-time fidelity. What factors determine the gap between fidelity and real-time aspects of virtual environments? The fidelity of virtual environments is positively related to a system’s real-time performance. Models and real objects are contradictory, reflecting a duality; for example, the denser the triangles in an object-geometry model, the fewer the differences between virtual and real appearance and the longer the time needed to render an image, as in the duality of space and time in algorithm complexity,³ where time and space represent interchangeable computational resources. If only VR researchers could provide a theoretical proof of the duality between fidelity and real-time performance, as well as identify its origin, VR developers would more likely be able to achieve a proper balance between them in applications.
7. Basic (meta) elements. What basic elements determine the physical behavior of real objects? Object-geometry models are constructed from points, lines, and faces and their topological transformations, while most colors in nature are generated from the colors red, green, and blue. What similar elements are involved in the physical behavior of real objects? and How can these elements (such as those for material construction and fluid movement) be categorized? Identifying them would help advance development of physics processing units in the same way graphics processing units have been developed over the past 10 years.
8. Material and behavior models. What new, deep understanding of the natural world and the forces driving its behavior are required to develop object/material-movement models and human-behavior models? Which models of object/material movement and human behavior are P hard, NP hard, and NP-complete? Which are incomputable? Answering would help scientists implement models and evaluate their credibility with real-world objects and usability in routine applications.
9. Image integration. Can image-based and graphics-based models be integrated seamlessly? Can geometry models and image-based environments be integrated seamlessly? What key features characterize seamless integration? How can they be evaluated? These questions are fundamental to augmented reality.
10. Performance evaluation. Most performance indexes lack metrics for VR systems, even when rough classifications are available. But, due to the importance of VR applications, research is needed for developing future technologies, and related toolkits are needed to achieve practical results.

Back to Top

Conclusion

Taking a computational point of view, I have sought to outline the 10 most important scientific and technological problems in VR, aiming to inspire the field of computer science to help develop related practical applications.

Back to Top

Acknowledgment

I would like to thank the anonymous reviewers for their valuable comments.

Back to Top

Back to Top

Back to Top

Back to Top

Back to Top