Researchers have been hard at work redefining the human-machine interface, in particular looking at new ways we can interact with computers through touch, without actually touching something.
Holograms are not new technology, but there is a futuristic frisson surrounding the topic. A computer-generated hologram is created by a sequence of three-dimensional (3D) images that are processed into a virtual image, a visual illusion. If you try to touch one, your hand will go through it.
What is new is the concept of touchable holograms: not just projected into the air, and not just superimposed onto an actual object, but “haptic holograms” that you can not only touch, but interact with and move. Computer haptics are the systems required—both hardware and software—to render the touch and feel of virtual objects. Haptic holograms take this one step further: you can now “touch” a 3D projection, a virtual object, and actually feel it.
Haptic holograms create virtual objects that have a digital interface, an interface that is feel-able as well as visible, by sculpting sound to make visible digital features feel like physical ones. The virtual 3D haptic shape becomes a tactile holographic display.
A Touching Story
The skin covering the hand is packed with receptors that communicate tactile feedback for light touch, heavy touch, pressure, vibration, hot and cold, and pain. This helps the brain understand subtle tactile details: smoothness, hardness, density, weight, and so on.
Ultrasound creates vibrations in the air, projected at a set distance to match the surface of the hologram. The skin “feels” these vibrations at different wave-lengths to simulate softness/hardness and more. This information enables a virtual, 3D image to be “touched.”
To assist in the design and development of tactile interface applications, Marianna Obrist, a visiting researcher at Newcastle University and Lecturer in Interaction Design at the University of Sussex, and her colleagues created a tactile vocabulary of 14 categories of haptic feedback, such as prickly/tingling, coming/going, and pulsing/flowing. “Pulsing” is a 16Hz vibration that stimulates the Meissner’s corpuscle receptors in the skin that are responsible for sensitivity to light touch.
Sriram Subramanian, professor of Human-Computer Interaction in the Computer Science Department at Bristol University, co-directs the Interaction and Graphics Group. The group, led by research assistant Ben Long, developed the UltraHaptics (http://ultrahaptics.com/) system, which creates haptic feedback in mid-air (https://youtu.be/H6U7hI_zIyU). Waves of ultrasound displace the air, causing a pressure difference. When multiple waves arrive at the same place simultaneously, a noticeable pressure difference is created at that point. According to Long, “Touchable holograms, immersive virtual reality (VR) that you can feel, and complex touchable controls in free space are all possible ways of using this system.”
The addition of a Leap Motion (https://www.leapmotion.com/) controller—an infrared sensor that tracks the precise position of a user’s fingers in 3D space—enables ultrasound to be directed accurately at a user’s hands to produce the sensation of touch, creating the impression of exploring the surface of an object, which enhances VR. The group is working on using more complex shapes with greater detail, possibly by having a greater number of smaller speakers to improve the resolution.
A key challenge, Subramanian explains, is in “understanding in greater detail how users perceive haptic feedback with a mid-air haptic system like ours, which is able to target multiple mechanoreceptors simultaneously. This will play an important role in improving the fidelity of the tactile feedback perceived.”
Subramanian envisions a range of applications for touchless haptics. This includes applications in automotive dashboards in which the user can interact with the dashboard without taking their eyes off the road (for example, the user can wave their hand in front of the dashboard to sense feel-able knobs and dials).
Another key application is in VR in developing lightweight, high-resolution head-mounted displays. The Bristol group believes 3D haptics can play a “huge role” in increasing user immersion in VR games and applications. Their spin-off company, Ultrahaptics, is focusing on embedding the technology in a number of different products ranging from alarm clocks to home appliances to cars.
Researchers at the University of Tokyo, led by Hiroyuki Shinoda, created the Airborne Ultrasound Tactile Display which, for example, allows a user to feel hologram raindrops bouncing off their hands. Later, Yasuaki Monnai, project assistant professor of the university’s Department of Creative Informatics, and his colleagues at the Shinoda-Makino Lab created a 2D touchscreen floating in 3D. Hapto-Mime (http://youtu.be/uARGRlp-CWg8) uses both ultrasound waves and infrared sensors to give hands-free tactile feedback. The interface is a virtual, holographic display on an ultra-thin, floating reflective surface. Ultrasound exerts a mechanical force where the beam is focused, which allows a virtual object, such as a piano keyboard or an ATM number pad, to be felt, as changes in ultrasonic pressure give the illusion of different touch sensations.
The Tokyo team is particularly interested in electromagnetic wave propagation and transmission systems, applying them to wireless communication, measurement, and human-machine interfaces. According to Monnai, they anticipate guiding human motions using the virtual image and force. “In our current system, users touch the hologram, but in future, it is also possible that the hologram touches users. This will enable, for example, having a virtual sport coach who tells you how to move your body by stimulating you with visual and haptic sensations at the correct timing and position.”
The Bristol group believes 3D haptics can play a “huge role” in increasing user immersion in VR games and applications.
The haptic feedback in their system is currently quite weak in strength. To present greater tactile sensation, the Tokyo team has modulated the temporal sequence of the force, as vibration is felt more vividly than stationary force, temporal sequence, or a waveform (burst, continuous wave, or other). Monnai stresses the importance of matching the sensation to the visual image, which is hampered by the lack of well-established guidelines for such design. The team continues to seek a way to generate a stronger force, and Monnai says they intend to extend both visual and tactile images from 2D to 3D, and then they hope to design a 3D touchable hologram that allows for both active and passive interaction.
Michael Page, an assistant professor in Toronto’s OCAD University faculty of art, currently is working on porting medical and scientific data to the new generation of holographic technology, with some of the data being multiplexed so it is interactive (touchable). His team created simulation tools for medical students, providing an auto-stereoscopic visual technology that projects 3D images without requiring the viewer to wear glasses.
“Holograms are at the very top of auto-stereoscopic volumetric viewing systems. No other medium provides a higher sense of realism,” says Page. One big challenge is creating the “self-contained viewing system for the holograms.”
Sean Gustafson, a Ph.D. researcher (who has since graduated) at the Hasso Plattner Institute in Germany, worked on novel spatial interactive technology, such as placing an imaginary iPhone on the palm of your hand. Patrick Baudisch and colleagues at Potsdam University have continued this work to explore other “imaginary interfaces” (screenless ultra-mobile interfaces), notably looking at tactile, spatial, visual, and sensed cues.
Rather than optical illusions, Baudisch and his team are now “betting on the real thing” by working toward developing personal fabrication equipment that works at interactive rates, ultimately in close to real time. Their WirePrint device, a collaboration between Hasso Plattner Institute and Cornell University, prints 3D objects as wireframe previews for fast prototyping by extruding filaments directly into 3D space instead of printing layer-wise. As Baudisch explains, “users interact by interactively manipulating the 3D shape of the device, while most of the know-how comes from the machine, supporting users in the process.”
Another approach is that of “data gloves.” As Patrik Göthe at Chalmers University in Sweden says, “The next paradigm of consumer technology is mostly about seamlessly moving the screen interface to our surroundings as holograms and projections.” His concept is for a partial glove, covering thumb and index finger, attached to the wrist. It can interact with a holographic keyboard via touch-sensitive fingertips.
Haptic feedback in general is likely to become mainstream. Apple has announced its Force Touch trackpad on the early 2015 MacBook Pro, declared to be “a tour de force of engineering” by AppleInsider magazine (http://bit.ly/1FKi2bg). You will feel clicks as small taps on your finger. Pressure-sensing APIs can also enable you to write your signature on the trackpad, with greater pressure creating broader strokes. Haptic feedback is already integrated into iMovie’s 10.0.7 update, and more apps for OS X and other iOS devices are undoubtedly in the pipeline.
Touching the Future
Touchable 3D holograms can extend the use of touch interaction to unconventional situations. Here are some hands-on applications:
Real estate—The latest developments in haptic feedback enable potential purchasers to actually touch the textures in a home when viewing a digital tour: rough stone walls, smooth marble, and so on. At the moment, these textures are only available on a sample-size scale, rather than as part of a life-size set, but the technology is still a useful sales tool.
Medical examinations—When haptic holograms are combined with computerized tomography (CT), magnetic resonance imaging (MRI), and ultrasound scans of an area of the body, surgeons will be able to “feel” a tumor, for example, in advance of a live operation. This technology is already available commercially from companies such as RealView Imaging in Israel, which has developed medical holography for interventional cardiology and diagnostic imaging.
Touchable 3D holograms can extend the use of touch interaction to unconventional situations.
Other potential applications include: museum visitors holding irreplaceable artifacts; sticky-fingered chefs checking recipes; users avoiding the germs invariably found on ATM keypads and other public touch electronic devices, and visually impaired users feeling their virtual interface. There is enormous potential for exploitation of the technology in the military, security, and education sectors, as well as in the arts.
Light Years Ahead
According to the January 2015 Holographic Display Market report by MarketsandMarkets, the touchable display market will experience a compound annual growth rate of more than 30% to reach $3.57 billion by 2020. Researchers feeling their way toward previously undreamed-of haptic hologram solutions should feel encouraged.
Further Reading
Gustafson, S. (2013).
Imaginary interfaces. Doctoral dissertation, Hasso Plattner Institute, University of Potsdam. http://bit.ly/1IL02DG
Gustafson, S., Holz, C., and Baudisch, P. (2011).
Imaginary phone: Learning imaginary interfaces by transferring spatial memory from a familiar device. In Proceedings of UIST 2011, 283-292. http://bit.ly/1yuQ8Tu
Hoshi, T., Takahashi, M., Nakatsuma, N., and Shinoda, H. (2009).
Touchable holography. Proceedings of SIGGRAPH 2009. http://bit.ly/1JKl4ju
Long, B., Seah, S.A., Carter, T., and Subramanian, S. (2014).
Rendering volumetric haptic shapes in mid-air using ultrasound. ACM Transactions on Graphics, vol. 33. http://bit.ly/1DHqlrl
Obrist, M., Seah, S.A., and Subramanian, S. (2014).
Talking about tactile experiences. Proceedings of ACM CHI 2013 Conference on Human Factors in Computing Systems (pp. 1659-1668). Paris, France. http://bit.ly/1yrnCln
Join the Discussion (0)
Become a Member or Sign In to Post a Comment