Sign In

Communications of the ACM


Control Without Controllers

View as: Print Mobile App ACM Digital Library Full Text (PDF) In the Digital Edition Share: Send by email Share on reddit Share on StumbleUpon Share on Hacker News Share on Tweeter Share on Facebook
hand grasping a doorknob and fiddling with a handheld device

Touch can sense a variety of human touches and hand postures, including grasping a doorknob and fiddling with a handheld device.

Credit: Disney Research

If "the most profound technologies are those that disappear," as Xerox PARC manager Mark Weiser wrote in 1991, then most human-computer interfaces are still relatively shallow. While the touchscreen-enabled pocket device has "disappeared" the computer in many contexts, a truly vanished device would never have to leave your pocket, or the hem of your clothes, or its spot embedded in your jewelry: Its interface would be your complex touch on the world's existing surfaces.

Scientists at Disney Research in Pittsburgh have advanced this vision with their Touché sensor project, which uses a new Swept Frequency Capacitive Sensing technique to distinguish several kinds of touch on any of five types of surface. These sensor-enabled surfaces could be used to control any kind of electronic device. Perhaps Touché's most interesting mode turns one's own body into a controller: In a demonstration video, a user controls the music player in his pocket by touching his hands and arms to play a song, change volume, and switch tracks.

The Disney paper "Touché: Enhancing Touch Interaction on Humans, Screens, Liquids, and Everyday Objects" has sparked enthusiasm among researchers and analysts, partly because it could transcend traditional computing devices. Engineer and human-computer interaction researcher Darren Leigh believes Touché's design is best suited for "interaction that's much more subtle, and perhaps much more natural" than existing solutions. "Previous touch technologies say where you were touching a device," he says. "Touché is different because it can tell how you're touching it."

But the project raises as many questions as it answers. It is unclear how much user training would be needed for acceptable results; radio interference caused by the frequency sweep could incite regulatory agencies to act; and the project's paper refers several times to the need for "future work" in various areas. Further, the market is glutted with more than a dozen touch technologies including those based on resistive, optical, and other types of capacitive phenomena.

Back to Top

Using Capacitive Profiles

Technically, Touché is unusual in that it measures capacitance in a range of frequenciesfrom 1KHz to 3.5MHz, cycled 33 times a secondrather than relying on a single frequency, as is most common. This in itself is not new: Antenna builders commonly use a network analyzer in this way to optimize signal and minimize interference. But Touché goes further by taking the resulting curve, which its researchers call a "capacitive profile," and comparing it to others using machine learning, thereby distinguishing among touches.

The results are impressive, if imperfect. Touché was able to distinguish among five types of doorknob touchone-finger, pinch, in a circle, grasping, and absentat nearly 97% accuracy. Dropping the most trouble-some one, the circle touch, raised accuracy to 99% with per-user training and 96% without. Similar results came from the other four modes studied, with users touching a table; a handheld device; their own bodies, as in the music-player example; and water in an aquarium.

Rather than try to predict the capacitive profile that would result from each touch, Touché's researchers fed their experimental data curves into a machine learning system. One of the paper's authors, Disney Research's Senior Research Scientist Ivan Poupyrev, described their approach as "brute force." "We just took all our data and fed it into a machine-learning system," he says. "We did identify some of the distinctive features, such as peaks and slopes. But we didn't try to understand the exact reasons why data looked that way or do much fine tuning." Fortunately, their research methods yielded an enormous amount of data to work with, and they have already started secondary analysis. "We didn't just take the curve, but also multiple derivatives," says Poupyrev. "Now it's time to go back and look at those curves and understand why they work this way. Then we can design specific algorithms to build something better." Such optimization could improve "walk-up" accuracy significantly.

Jonathan Westhues, a self-employed engineer and designer of capacitive systems, points to other ways of extracting information from the project's data. "It would be cool to see how much that additional info [Touché's Swept Frequency Capacitive Sensing system] buys you over single-frequency sensing," Westhues says. "You could train a classifier using just one scalar capacitance, then train a classifier using the full set of data over frequency. If you compared those, and saw that the recognizer trained with the full set of data performed better, you'd have mathematical proof that there's extra information in there. The researchers are in an excellent position to test it, possibly using their existing data."

Even without follow-up optimization and analysis, Poupyrev believes the current results could be good enough for some purposes. He points out that "the sweet spot for accuracy depends on the application. For example, when you're buying something from a vending machine, you don't want to learn how to use it. You just want the stuff, so it has to be 99.9% accurate right away. At the other extreme are home controlsyour TV or home stereo and such. People have incentive to do it right, and therefore they can quickly learn the exact gestures that would get them what they want."

Back to Top

Market Possibilities

The Touché paper is short on specific applications, focusing instead on general categories of interaction. But the payoff could be high for creating touch interface applications that can compete with existing ones. Jennifer Colegrove, vice president of emerging display technologies at DisplaySearch, says the market for touchscreens has been on the upswing since 2009 when she sized it at $4 billion. It jumped to $7 billion in 2010, then to $12 billion in 2011; Colegrove predicts it will exceed $20 billion in 2017.

Rather than try to predict the capacitive profile that would result from each touch, Touché's researchers fed their experimental data curves into a machine learning system.

Although Touché would not be appropriate for all types of applications in the touchscreen space, it could capture some parts of the market, particularly those that require instrumenting nontraditional surfaces such as medical devices and tabletops.

But substantial roadblocks stand in the way. According to Colegrove, competing touch technologies are cheaper, easier to get, and backed by known manufacturers. "The current touchscreen market is about price competition," she says, "and I don't see Touché as a threat in that market within three years. Also, the market depends on its suppliers, and Touché is still in the early stage, with only one institute doing it. Other touch technologies have 20 or 80 suppliers. Among them, Intel is trying to grab market share with projected capacitive [technology]."

Another barrier could come in the form of government regulation. "Because Touché radiates radio frequencies, anyone who tries to commercialize it will have problems with Federal Communications Commission [FCC] certification," says Leigh. "It sweeps from audio frequencies [starting at 1kHz] through the AM radio frequencies, and then into the shortwave bands [up to 3.5MHz]. The FCC controls everything over 9kHz. The AM band is already a very noisy part of the spectrum, and AM radios are incredibly sensitive devices."

But if industry can work out those issues, a new world of as-yet unimagined applications could appear. So says Brandon Taylor, an engineer at Samsung Electronics who developed a "graspable" user interface that could, for example, let players vary pitch selections in a baseball video game based on how they grasp the ball. "In my work, I asked, In what types of interactions is fine manipulation really important?" he says. "If Touché produces a rich set of data about how a user's hand is actually manipulating an object, you could start to understand some of the differences between a skilled artist in a field and a beginner. That could provide a different insight into how people learn these skills and how they're really manipulating things. But that means going from 'We can determine there are two fingers in water' to 'We can understand why someone is a virtuoso on the piano.'"

Poupyrev believes Touché could also be broadly valuable for increasing accessibility. "We don't talk about this in the paper, but users can define their own gestures," he says. "That's very interesting, because we design our world assuming that people all have five fingers and can use them. But there are a lot of people with injuries or disorders who are often left out. So we usually have to design something special for them that often could be ugly, expensive, heavy, and sets them apart from everybody else. But people who are in these situations don't want to look disabled or wear funny devicesthey want the same ones as everyone else. With our technology we might make devices easily adaptable to any special needs."

The Touché paper is short on specific applications, focusing instead on general categories of interaction, but the payoff could be high for creating touch interface applications that can compete with existing ones.

Touché is an example of research that combines existing technologiescapacitive sensing, network analysis, and machine learningrather than introducing truly new ones. Taylor again recommended that follow-on research look beyond traditional computer applications.

"The real root of this sort of research isn't in designing another way to right-click on a mobile phone," says Taylor. "Instead it's 'If we broaden our scope of thinking, what does this allow us to do that isn't already being done in another way?' Maybe these sensors could figure out what you're doing and react appropriately. Are you busy writing something? Should I route things to voicemail because you're engaged in a behavior that I can recognize as being occupied? That's where we can see a real breakthrough with these types of interactions."

Back to Top

Further Reading

Poupyrev, I., Yeo, Z., Griffin, J.D., and Hudson, S.
Sensing human activities with resonant tuning, Proceedings of the 28th International Conference on Human Factors in Computing Systems, Atlanta, GA, April 1015, 2010.

Sato, M., Poupyrev, I., and Harrison, C.
Touché: enhancing touch interaction on humans, screens, liquids, and everyday objects, Proceedings of the 2012 ACM Annual Conference on Human Factors in Computing Systems, Austin, TX, May 510, 2012.

Taylor, B.T., and Bove, V.
Graspables: Grasp-recognition as a user interface. Proceedings of the 27th International Conference on Human Factors in Computing Systems, Boston, MA, April 49, 2009.


Zimmerman, T.G., Smith, J.R., Paradiso, J.A., Allport, D., and Gershenfeld, N.
Applying electric field sensing to human-computer interfaces, Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, Denver, CO, May 711, 1995.

Back to Top


Tom Geller is an Oberlin, OH-based science, technology, and business writer.

Back to Top


UF1Figure. Touché can sense a variety of human touches and hand postures, including grasping a doorknob and fiddling with a handheld device.

Back to top

©2012 ACM  0001-0782/11/01  $15.00

Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and full citation on the first page. Copyright for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, to republish, to post on servers, or to redistribute to lists, requires prior specific permission and/or fee. Request permission to publish from or fax (212) 869-0481.

The Digital Library is published by the Association for Computing Machinery. Copyright © 2012 ACM, Inc.


No entries found