Sign In

Communications of the ACM

Research highlights

Technical Perspective: Innovative Interaction: From Concept to the Wild

View as: Print Mobile App ACM Digital Library Full Text (PDF) In the Digital Edition Share: Send by email Share on reddit Share on StumbleUpon Share on Hacker News Share on Tweeter Share on Facebook

The history of the relationship between writing systems and technology is as long as it is varied. The development of the electrical telegraph in the 1830s gave rise to Morse code, which in turn brought us what in engineering terms could be called a time-multiplexed one-button keyboard. Toward the end of that century, we saw a shift to space-multiplexed, multikey devices that evolved into today's QWERTY keyboard and touch typing.

Likewise, the challenge of entering text using portable gadgets has a long history. Arguably, the world's first mechanical mobile text entry device was the Permutation Typograph, or Pocket Printing Machinea miniature typewriter invented in the early 1850s by Benjamin Livermore of Hartland, Vermont.4 This remarkable device could do what most of today's "modern" mobiles cannot: be operated eyes-free, in one's jacket pocket, with one hand. This could be accomplished due to its six-button chord keyboard.

Hence, before even reaching the 20th century, we had examples of three different approaches to entering text by striking keys. But of course, the lowly pencil (dating back to at least the early 1500s) reminds us there is at least one other way to capture textone that involves drawing marks using appropriate gestures and tools. For many of us the first experience doing sousing a portable digital appliancecame in 1996 when we first entered text into our Palm Pilot using Graffiti. I suspect that few realized the basics of the notation we were using dated back to 63 B.C., and a single-stroke shorthand, Notae Tironianae, developed by a slave of Cicero, Marcus Tullius.2

The Palm Pilot also let us use our finger or the stylus to enter text by tapping on a graphical keyboard. Yet, it was certainly not the first consumer device that enabled the same hardwarea finger or stylus coupled with a touch-sensitive surfaceto support entering text either by "typing" on a virtual keyboard, or by recognizing what was "written" on the screen. My favorite early example, but far less well known, was the 1984 Casio PF-8000 Data Bank.3

What is both interesting and important is the Palm Pilot, and nearly all other examples, share one particular constraint: despite using the same transducers, the two modalities, tapping vs. writing, are almost always mutually exclusive. You can use one or the other on the same device, but virtually never in combination.

Understanding this constraint, and the extent to which the weight of prior art falls in accord with it, may provide one of the best aids to fully appreciating the creative insight in the following paper: that you can have your cake and eat it too, and by virtue of elegant design, you can also end up with an ever-improving balanced diet. In short, what the authors have done is recognize and refine a technique whereby one can achieve the best of both worlds, and through that combination, exceed by far the optimal performance of either tapping or writing alone.

Creative insight, however, is not the same thing as innovation. The graveyards of technology are littered with the corpses of products and companies that did not understand the difference. Execution of how one realizes a concept is key. It is in this regard that I see the importance of this paper transcending the spheres of mobile computing, text entry, HCI, or design. For me, it is an exemplary demonstration that reinforces that computer science is a science, and that today the nature of that science has a breadth that extends beyond its traditional roots. It is a nature where human performanceboth user/customer and developerare front and center.

Yes, there is a precedent for the concept of tracing as well as tapping on a touch keyboard. But as Brian Arthur argues so eloquently, there is a precedent for all new technologies.1 The importance and the creativity are reflected in the insight with which those precedents are explored, tested, and combined. And here is the elegance in this work. From the beginning, there is a core idea that stemmed from an insight that comes from who knows where. The authors draw on a range of theories and predictive models in order to get a sense of what they are working with. They reduce what would otherwise be a bewildering array of possible approaches to a manageable set of reasonably plausible ones. In so doing, we see a transition from creativity in the quasi-artistic sense to creativity in applied math. And from there, they apply their knowledge and skills in traditional computer science to implement prototypes that can be tested in studies derived from their competence and creativity in experimental psychology. And finally, from the results, they render the technology in product form, and apply business skills to bring the technology to market.

In this day and age, where the dominant change in computation over the past 40 years is most evident not in the technology per se, but who is doing what, where, when, how, with whom, why, and for how much, it seems to me this is one of the only ways (other than blind luck) that can consistently bring new, innovative products to market.

In short, this work is outstanding and worthy of attention. But as an example of a process of innovation, this is one of the best examples I have seen that demonstrates how design, cumulative science, engineering, and empirical studies can work together on both the machine and human sides of the equation. This is decidedly not just another usability study or example of a show-and-tell demo. I hope you enjoy it on all levels, as much as I have.

Back to Top


1. Arthur, B.W. The Nature of Technology: What It Is and How It Evolves. Free Press, New York, 2009.

2. Buxton, W. Piloting through the maze. interactions 12, 6 (Nov. + Dec. 2005), p. 10.


4. Legrand, J. Among the jostling crowd: Livermore's permutation typograph. ETCetera J. Early Typewriter Collector's Association 81 (Mar. 2008), 35.

Back to Top


William Buxton ( is a Principal Researcher at Microsoft Research, and a Distinguished Professor of Industrial Design at the Technical University of Eindhoven.

©2012 ACM  0001-0782/12/0900  $10.00

Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and full citation on the first page. Copyright for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, to republish, to post on servers, or to redistribute to lists, requires prior specific permission and/or fee. Request permission to publish from or fax (212) 869-0481.

The Digital Library is published by the Association for Computing Machinery. Copyright © 2012 ACM, Inc.


No entries found