Technical Perspective: Innovative Interaction: From Concept to the Wild

The history of the relationship between writing systems and technology is as long as it is varied. The development of the electrical telegraph in the 1830s gave rise to Morse code, which in turn brought us what in engineering terms could be called a time-multiplexed one-button keyboard. Toward the end of that century, we saw a shift to space-multiplexed, multikey devices that evolved into today’s QWERTY keyboard and touch typing.

Likewise, the challenge of entering text using portable gadgets has a long history. Arguably, the world’s first mechanical mobile text entry device was the Permutation Typograph, or Pocket Printing Machine—a miniature typewriter invented in the early 1850s by Benjamin Livermore of Hartland, Vermont.⁴ This remarkable device could do what most of today’s "modern" mobiles cannot: be operated eyes-free, in one’s jacket pocket, with one hand. This could be accomplished due to its six-button chord keyboard.

Hence, before even reaching the 20^th century, we had examples of three different approaches to entering text by striking keys. But of course, the lowly pencil (dating back to at least the early 1500s) reminds us there is at least one other way to capture text—one that involves drawing marks using appropriate gestures and tools. For many of us the first experience doing so—using a portable digital appliance—came in 1996 when we first entered text into our Palm Pilot using Graffiti. I suspect that few realized the basics of the notation we were using dated back to 63 B.C., and a single-stroke shorthand, Notae Tironianae, developed by a slave of Cicero, Marcus Tullius.²

The Palm Pilot also let us use our finger or the stylus to enter text by tapping on a graphical keyboard. Yet, it was certainly not the first consumer device that enabled the same hardware—a finger or stylus coupled with a touch-sensitive surface—to support entering text either by "typing" on a virtual keyboard, or by recognizing what was "written" on the screen. My favorite early example, but far less well known, was the 1984 Casio PF-8000 Data Bank.³

What is both interesting and important is the Palm Pilot, and nearly all other examples, share one particular constraint: despite using the same transducers, the two modalities, tapping vs. writing, are almost always mutually exclusive. You can use one or the other on the same device, but virtually never in combination.

Understanding this constraint, and the extent to which the weight of prior art falls in accord with it, may provide one of the best aids to fully appreciating the creative insight in the following paper: that you can have your cake and eat it too, and by virtue of elegant design, you can also end up with an ever-improving balanced diet. In short, what the authors have done is recognize and refine a technique whereby one can achieve the best of both worlds, and through that combination, exceed by far the optimal performance of either tapping or writing alone.

Creative insight, however, is not the same thing as innovation. The graveyards of technology are littered with the corpses of products and companies that did not understand the difference. Execution of how one realizes a concept is key. It is in this regard that I see the importance of this paper transcending the spheres of mobile computing, text entry, HCI, or design. For me, it is an exemplary demonstration that reinforces that computer science is a science, and that today the nature of that science has a breadth that extends beyond its traditional roots. It is a nature where human performance—both user/customer and developer—are front and center.

Yes, there is a precedent for the concept of tracing as well as tapping on a touch keyboard. But as Brian Arthur argues so eloquently, there is a precedent for all new technologies.¹ The importance and the creativity are reflected in the insight with which those precedents are explored, tested, and combined. And here is the elegance in this work. From the beginning, there is a core idea that stemmed from an insight that comes from who knows where. The authors draw on a range of theories and predictive models in order to get a sense of what they are working with. They reduce what would otherwise be a bewildering array of possible approaches to a manageable set of reasonably plausible ones. In so doing, we see a transition from creativity in the quasi-artistic sense to creativity in applied math. And from there, they apply their knowledge and skills in traditional computer science to implement prototypes that can be tested in studies derived from their competence and creativity in experimental psychology. And finally, from the results, they render the technology in product form, and apply business skills to bring the technology to market.

In this day and age, where the dominant change in computation over the past 40 years is most evident not in the technology per se, but who is doing what, where, when, how, with whom, why, and for how much, it seems to me this is one of the only ways (other than blind luck) that can consistently bring new, innovative products to market.

In short, this work is outstanding and worthy of attention. But as an example of a process of innovation, this is one of the best examples I have seen that demonstrates how design, cumulative science, engineering, and empirical studies can work together on both the machine and human sides of the equation. This is decidedly not just another usability study or example of a show-and-tell demo. I hope you enjoy it on all levels, as much as I have.

Back to Top

Back to Top