As I wrote in my February 1997 Communications essay on post-WIMP user interfaces, user-computer interfaces are a necessary evil. They imply a separation between what I want the computer to do and the act of doing it. Ultimately I prefer the computing environment would simply carry out my wishes via a form of telepathy—cogito ergo fac (“I think therefore do it”). Needless to say, such a means of instructing would present unimaginably difficult problems, not just in the complete interpretation and execution of DWIM—do what I mean—but equally in preventing the analog to the Midas touch, where my thoughts are taken too literally and at times when I don’t want them acted on at all. “Be careful what you wish for,” indeed… .
Given that, for better or for worse, we’re very far from being able to perform Gibson’s jacking in, using just our brain waves, what may we wish for during the next several decades to get relief from the limitations of today’s WIMPy interfaces? We can be relatively sure that a number of ongoing trends will, if anything, accelerate during this short period, courtesy not only of Moore’s Law, but also of clever device engineering and continuing progress in algorithms. Among these are:
- Ubiquitous computing with a profusion of form factors, à la Mark Weiser and beyond, will liberate us from the tyranny of the desktop to allow computation and communication any place, any time. Laptops will be replaced by tablet computers and they, in turn, by digital paper. Walls in our offices and homes will be reactive displays. All the digital accessories cluttering our briefcases, pocketbooks, and belts will merge into much more general-purpose communication devices with knowledge of our preferences, our context, and our geographic location. Our environments will become smarter, more responsive, and more accommodating, with embedded sensors, actuators, and often invisible embedded computing in our appliances, our rooms and furniture, our vehicles, our clothing, and, via MEMS and nanotechnology, even in our bodies. Prostheses will become far more powerful, going far beyond today’s cochlear implants.
- We’ll move from human-computer interaction to human-computing environment interaction and human-human interaction mediated by this federation of devices. Computing, today largely a solitary vice, will metamorphose into computer-supported collaboration for work and play.
- Immersive virtual and augmented reality technology and applications will become routine.
- Multimodal post-WIMP interfaces will combine multiple parallel sensory channels such as speech and gesture; perceptual user interfaces will provide unobtrusive (for example, vision-based) sensing and mimic aspects of human communication, reacting to our identity, posture, gesture, gaze and even mood and intent.
Interface Implications and Research Issues
New capabilities will make it possible to come closer to meeting as-yet unsatisfied demands in user interfaces. First, they must impedance-match, indeed leverage our human sensorium far better than today’s interfaces. For example, we need to make much greater use of speech input/output, both for normal users and for those with vision problems. Second, for ease of use and efficiency, UIs must rely on much greater knowledge of our preferences, peculiarities, and abilities—one size does not fit all—so interaction styles and tools that are task-, user-, and computing device/environment-specific will evolve. Third, universal design principles need to be developed for differently-abled as well as normal users. A number of hard research problems need to be solved before we will have significant progress in UIs, some of which include:
We are not permitted to choose the frame of our destiny. But what we put into it is ours.
—Dag Hammarskjöld, U.N. Secretary General 19531961
- The component technologies are still unsatisfactory. For example, we need robust speech recognition and generation based on natural language understanding, and even on commonsense knowledge and reasoning. Particularly for virtual reality and augmented reality, huge advances are needed in device technology to make them robust, unobtrusive, and with high spatiotemporal resolution. Haptics are especially difficult but very important in our interactions with the real world.
- When hundreds, potentially thousands of devices are part of a single user’s computing environment, what is the UI to that federation? Obviously we have to transition from direct manipulation of everything in our environment, however empowering direct manipulation of a single computer may be, to a mixture of direct manipulation and appropriately robust, intelligent, and trusted agent technology. We want to be able to dispatch agents, monitor their progress, and retain control. Managing hundreds or thousands of agents, even in hierarchies, is much more difficult than managing one, for example, a single buying or selling agent.
- Direct manipulation itself must also be at a higher level, not just point-and-click. For example, when we switched to WIMP GUIs, we lost the ability to create high-level macros in command language interfaces. Now we need such parameterized abstraction back in a new form.
- There are very hard systems and knowledge representation problems in how such a federation implements a distributed model of data and knowledge gathering and dissemination amongst its members and users.
- We need seamless computing in which our current state moves with us as we move from one locale and its device(s) to another. The interface must adapt automatically to current conditions and user needs. I may start a set of tasks in the bedroom with my eye-tracked ceiling display, continue in the shower with largely audio interaction, move to the kitchen table display still largely using voice but adding gestures, move to my voice-directed nonintrusive head-up display in the car, and continue in the office wearing lightweight head-tracked stereo glasses that turn the office into an augmented reality environment in which the surfaces surrounding me become my displays.
- Finally, this symphony of technologies, devices, interaction styles and intelligent software will become a user-unfriendly nightmare if we focus only on technology development and ignore the bigger picture: enabling and empowering people to be creative, productive, and less constrained by the limitations currently imposed by the technology. We need to employ user-centered design every step of the way.
User interfaces will finally get out of their desktop metaphor, WIMP GUI rut and use more natural and efficient interaction mechanisms and styles that take better advantage of our human capabilities.
Conclusion
User interfaces will finally get out of their desktop metaphor, WIMP GUI rut and use much more natural and efficient interaction mechanisms and styles that take better advantage of our human capabilities, and indeed let us transcend them—prostheses for the mind, using our senses far more fully than is possible now. But such advances must be based on a deep understanding of the human component of the system that links humans and computational devices of all kinds.
Human-computer interaction has always gotten short shrift in the computer science community as a “soft” subject, but it is more needed than ever if we are to leverage the great advantages we may expect in raw computation, storage, bandwidth, and device technology. Computer scientists need to learn not just about human-computer interaction but also about underlying principles of perceptual and cognitive science, social science, and even the design arts to be able to make the most of human capabilities and limitations.
Join the Discussion (0)
Become a Member or Sign In to Post a Comment