Today’s computers can process information at incredible speeds, and have the flexibility to store and display data in many different forms. But when we compare the things we can do with the actual shape of these computers with the things we can do with that of other real-world tools, it seems a lot is lost. For example, a simple piece of paper, while holding graphical or written information, can be folded into most any shape, wrapped around products, or torn into bits and recycled entirely. Try doing that with your Blackberry.
The chief reason for the limitations of today’s computer shape is the rigid planar structure of its LCD screen. The requirement to fit and protect the LCD, keyboard, and electronics causes the laptop computer to be rigid too. In this article, we argue that this planar rigidity in interface design generally limits the usability of our computers, in terms of their possible affordances.
A world of design around us is filled with “blobjects” [7]: tools with curved forms. Organic shapes are employed by the likes of Karim Rashid and Frank Gehry, who aptly apply materials like rubber, plastics, and even curved sheet metal in their designs. The first property that appears missing in computer displays, then, is the ability to have any organic shape, for example, curved like the light fixtures in Figure 1. There are clear advantages to the use of such displays, such as when working with curved data sets, like 3D models or geographical information. Another missing property is deformability. While fashion designers like Yves Saint Laurent take deformability for granted, it is not at all common in human-computer interactions. Yet deformability eases many real-world tasks, like storing things, or reading this magazine article, for example. The page flip is, in fact, a wonderfully effective way of navigating documents. Its affordance and ability to open a document at a random location is not easily mirrored by a mouse click. Deformability also allows tools to adapt their functionality to different contexts: a newspaper can serve information equally well as fish. Clearly, this kind of shape-shifting flexibility is not found in your average e-book reader.
But might it one day be? New materials, such as E-Ink and Organic light-emitting diode (OLED) displays by Polymer Vision and Sony not only mimick the high contrast but also the deformability of paper, potentially making flatland interfaces a thing of the past. Interaction designers and researchers around the world are starting to work alternate, virtually analog, degrees of input. This design stream aims to develop computers that can take on any shape or form: from an aluminum can to a Lumalive jacket.
Organic Design: Natural Morphologies as Inspiration for Industrial Design
With displays in any form will come a wealth of interactive blobjects that literally shape their own functionality. E-book readers that page down upon a flick of the computing substrate. Beverage cans with browsers displaying RSS feeds and movie trailers. When pondering the design space of such future blobject computers, the anatomies and morphologies of biological organisms form an interesting source of inspiration. Natural organisms almost exclusively rely on flexible materials and non-planar shapes. For example, the leaves of plants form resilient solar panels that bend rather than break when challenged. They are not just flexible to adapt to their environment, they also grow and adjust shape to maximize solar efficiency. Computers may one day do just that. Haeckel’s Art Forms in Nature [1] was one of the first catalogues to celebrate organic morphologies. The book, which came out in 1904, was an instant hit with designers protesting modern industrialist art. Art Nouveau designer Constant Roux even used one of Haeckel’s plates on invertibrate morphologies as an inspiration for a chandelier (see Figure 2a).
Organic Architecture: Balancing Industrial with Natural Design
Haeckel’s radiolarians also inspired Art Nouveau architect René Binet’s entrance gate to the 1900 Paris World Expo. But within 10 years, the forces of industrialism would supersede Art Nouveau with their engineering aesthetics. Only in 1939 would American architect Frank Lloyd Wright challenge the Modernist perspective, coining the term Organic Architecture [3] to capture a new philosophy promoting a better balance between human and natural design. His most famous work, Fallingwater in Pennsylvania, still stands as one of the great triumphs of American 20th century design. With Fallingwater, Lloyd Wright did not intend to copy nature, like his Art Nouveau predecessors. Instead, he created a perfect harmony between the aesthetics of nature and those of modern architecture. Its concrete cantilevers, suspended over the ever-changing flows and rocky outcroppings of the waterfall over which they were built, allowed Wright to literally balance the planar geometries of modernity with the irregular flows of natural design. Having resided in Tokyo, Lloyd Wright’s aesthetic was perhaps inspired by Japanese design philosophies like wabi-sabi, which emphasize natural imperfection and impermanence over Western controlled planar perfectionism.
Tangible and Ubiquitous: Embedding Computers in the Natural World
Indeed, what might computers look like if they were designed with a little more wabi-sabi? More curved, like a piece of earthenware, more flexible, like a sheet of Japanese Washi rice paper and more delicate, like handmade knitware? Perhaps a good start would be something like PingPongPlus, one of a string of Tangible User Interfaces designed by Hiroshi Ishii’s research group at the MIT Media Lab in the mid-1990s. Ishii was one of the first scientists to design computers that were completely integrated into the user’s natural ecology. His goal was to reduce the computer’s heavy demands on limited-capacity focal attention of users, in favor of faster, high-capacity, peripheral channels of perception.
PingPongPlus was a table tennis table featuring a video projection of water, along with a school of fish. Whenever the ping-pong ball hit the table, its position would be sensed using microphones. A hit caused the water projected on the table surface to ripple on the spot, and the virtual fish to scatter. The combined use of projection on a planar surface, with natural objects tracked as input, allowed tangibles to seamlessly integrate computer elements in a physical game, without affecting its speed or physicality. Tangibles allowed for a tighter coupling between input and output, or, as Ishii more aptly put it: between bits and atoms. However, tangibles lacked the ability to display directly onto most input objects (such as the ball), as rigid displays are difficult to integrate into non-planar objects. They also lacked the ability to track multi-touch coordinates on the surface of such objects. And because their shape was not actuated, consistancy between bits and atoms could not always be maintained. As a consequence, tangible designs focused largely on the use of objects as devices for input.
Organic User Interfaces: Designing Computers in Any Way, Shape, or Form
Some key developments in computing are now changing this equation. First, advances in flexible input technologies, like Jun Rekimoto’s capacitive SmartSkin, now allow for any surface to sense two-handed, multi-finger touch. Jeff Han’s computer vision multi-touch displays changed the way designers think about the connection between input and output. Tovi Grossman took input into a different dimension altogether. His use of ShapeTape, optical fibers that sense bending, eased 3D drawing through direct representation of shape. Something is different about input today, with Nike marketing shoes with accelerometers that sense the pace for iPod beats. In Italy, Danilo De Rossi is sewing piezoelectric sensors into clothes that monitor vital functions, and keep their wearer healthy. And in 2007, Lite-On designers won the red dot award for their concept jelly input blobjects that can be molded to fit the hand (Figure 2b).
The second development is that of so-called electrophoretic ink (E-Ink for short) at MIT. E-Ink, now a Massachusetts-based company, designs displays that behave much like printed paper. E-Ink displays reflect light directly from their environment, and as such are much more energy efficient than LCDs. E-Ink allowed companies such as Philips to start pondering the design of flexible polymer display substrates. The Philips spin-off Polymer Vision has demonstrated Readius: the first smartphone with a foldable electrophoretic display. Much like a paper scroll, this display can be rolled up into the body of the phone. Last February, Sony unveiled its first full-color flexible Organic LED display, the size of a wristwatch. And at CES 2007 Philips Research introduced Lumalive, a display made of tiny LEDs that are woven into the fabric of clothing.
The most remarkable development, however, is that of shape-changing materials inspired by organic compounds found in plant and animal life. Materials such as shape memory alloys, which mimic the behavior of muscles, are used to create interactive knitware that is responsive to environmental stimuli. Smaller actuators mean computing devices can now be built that adjust their shape according to some computational outcome, or depending on interactions with users. This will eventually result in displays that are not just volumetric, but that flexibly alter their 3D shape.
Together, the three developments mentioned here allow for a new category of computers that feature displays of almost any form: curved, spherical, flexible, actuated, or arbitrary. E-Ink displays will first find widespread use in e-books, mobile appliances, and advertising. With cost coming down, a logical next step would be curved or flexible displays on products like bottles, boxes, furniture, sportswear, and toys. How will users interact with such oddly shaped displays? What will their user interfaces look like? One thing is clear: they will look very different from the ones we use today. Rather than relying on planar GUIs, they will feature more organic user interfaces.
Defining Organic User Interfaces. “An Organic User Interface is a computer interface that uses a non-planar display as a primary means of output, as well as input. When flexible, OUIs have the ability to become the data on display through deformation, either via manipulation or actuation. Their fluid physics-based graphics are shaped through multi-touch and bi-manual gestures.” OUIs aim to support a number of design goals that transcend traditional usability. Their learnability, for example, is governed by the clarity of their affordances, and by their ability to adapt these to new contexts of use. With their emphasis on flexibility and user satisfaction, OUIs inspire users to be creative rather than merely productive. OUIs also promote well-being through diversity of posture and ergonomic fit. Although OUIs should be designed for sustainable use, they need not necessarily be made out of organic materials for their interface to still be considered organic. In 1989, British architect David Pearson formulated it as follows [4]:
“Let the design:
be inspired by nature and be sustainable, healthy, conserving, and diverse;
unfold, like an organism, from the seed within;
exist in the ‘continuous present’ and ‘begin again and again’;
follow the flows and be flexible and adaptable;
satisfy social, physical, and spiritual needs;
‘grow out of the site’ and be unique;
celebrate the spirit of youth, play and surprise;
express the rhythm of music and the power of dance.”
These words inspired the following three principles for OUI design.
Input Equals Output. In a GUI, input devices are distinct and separable from output devices: a mouse is not a display. However, this separation is typically not true for interactions with physical objects. Paper documents may be stacked, moved around, and folded with a kind of physical immersion that can only be dreamed of in GUI windows. This immersion can only be achieved through a complete synergy between multi-finger, two-handed manipulations and corresponding visual, haptic, and auditory representations. This means displays should sense their own shape as input, as well as all other forces acting upon them. In an OUI, such input is not distinguishable from its graphic output: users literally touch and deform the graphics on display. As such, OUI displays render as a real-world object, into the shape that best supports data interpretation. Bits and pieces of such behavior are already found in Apple’s rigid planar iPhone display: its multi-touch screen, for example, senses the deceleration of the finger, providing an impulse to the physics engine to scroll the menu as if it was flicked.
Function Equals Form. According to Lloyd Wright’s mentor, Sullivan, the shape of a building or object should be predicated on its intended function, not its precedent: “form follows function.” Bauhaus popularized his credo by abolishing embellishments in everyday design. Lloyd Wright thought this was a mistake: according to him, form should not follow function: “they should be one, joined in a spiritual union.” This notion resonates with Gibson’s ecological approach to visual perception, embodied in the concept of affordance: the qualities of an object that allows a person to perceive what action to take. That is, the form of an object determines what we can do with it. This describes well what Organic User Interfaces excel at: the physical representation of activities. Picking up a display activates it for input. Rotating it changes view from landscape to portrait. Likewise, bending the top right corner of the display inward may invoke a paging down action, while bending it outward pages up. Bending both sides inward causes content to zoom in, while bending outward zooms out. An example of such behaviors is found in Parkes’ Senspectra, a molecular modeling toolkit (see Figure 3). When a user bends the molecular model, LEDs embedded in the nodes glow according to the amount of strain exerted upon the optical fibers that connect them.
Form Follows Flow. Today, more than ever, meaning is appropriated by context. Similarly, OUIs adapt their form to better suit different contexts of use. Their shape fluidly follows the flow of user activities in a manifold of physical and social contexts of reuse, as well. A simple example is found in the use of folding in clamshell cellphones. Opening the clamshell activates the phone, a very strong affordance indeed. Closing it ends its functionality, deactivating the keys, protecting the display and reducing its footprint, all in one movement. A more profound example is found in the Readius: its display folds out when needed, thus doubling the available screen real estate when the activity such requires. Like clothing, forms should always suit the activity. Clothing fits the body while closely following its movements, and can even be deformed to serve other functions, like holding objects, if necessary. Thus, if the activity changes, so should the form. This kind of adaptation is best exemplified by actuated OUIs like Lumen [6], a display that changes shape in 3D, or by SensOrg, an electronic musical instrument with a flexible arrangement of inputs that are molded to varying physical or creative demands [10]. The Moldable Mouse in Figure 2b shows how malleability also reduces the risk of painful repetitive strain injuries. Its body—made of non-toxic polyurethane-coated modeling clay with stick-on buttons—allows input to literally follow the shape of the hands.
Early Examples of Organic User Interfaces
One of the first systems to exhibit OUI properties was the Illuminating Clay project [5]. Bridging the gap between TUI and OUI, it was the first interactive display made entirely out of clay. With their hands, users could deform the clay model, the topography of which was tracked by an overhead laser scanner. The scan served as input to a 3D landscape visualization, which was then projected back onto the clay surface. Illuminating Clay illustrates the blurring between input and output device: Users could, for example, alter the flow of a river by molding valleys into the clay. Here, function is also triggered by form, which literally follows the flow of interactions.
Gummi: Flexing Plastic Computers. Another early envisionment of an OUI was Gummi [8]—a pressure-sensitive PDA that simulated a flexible credit card displaying an interactive subway map. Bending the display would cause this subway map to zoom in or out, while a touch panel on the back would allow users to scroll. Again, function equals form: the shape of the display affords zooming, and the interplay between haptics and visuals reinforces this functionality. (For more information, see Carsten Schwesig’s article in this section).
Paper Computers. Books and paper also form powerful sources of inspiration for flexible computer design. Paper is particularly versatile as a medium of information. According to Sellen and Harper [9], users still prefer paper over current computer displays because it makes navigation more flexible. Paper input is direct, two-handed, and provides a rich synergetic set of haptic and visual cues. Paper supports easier transitions between activities by allowing users to pick up and organize multiple documents two-handedly. Paper is also extremely malleable: it can be folded—a primary source of input when constructing models—or bent, most often applied in navigation. Paper can be randomly arranged, or in stacks, and can even contain other objects. With the development of flexible E-Ink displays we can imagine that in the future our computers will be indistinguishable from a sheet of paper. One of the questions we have been trying to answer is how will we interact with such flexible computers?
We experimented with the use of Foldable Input Devices (FIDs) for GUI manipulations by tracking the shape of several cardboard sheets that featured retroreflective markings (see Figure 4). Behaving like real paper documents, 3D graphics windows follow the shape of associated FIDs. When FIDs are stacked, so are the window sheets in the GUI (Figure 4a). Stacks of window sheets are sorted with a shake and browsed by leafing action (Figure 4b). Using a special FID, window sheets can even be folded into threre-dimensional models, further blurring the distinction between a window sheet and its content (Figure 4c).
Inspired by Wellner’s DigitalDesk [11], Paperwindows [2] was the first computer made entirely out of three-dimensional sheets of paper (see Figure 5). It simulated a flexible, high-resolution, full-color, and wireless E-Ink display of the future.
Paperwindows are regular sheets of paper, augmented with eight retroreflective markers. These markers allow a Vicon to capture the motion and interactive shape of the paper, which is modeled as a non-uniform rational B-spline (NURBS) surface textured with the real-time content of an application window. When projected back onto the paper, the 3D models correct any projection skew caused by paper folds, giving the illusion the paper is, in fact, an interactive print. We experimented with a Web browser of which most functions were accessible through changing the shape of the paper—the primary display of the computer. Bending the sheet around its horizontal axis would cause the Web browser to page down or up. Bending the document back around its vertical axis would cause the Web browser to go back or forward in its browsing history. Fingers were also tracked: a link was clicked by touching it. A paper window was activated by picking it up. Information could be copied from one document to the next by rubbing two windows onto each other. Documents could be enlarged through collation, and sorted by stacking. Such physical interaction techniques remove any distinction between input and output: in paper windows the shape, location, and orientation of the display is the primary form of input.
From Interactive Blobjects to Curved Computer Interactions. In the Interactive Blobjects project, we are today exploring opportunities afforded by marrying everyday objects with oddly shaped displays through tracking and projection. For example, Dynacan, the dynamic beverage can shown in Figure 6a, is an early prototype of a fully recyclable curved computer. Its display features Flash animations, videos, and RSS feeds. Future versions will be made of flexible full-color E-Ink, powered by a processor and battery pack inside the can. Users can scroll by rotating the can, which is sensed by a set of accelerometers. Electronic components can be made detachable prior to recycling the can. Dynacan is part of a larger workbench investigating OUI design. Figure 6b shows how any piece of cardboard, curved or a cube, can simulate a computer interface. By selecting dials, menus, and interactive skins from a palette of interaction styles (shown in the background) a simple cardboard box is turned into a fully functional iPod. Press a finger on the palette and the iPod becomes a fully functional iPhone instead. More complex blobjects are also possible, like architectural cardboard models with live animated textures, or interactive spherical displays, like the Google Earth browser in Figure 6c.
The examples described here only scratch the surface in terms of ways in which we might interact with future computers of any shape or form.
Conclusion
The examples described here only scratch the surface in terms of ways in which we might interact with future computers of any shape or form. Possibilities include computers with displays that are curved, flexible, and that may even change their own shape in order to better fit the data, or user for that matter. In Organic User Interface design, these computers will no longer be conceived of as distinguishable from the world in which they live. All physics acting upon displays, including their shape, will be used to manipulate information. Functions will be triggered through form changes that follow the flow of the ever-changing world of the user. In a world where multi-tasking is increasingly common, the chief purpose of an OUI is to interweave a plurality of highly contextualized, interspersed activities across a variety of disconnected contexts. One challenge will be for it to do so in a manner that carries consistency across activities and contexts. Rather than a single OUI acting as an advanced Swiss army knife, users need to utilize the OUI that comes in the form most appropriate for a particular activity. Flexibility should not be misinterpreted as “one OUI fits all”: it is exactly in celebrating the diversity of display shapes that a wealth of OUI designs will find their purpose.
In the not-so-distant future, curved, full-color, flexible LEDs, OLEDs, or E-Ink displays will appear in our homes, furniture, e-books, jewelry, and clothing. When tired of the color of your suit, the pattern of your wallpaper, or the interface on your cellphone, you simply download a new one from an online store. Some hardware interfaces may one day be monitized like software entirely, substituting the wasteful trend of buying new atoms with that of more eco-conscious bits. That would be a final frontier in the design of computer interfaces that turns the natural world into software, and software into the natural world.
Figures
Figure 1. FEEK pebble lights by Karim Rashid [
Figure 2. (a) Chandelier with jelly morphology (C. Roux, circa 1907); (b) Concept moldable mouse with jelly anatomy (Lite-On, 2007).
Figure 3. Exploring molecular strain in Senspectra.
Figure 4. Some possible interactive shapes for paper computers: (a) Organizing window sheets in a stack; (b) Leafing through window sheets; (c) Folding window sheets into three-dimensional content.
Figure 5. Paperwindows: A prototype paper computer.
Figure 6. Interactive Blobjects: (a) Dynacan computer with flash animation; (b) iPod form factor on an interactive cardboard design bench; (c) Zoom gesture on spherical screen with Google Earth.
Join the Discussion (0)
Become a Member or Sign In to Post a Comment