Digital Memories in an Era of Ubiquitous Computing and Abundant Storage

Gordon returns home from a business trip, and the photos taken automatically by his hat-mounted camera (see Figure 1a) begin to appear on the screensaver built into the screen on his refrigerator (see Figure 1c). One of a lunch with colleagues reminds him of an email message he wants to re-read, remembering he last opened it during a meeting after the lunch. He uses his tablet PC to display the list of photos on the refrigerator and looks up the lunch appointment associated with the photo. Seeing the meeting, he requests all email accessed during that time.

Later, when sharing the photos with his friends, he wants to find one showing the Sydney Opera House, recalling that he saw it on a hot afternoon. He searches for photos taken when his personal sensor read higher than 80^o Fahrenheit. There are 500 such photos, so he switches to a map view to remind himself where they were taken. Selecting the appropriate neighborhood in Sydney, he locates a well-composed photo of the opera house. He browses through more photos, labels a number of them “share with friends” and the events from the trip in his calendar “public,” and an attractive photo story is automatically created on his blog, with access limited to friends he previously identified and listed in his blog’s security filter.

The next morning, Gordon’s body sensor (see Figure 1b) alerts him he has a fever of 101.3^o Fahrenheit. His trend-analysis software notes that he has been coming down with colds after business trips lately, so he forwards the analysis to his doctor, requesting advice on how to avoid a recurrence. He feels well enough to go to work, however, but now can’t find his hat. The last time he recalls wearing it was when he last did his laundry. He accesses the log of his clothes dryer’s embedded server to report the last time it was used. Scanning photos taken immediately after he took the clothes out of the dryer by wall-mounted cameras, Gordon notices one of himself tossing the hat onto his bedside table. A look behind the table reveals the missing hat.

While this vision is not yet fully realized, it is becoming possible as a consequence of making everyday objects computationally enhanced and networked. Embedded processors and network connectivity are being added to refrigerators, pens, meeting rooms, and living rooms. Audio/visual capture is becoming wearable, as is biometric sensing for health applications. Today’s low-cost abundant storage makes it possible to record most life experiences involving audio, video, and other types of data. Future networking promises to allow us to view and manage personal information from any device, any place, at any time. Incorporating compact, inexpensive, self-activating sensors, these technologies promise to let us capture most of our lives easily and passively, so we no longer have to interrupt our interactions with one another to shoot movies or take photos.

Here, we discuss what we might do with a life’s worth of digital memories and the applications that might prove useful. We also consider reasons why we might not want to keep everything, outline research challenges, and identify the leverage that having a complete life record could bring to personal information management problems. The first questions someone might ask about a completely digitized life are: Why bother?, and What might I do with all the stuff I collect? Some answers include:

• Memory. Finding things (such as keys and eyeglasses); replaying learning and teaching experiences; reviewing research and travel; remembering names of people and places; and reviewing discussions and meetings;
• Share personal experience. Reliving experiences with lost or distant loved ones; improving communication between grandparents and grandchildren; and sharing everyday events with people separated by distance;
• Personal reflection and analysis. Understanding personal development; reviewing conflicts; finding situational patterns correlated to emotional states; and improving health via medical monitoring;
• Time management. Improving productivity at and away from the workplace; improving coordination among family, friends, and co-workers; and identifying relevant or proximate information, given the current context (including but not limited to location); and
• Security. Using information for legal purposes (such as to resolve arguments and prove alibis); for security purposes (such as personal video recordings that might include evidence of, say, a possible terrorist in a public location).

Figure 2 outlines digital-memory applications in terms of who controls and who uses a person’s digital memories. The applications change over the course of one’s life, as does the person using the application. Parents and caregivers use and control their children’s information; for example, they may need access to a child’s dietary or homework information; parents have always felt responsible for recording and keeping artifacts their children might want later in life, including photos, memorable schoolwork, artwork, performances of every kind, and official documents. One’s adult life will involve personal control and use. In old age, caregivers will again take the lead; for example, a gerontologist might access a patient’s medical history, and adult children acting as caregivers will access a parent’s calendar to schedule medical appointments. After the person’s death, a will’s executor might need access to personal financial information, and younger family members may want to learn about their roots through a lifetime of collected digital memories. The need to change access and control is illustrated by a news story last year about a U.S. Marine killed in Iraq, whose Internet service provider—Yahoo—refused to grant his family access to his email in order to fulfill its privacy promise to the Marine.

It’s fascinating to speculate about the patterns that can be mined from digital memories. Poor health might be readily correlated with certain locations or activities. The onset of a treatable medical condition may be detected by sensor data and brought to a person’s attention much sooner than it would be otherwise.

An argument can also be made against retaining records of absolutely everything in one’s life. In some instances we may not want a complete and objective memory of the past. For one thing, it may be too painful; we might not want a clear memory of physical or emotional trauma. Sequences of actions are often negotiated, not literally determined, in dispute resolution. Records are also discarded for legal reasons; for example, we may not want our digital memories subpoenaed and possibly used to establish a fact in a way real memories can’t, thus allowing us to retain plausible deniability. Moreover, there are privacy concerns: for example, the more one stores, the more likely it is that personal information could be compromised.

Admittedly, this is only a quantitative, not qualitative, difference from the privacy concerns we have with our PCs today, which already contain sensitive information. Nonetheless, a truly comprehensive record of one’s life might make a scary situation scarier. Digital memories must systematically examine the legal and social issues evoked by worst-case hypothetical scenarios and devise legal, technological, and social solutions to avoid societal creep toward a privacy crisis.

A personal strategy of recording and keeping everything raises concern that clutter may obscure valuable things and add to the burden of personal information management. Keeping everything may have negative side effects (such as distracted attention, information overload, and less-effective searching and browsing) [6]. However, there is a difference between keeping everything and making everything visible; a technological solution can hide details and deletions, thus eliminating clutter and the oppressive task of managing it while still retaining the records for future use. Although it seems tempting to label many artifacts of everyday life as disposable, in practice it is often impossible to predict exactly which items in our lives we might value or need in the future. This phenomenon is called post-value recall; the true recall value of information is not completely known until after the information is archived. Accordingly, some of us today have filing cabinets full of paper, not because we believe we will use every document again, but because we cannot predict which papers we may actually need later.

The recorded information also serves as a possible anchor for future events. It can be indexed, marked, and annotated for prospective remembering, that is, remembering events that will happen in the future. Attending a wedding, for example, may help us remember the couple’s wedding anniversary.

Fortunately, increasing the number and types of data we keep is an advantage, as well as a challenge. With scale comes more opportunities to correlate, most likely based on time or place, but possibly on any common attribute, and such a correlation can be leveraged to help us find things that are only incidentally related, connected to each other by the common attribute; records of everyday information may help us tell meaningful stories about our lives by culling minor but evocative details; for example, having a personal location record may allow us to find a document according to where we were when we last edited it (“I worked on this while visiting Seattle”), and photos can be connected to calendar events with the same time value to turn a calendar into a photo diary.

Back to Top

Research Challenges

Recording, creating, receiving, storing, and accumulating digital materials is easy, but managing and using them sensibly is difficult, especially as time passes and their immediacy fades. These challenges are evident from the five scenarios posed so far:

• Identifying valuable or important portions of our personal records, especially over time and in context;
• Interpreting and correlating data from multiple sources to make sense of what we have;
• Viewing and interacting with records that are an accumulation rather than a meaningful collection;
• Keeping personal information safe without impeding our ability to access or preserve it; and
• Sharing this information in a way that makes sense in the face of changing needs, without creating so much overhead that we either share everything or share nothing at all.

The primary research challenge in digital memories is how to cope with the vast quantity of material. Summarization, abstraction, and data mining approaches will help identify important items, though what is important to one person may not be important to another, what is important in one context (such as at work) may not be important in another (such as at home), and what is important today may not be in the future (and vice versa). Multiple levels of detail and resolution are desirable for reviewing all captured media, especially sensor data.

Making use of the increasing number and types of data sources (primarily from ubiquitous sensors) poses another challenge. The information must be abstracted and displayed in useful, insightful, and attractive visualizations if we are to be able to interpret and use it properly. For example, GPS data is easier to interpret when presented as labels on a map than when presented as a list of latitude–longitude pairs.

Viewing and interacting directly with our digital memories—even the pleasurable ones—is also a significant research challenge. As anyone who has struggled through friends’ and family members’ personal vacation videos knows, many recorded memories are gratefully skipped. Imagine if all of your vacations were recorded. How would you find that key event you were looking for or view it without a specific aim in mind? The answer may lie in the interconnection of the various types of data, in which one data source functions as an index to other data sources. How the user interface to such an interconnected browsing mechanism works and how to make it accessible to the population in general, rather than just to techies, is especially challenging.

If the question for an individual user is whether or not to store digital records, security is not an issue. We can lock up a hard drive in exactly the same way we lock up personal papers and photos to keep them safe and secure. Moreover, it is easier to replicate digital media and digital records and to store copies at other locations to ensure the information survives physical disaster. However, convenience often trumps security concerns, and ease of access makes it desirable to connect our store of personal digital memories to the Internet. Instead of relying on the simple protection of one locked door in one place to keep one’s data safe, we may have inadvertently created digital doors in millions of virtual neighborhoods for every thief in the world to try to break into.

Even presuming that technological measures do in fact keep private information safe, simply specifying who should have access to the information may become a burden to the user. The choice between private and public could become onerous, given the volume of data and the diversity of data sources. But a solution must anticipate that people have nuanced ideas upon which they base their designations of with whom and under what conditions information is shared.

Back to Top

Digital Memories Research

In his now-iconic article “As We May Think” in July 1945, Vannevar Bush, director of the U.S. Office of Scientific R&D during World War II, popularized a vision of a personal storage system that included self-initiated recordings from a walnut-size head-mounted camera and voice recorder [1]. His storage ideas were realized as digital technology in the 1960s by Douglas Engelbart, whose hypermedia groupware system supported bookmarks, hyperlinks, recording of email, and a journal [3]; Hypertext visionary Ted Nelson advocated keeping personal recordings of everything and suggested novel computational infrastructure [9].

Later, Bush’s camera was implemented and extended by wearable computing researchers, including Steve Mann, a professor at the University of Toronto, who have also considered the related social, artistic, and legal issues [3]. Brian Clarkson, then a graduate student at MIT, performed an experiment of continuous recording for 100 days with fisheye video cameras mounted on his front and back [2]; Microsoft Research’s SenseCam wearable camera has built-in sensors—light, infrared, temperature—in order to take pictures at suitable times [4]. Ricoh Innovation’s Jimminy—with heads-up display, one-handed keyboard, and location awareness—runs note-taking software that selects old notes to display on screen based on current location, the people nearby, and the text of the notes being written [10]. The University of Tokyo has a system that continuously captures video, along with sensors that include GPS, gyroscope, accelerometer, and a brain-wave sensor [5]. Wearable A/V capture systems have even been augmented through interaction with a robot in an exhibition setting [11].

---

There is a difference between keeping everything and making everything visible.

---

In the early 1990s, Rank Xerox EuroPARC pursued digital-memories-related projects (such as the use of Active Badges and automatic photo-taking to generate user diaries) [7]. Storage for digital memories has been explored by groups like MIT’s Haystack (haystack.lcs.mit.edu/), Microsoft’s Stuff I’ve Seen (research.microsoft.com/adapt/sis/), and MyLifeBits (see the article by Gemmell et al. in this issue).

For the past two years, ACM has sponsored the Continuous Archival and Retrieval of Personal Experience (CARPE) workshop, and IEEE Multimedia has announced it will publish a special section later this year on CARPE. The Pervasive 2004 conference included a workshop on memory and the sharing of experiences. The U.K.’s Engineering and Physical Sciences Research Council has designated “Memories for Life” a grand challenge, and in the U.S. DARPA’s Advanced Soldier Sensor Information System and Technology program funds research to “exploit soldier-worn sensors to augment the soldier’s recall and reporting capability.” It is clear that the growing availability of low-cost storage, coupled with improved technology for recording multimedia data and the ubiquitous use of sensors, has stirred researcher (and public) interest.

Back to Top

Conclusion

While this exploration may seem futuristic, many of the required components are commercially available today with more on the way. Commercial systems (such as Deja View’s Camwear, www. mydejaview. com) that support information capture include head-worn video capture and the continuous monitoring and recording of physiological information (such as through devices by BodyMedia, www.bodymedia.com). Meanwhile, plentiful storage encourages everyone to keep more and more of their memories in digital form. Genres and interactions that were formerly transient have become relatively permanent simply because keeping them is not only possible but easy and cheap.

However, difficult technological, legal, and social issues must also be solved to make lifetime recording valuable. Much thoughtful research remains to be done in nearly every related area, including privacy, security, user interfaces, sharing content analysis, data mining, and summarization. We look forward to broad and active research into digital memories.

Back to Top

Back to Top

Back to Top

Back to top