Preserving the Past with Immersive Technologies

Inside the U.S. Holocaust Memorial Museum in Washington, D.C., is an exhibit called the Tower of Faces (https://bit.ly/3cxI3Ik), which uses augmented reality to tell stories behind some of the 1,041 photos of people from the small town of Eishishok, in what is now Lithuania. The tower soars 50 feet high across 30 rows, displaying the faces of the town’s inhabitants, nearly 4,000 of whom were massacred when the Germans invaded during World War II.

When visitors walk into the tower, they can pick up one of several iPads and hold it up to an image on the wall, which will then play a video that transports them into the town. The video first appears in color and then fades to black and white, while a narrator reads a brief script about the person. One tells the story of Szeina, an actress who was fluent in five languages and owned a hotel on the market square the Nazis took over to use as their local headquarters.

“It’s a beautiful experience … there are many, many photos looking out at you—people riding bikes, outside in the snow, at a wedding banquet, and on the stairs of their houses,” says Sarah Lumbard, director of museum experience and digital media at the Holocaust Museum. The photos depict people of all ages prior to the massacre in September 1941.

Many of the photos were supplied by Yaffa Sonenson Eliach, whose grandmother was a photographer in Eishishok and is herself a survivor. The immersive experience opened in April 2022 and the idea was to “create a spark of life” so the residents of Eishishok will be remembered, Lumbard says.

“Our question was: How do we bring them to life, just for a moment, and have it not just be a memorial but have victims of the Holocaust come to life and treat them with respect and engage visitors to really see them as people,” Lumbard says.

Figure. The interactive Heroes and Legends attraction at NASA’s Kennedy Space Center Visitor Complex in Florida, featuring the U.S. Astronaut Hall of Fame.

Digital technologies such as virtual reality (VR), augmented reality (AR), and three-dimensional (3D) graphics are making it possible for museums and other institutions to preserve historical events and tell the stories of those events in an engaging way. In the case of VR, the technology actually takes them to another time or place away from where they physically are.

As research firm Gartner says, “the future of digital experience is immersive” (see https://gtnr.it/3BqmqU2).

“In this way, we can experience historical events and places that are long gone in an immersive way—kind of like IMAX taken to the next level,” explains Tuong Nguyen, a senior principal analyst at Gartner. “So instead of just seeing it on a flat screen, like TV or movies, or seeing it all around you like an IMAX movie, VR can potentially enable people to explore that space with 360 degrees of freedom.”

AR can change how someone experiences the world in front of them or around them, usually in a visual way, integrating information such as text, graphics, and audio with real-world objects. “The idea is to show users how things looked in the past with actual video and photos from these time periods,” Nguyen says.

For the Smithsonian’s National Museum of Natural History, also in Washington D.C., the impetus behind developing a mobile app called “Skin and Bones” (https://s.si.edu/3wGy0aT) using 3D augmented reality and 3D tracking was to attract more visitors to its Bone Hall by telling stories about some of the specimens on display.

Visits and dwell time in the hall and the experience people were having in the Bone Hall “fell far short of any measure of what a visitor experience should be in a modern-day exhibition,” says Robert Costello, national outreach program manager, who developed the mobile experience. In fact, most visitors were using the Bone Hall as a passageway from one section of the museum to another, rather than a destination, Costello says. The average time spent in a modern exhibition at the museum is between 10 and 20 minutes, and most of those visitors were spending less than two minutes in the hall, which has a storied history, he says.

The Bone Hall opened in 1881 as one of the first exhibitions at the Smithsonian, displaying skeletons of fish, reptiles, amphibians, birds, and mammals. Costello’s challenge was to create a new experience for visitors without changing anything about the exhibit. “That seems like a pretty impossible task, but through technology, it’s quite doable,” he says. “With augmented reality in particular, you can superimpose the graphics onto the skeletons to extract the stories you think are of relevance to the public,” he says. “And that stays true to the exhibit.”

One story that comes to life in the app is that of a blue catfish skeleton and cast, which were part of the museum’s original exhibit. Costello tracked down the skeleton’s history, went looking for the cast of the fish from the 1881 exhibit, and found it in storage in the museum’s attic. He had the cast optically scanned, and that became the AR model officials super-imposed on the skeleton.

“I thought it was an important historical story and as far as I could tell, the skeleton and cast had not been together since the 1880s,” Costello says. The fish is an invasive species and is still around. Once the free Skin and Bones AR app is downloaded, a user in the hall can direct their device’s camera at the skeleton, triggering 3D graphics to see how the live fish moved, and learn its history.

The idea was to “have an experience in a hall that was underperforming that would wow you and surprise you and entertain you, so you’d walk away with memories you wouldn’t otherwise have,” he says.

It took approximately one year to develop the content for the app, which also includes an interactive activity that challenges a user to identify bats from their calls, and objects from a bat’s location. The app was released in 2015 and has been downloaded “tens of thousands” of times, Costello says.

The idea was to “have an experience … that would wow you and surprise you and entertain you, so you’d walk away with memories you wouldn’t otherwise have.”

The Kennedy Space Center developed an interactive experience called “Heroes and Legends” (https://bit.ly/3wItpF0) to educate visitors about the early years of the U.S. space program, “while exploring the concept of heroism and the qualities that define the pioneers who inspired a generation,” says Rebecca Burgman, senior manager of public relations/communications for the center.

The experience requires the use of a number of technologies, such as holograms and AR. There is also a “Through the Eyes of a Hero” custom-built theater that features a multi-sensory 4D experience during which “guests vicariously join NASA’s heroes and legends on the most perilous stages of their missions,” says Burgman. Unlike 3D presentations that engage the visual and auditory senses, 4D incorporates additional stimuli to address other senses such as smell, touch, and motion.

A display of the Gemini 9 space capsule offers a realistic hologram of astronaut Gene Cernan climbing out of the tiny spacecraft, Burgman says. Visitors also can download an app called “Edge of Home,” where they can see the International Space Station in the way an astronaut would see it. “They will also be able to experience an extravehicular activity walk around the outside of the ISS while learning facts and figures about each module,” she says.

The app KSC 360 expedition supplies facts about each rocket in the Rocket Garden (https://bit.ly/3cSzlow), where visitors can view authentic rockets that have never flown in space, gaze at Space Shuttle Atlantis as it floats in orbit, take a ride on the Moon in a lunar rover at the Apollo/Saturn V Center, and sit in the commander’s seat of the Mercury-Atlas 8, Gemini 9, and Apollo 14 space capsules, according to Burgman.

“Users will see every switch, gauge, and lever as only astronauts Wally Schirra, Gene Cernan, and Alan Shephard have seen before,” she says.

Schools and enrichment programs also can take advantage of immersive technologies with VR headsets, so students can experience history as if they were there. Claudia White, founder and executive director of the nonprofit Youth Global Perspectives (YGP), brought in a platform from TechRow (www.techrow.org) that provides 3D content on VR headsets, to help disadvantaged youths experience historical events—including from their own history—about which they may not know much, or anything at all. The students use the headsets at YGP’s headquarters and other community centers in Indianapolis and South Bend, IN.

“Our big focus is to teach students history and culture mainly through the lens of people of color, because our students are mainly African-American,” White says. A former social studies teacher, White says she saw the same issue with her students that she herself experienced: “not being able to identify with their culture and making derogatory comments about themselves. They’re not learning positive material … and they are starting to believe the narrative that Black people are dumb, violent, and poor,” she says. “We want them to see themselves in a positive way.”

Feedback about the Tower of Faces has been focused on how the exhibit honors the community of Eishishok and brings its inhabitants back to life.

One lesser-known period in Black American history is the Tulsa Race Massacre of May 31, 1921. Students wearing headsets can watch the the actual event.

“TechRow uses the technology to tell the stories that are not being portrayed in schoolbooks the way they should,” White says. “They go out of their way to include information that almost feels like it’s being withheld from students. It gets people included in something bigger than themselves.”

The question of how effective immersive technologies are at preserving historical events and telling those stories comes down to the developer of the content, says Costello.

“There’s a lot of technologies that preserve historical artifacts or information, but it’s really the intent of the developer that matters because it’s the developer, the person doing the preservation work, who creates the content for communicating about that object or event,” he says.

Technology is merely the vehicle for doing so, albeit while serving an important purpose. Costello says “3D is a great insurance policy” that can be used to preserve museum specimens and artifacts. He cites, as an example, the National Museum of Brazil in Rio de Janeiro, which was heavily damaged by fire in 2018. “That’s a huge cultural loss,” Costello says. “Had the specimens and artifacts been preserved digitally through 3D technology, the legacy of those collections could have lived on.”

The Holocaust Museum’s Tower of Faces exhibit opened to the public in April 2022. The content took 18 months of research to prepare, to ensure “we knew who was in the photo and their fate,” Lumbard says.

The use of AR creates “an experience that’s easy to use that will pull you into the history and lives of these people living in a lovely town and, over the course of less than a week, almost all of them were massacred … not in [concentration] camps and in some cases, by people they know,” she says.

Feedback about the Tower of Faces has been focused on how the exhibit honors the community of Eishishok and brings its inhabitants back to life, Lumbard says. “What better way to serve the memory of the victims than to have people see them as people murdered by Nazis; not just numbers, but people with dreams.”

 Further Reading

Kennedy Space Center Virtual Reality Applications https://www.kennedyspacecenter.com/info/vr-applications

Leslie, S.
The Key Role Of VR In Preserving Cultural Heritage, Arts Management & Technology Laboratory, 2022.

Carlsson, R.
How virtual reality is bringing historical sites to life. Museum Next, 2020.

Crutcher, M.
Artificial reality is the likely future of preservation. National Council on Public History, 2020.

Melnick, K.
Preserving Historical Landmarks in VR Should Be A Priority. VRScout, 2019.

Voinea, G.D., Girbacia, F., Postelnicu, C.C., and Marto, A.
Exploring Cultural Heritage Using Augmented Reality Through Google’s Project Tango and ArCore. Springer. 2018.