The Role of Computing in the Study of Latin American Cultural Heritage

Latin America (LATAM) has witnessed many of the world’s most important ancient cultures. From the Mayas to the Incas, countless cultures emerged, leaving us with both the physical and cultural legacies where archaeologists, anthropologists, and historians seek to decipher the history of our ancestors. Computing plays an important role in building tools that allow specialists to research cultural legacies.

In this article, we present some of the recent efforts to build state-of-the-art computational technology to preserve and conserve cultural heritage in LATAM. We describe how the analysis of symmetries in shapes could help to restore damaged archaeological objects in Peru, and present recent efforts to apply deep learning and data-driven methods to assist in digitizing pre-Columbian objects. We also show how deep learning could help discover new geoglyphs in Nazca, Peru, through computer-vision methods and discuss the challenges in this area and how they can be addressed.

The Larco Museum is one of the most important museums in Lima, Peru, housing an invaluable collection of pre-Hispanic and pre-Inca archaeological material from all over Peru. It is for this reason that the conservation and restoration work within the museum is no small challenge. In 2017, in collaboration with the museum, we started an initiative to develop computational tools to support conservation work, in particular the analysis of incomplete or damaged archaeological objects.

The museum managed to scan 41 archaeological pieces into 3D models using a low-power laser scanner where all the pieces were objects with missing parts. The aim was to develop a method that takes the incomplete 3D object, repairs it virtually, and then uses the repaired 3D model to speed up the actual physical repair. Our starting point was the results we had previously obtained during research for the PRESIOUS project,⁶ a European Union endeavor developed between 2013–2016 that made significant advances in digital heritage conservation. In that project, we proposed a method for making virtual repairs under the assumption that objects were symmetrical by reflection.^5,9 The method made use of heat-diffusion theory to compute a function on the surface of the 3D object, such that points that were symmetrical had similar function values. However, these results from the PRESIOUS project could not be used directly on the Larco museum data because the geometric characteristics of the Peruvian objects were very different from those of the objects used in Europe.

The main difference was that the Peruvian objects were mainly rotationally symmetrical. Moreover, they contained many decorations or handles that were not part of the overall symmetry, so we had to create a new method able to detect partial symmetries. A partial symmetry is evidenced when only a part of the object is symmetric. The result of this research was an algorithm to detect points on the surface with symmetric correspondences, which was also robust to missing geometry and partial symmetries.¹⁰ This allowed us to handle partial symmetries (see Figure 1). On the other hand, we developed a clustering algorithm between circular sets of potential symmetric points that very accurately determined the axis shared by the circles of the largest cluster and that corresponded to the potential symmetric axis of the object. Figure 1 shows the result of the symmetry detection algorithm on some objects from the Larco collection.

Using the detected symmetry, it was possible to replicate the geometry that corresponded to that symmetry and complete the missing parts of the object.

Deep Learning for CH Conservation

The emergence of deep learning changed the perspective on how to computationally solve problems in cultural heritage. The first approach we investigated in 2017 was the use of generative adversarial networks (GANs) to repair archaeological objects.² This method proposes to train a conditional GAN that receives an incomplete object in volumetric representation and produces a potential complete object for that input. One of the main challenges of this type of method is it is necessary to have a good amount of data to train the model. Unfortunately, until then there was no dataset large enough to train models. Our approach was to use standard dataset models such as ShapeNet¹ and ModelNet¹³ but having the same geometric structure of the archaeological objects and then just refine the training with specific models through fine-tuning. Although the results were encouraging, there were two major problems: The volumetric representation only allowed a degree of resolution that was not optimal for the reconstruction problem, and the process of transferring knowledge from general objects to archaeological objects was not necessarily a good assumption because of the high variability of shapes in the cultural heritage objects.

In 2020, we undertook a collaboration with the Josefina Ramos de Cox Museum in Lima, Peru to digitally acquire as many 3D objects as possible from their collection. After more than a year of work, we were able to collect close to 1,000 pieces corresponding to the pre-Inca cultures of the central coastal zone of Peru. To the best of our knowledge, this is the largest collection to date of archaeological objects available in the world.

One of the first advances made with this dataset was to analyze its potential use with machine-learning (ML) techniques. In 2021, we organized a track within the SHREC competitions,⁸ where two learning tasks were proposed with this dataset: classification by shape and classification by culture.¹¹ SHREC is an initiative to foster the creation of datasets and benchmarks in the community of shape analysis. In our proposed track, different teams from all over the world tried to solve the tasks with different methods, ranging from classical geometric methods to the use of neural networks on 3D data to make the classifications. In this comparison, we made two findings: Representing 3D objects as a set of rendered images gave the best results for classification by shape, and classification by culture is more challenging as it involves higher-level semantic information. Both findings are open doors for future research directions.

Moreover, within the original dataset of the Josefina Ramos de Cox Museum collection, there is a set of 82 archaeological objects that possess highly variable and visually rich color engravings (see Figure 2). Working together with specialists at the museum, and in collaboration with colleagues from the Technical University of Graz, we developed an application for the annotation of engraved patterns on the objects. As a result, we now have the textured 3D objects, the reference patterns, and the annotations of the occurrences of each pattern on the objects.⁴ This information is useful as a starting point to test detection, matching, retrieval, and even generative models to analyze the patterns.

Within the collection of objects, there were artifacts with imperfections both natural and introduced by the acquisition system (see Figure 2). There were artifacts that were missing the base, so the captured 3D meshes had a noticeable hole in the bottom of the artifacts. The hole was large enough to be treated as a simple hole in a triangular mesh, so generating that missing information was a major challenge. But unlike the Larco museum experience, we now had enough data to do something at a data-driven level. The first decision we made was to use a better representation for the models, and we decided to use a point cloud representation. In this representation, a 3D model is defined as a set of points that fall on the surface of the object.

Our first contribution to solving this problem was an encoder-decoder neural-network architecture that receives an incomplete point cloud and generates the missing point cloud.¹² A second neural network then refines the result so that both point clouds (the original and the prediction) are merged and combined into a single point cloud representing a continuous surface. Figure 3 shows the general architecture and an example of input and output obtained by the neural network after being trained.

The logic behind the proposed architecture is that the encoder network analyzes the input object and transforms the input into a latent vector containing information about the overall structure of the object. Subsequently, the decoder network takes the latent vector and generates a point cloud that represents the missing part while preserving the structure of the input object. At this point, we regularize the generation problem by incorporating guidance information during training, so the network knows approximately what it must learn to generate. However, the encoder-decoder network alone does not contain any prior information about the surface regularity between the incomplete object and the predicted part. It is for this reason the refinement network learns a shift function over the predicted point cloud to improve the inter-point density between the two sets.

Our second contribution was a training methodology that allowed us to complete objects while preserving their structure but avoiding overfitting at all costs. We compiled a collection of objects with similar structures to the objects we wanted to repair. We used a combined dataset between objects from the Pottery Benchmark,³ objects from the Bowl and Jar classes of the ShapeNet dataset, and complete objects from the Josefina Ramos collection of Cox. During training, each object was randomly split into an incomplete part and a missing part, and the network was trained to learn to complete the incomplete object. Since the splitting was random, the network never saw the same training sample during the entire training. In this way, we ensured that the network learned to recover the structure of the objects.

The implementation, data, and experiments performed in this work are publicly available and can be found in the original publication.¹²

More advanced deep-learning techniques can help to analyze interesting geometric patterns and meaningful semantic information from ancient objects. Nevertheless, we must enrich the existing datasets using expert-based annotated data. Our recent effort is to build annotation tools for experts and let them introduce information about geometric structure (symmetries) and part segmentations (see Figure 4). Our goal is to use this data to make learning models even more specific to cultural heritage objects and their characteristics.

Detection of Geoglyphs in Nazca Lines

The Nazca Lines are a unique worldly heritage, composed of giant geoglyphs made on a desert terrain that generally can only be appreciated from the air, given their magnitude. To preserve this legacy, one of the first necessary objectives is to detect new geoglyphs that are still lost in the desert. This task is very challenging given that the only way to do so is with the naked eye from the air or from high-resolution aerial images.

Recently, a collaborative work between researchers from Japan and France proposed a new methodology to detect geoglyphs in the Nazca desert.⁷ This method uses deep learning to analyze aerial images and accurately determine the possible occurrence of a geoglyph. However, an interesting challenge is how to achieve the goal if there is little data and there is high variability among existing geoglyphs, so there is no recurrent pattern that can be exploited by neural-network architectures to process images.

The researchers in charge of this project formulated the problem as an object detection problem in computer vision. The first step was to label the images of known geoglyphs to have a base dataset on which to train a model. However, the authors propose to perform a different labeling than that used in typical detection problems. They propose to divide a geoglyph into smaller parts they call “elements” that contain salient features of the shapes. Under this logic, it is easier for a neural model to learn to distinguish simple patterns than to over-fit entire geoglyphs. Also, once an element is labeled, different versions of it are re-created by applying transform operations to the images and taking different sub-images around the element to add some scale invariance to the same pattern. What was achieved was to obtain a representative dataset of geoglyph features, and the detection neural network learned to differentiate where these features exist in new images.

Thanks to the experiments carried out and the definition of this specific data-management methodology, the study made it possible to detect four new, previously undiscovered geoglyphs. As specified by the authors of the study, this method made it possible to reduce the detection time by a factor of 21 compared to the naked-eye process.

Challenges

Computing can be a game-changing technology to accelerate the progress of cultural heritage research in LATAM. In this article, we show just a small sample of all the efforts being made on the continent, and that makes us think that computing still has a lot of potential to support interdisciplinary and consensual work to preserve our heritage. However, there are still challenges to be addressed, such as the following:

• Initiatives at the government level. Research in computing for cultural heritage should be a priority in most research and development funding programs in LATAM governments. This could promote long-term lines of research that could serve as a basis for significant findings in the region.

• Data access. LATAM has an extensive and varied cultural heritage. The region should be a leader in providing information and data to conduct research on cultural heritage, not only at the regional level but also to draw the attention of international communities and motivate the global scientific community to solve challenging problems.

• Academic workforce. The study of cultural heritage requires the participation of various areas of computing. This article describes initiatives from computer graphics and computational vision; however, data management, software development, infrastructure development, human-computer interface, and collaborative computing, to name a few, are required. This is a call to action to try to motivate researchers from different areas to take an interest in problems related to our cultural heritage. The technical challenges span from the research and implementation of low-cost, AI-assisted digitization methods to the automatic management and processing of multimodal cultural heritage data.

The approach to these challenges has an ultimate objective: to allow us to take care of the legacy of our ancestors through computation, which will enable us to better connect society with its past, from which we have much to learn.

Acknowledgments

Thanks to all my students and colleagues who have collaborated with me over the years to bring forward some of the ideas presented in this article. I would also like to thank my grant ANID Chile—Research Initiation Program—Grant N° 11220211.