Digital Twins and Artificial Intelligence as Pillars of Personalized Learning Models

Modern educational systems have not really evolved enough to meet the needs of modern students.²¹ No wonder, the percentage of dropouts from university studies is quite high (40% in the U.S. and 10% in Europe^7,9). The university student profile has changed over the years. While yesterday’s students were mainly full-time, today’s students face challenges such as work commitments, family obligations, financial constraints, physical impairments, and learning models that do not adequately engage students or help them understand core concepts.¹¹ One might think that this issue concerns only those who fail to complete their studies, but this is view is shortsighted. Today’s educational system deficiencies will affect the welfare of tomorrow’s society.

Back to Top

To improve current learning models, academic institutions around the world agree that the time has come to improve the world of education, moving from a traditional approach—where learning is standardized and available only to those with access to educational buildings—to a new paradigm that enables students to personalize their educational pathway, so they can progress at their own pace.^19,21 Future learning models must address key concerns, such as reducing dropout rates, supporting students with psycho-physical impairments, integrating on-site and online students, and personalizing the learning experience.

Digital twins—digital replicas of students—and artificial intelligence (AI) will be the pillars of innovation, accessibility, and personalization in future learning models.¹⁹ The good news is that we can build these models today: AI algorithms have made great strides in recent years, and the use of technology in education has increased enormously. Indeed, while the COVID-19 pandemic has, on one hand, strongly hampered the learning process for many people around the world, it has, on the other hand, provided a huge push toward the use of technology within classrooms (for instance, recording and streaming facilities) and within generic university spaces, such as libraries, study lounges, and hallways (for example, thermographic cameras and Internet of Things devices such as IoT sensors to monitor indoor occupancy and people flows.)

But why are current systems struggling to support today’s university students? Attending classes should be the simplest thing in the world to do, and for most students it is. For some, though, this activity poses serious problems. Let us introduce Bob, a part-time student with a job that does not allow him to attend all on-site lectures, and Alice, a student with psycho-physical impairments whose conditions force her to miss some on-site lectures. As freshmen, they are motivated, but soon they realize that the educational system is too standardized: the teaching pace does not match their own learning pace. As a result, they stop attending lectures, they procrastinate studying for exams, and, sooner or later, they will drop out.

Alice and Bob need an accessible, personalized, and inclusive learning model that supports their learning activities with suggestions about the educational material they need to study in order to learn at their own pace. For instance, suppose each on-site lecture is also delivered live via streaming technologies, transformed into engaging educational material (for example, video pills, handouts), and released quickly enough before the next lecture. Bob could keep up with the teaching pace by accessing this material when and where it best suits him. Similarly, Alice could watch the lectures she missed and could delve further into topics using the engaging educational material. In both cases, they can keep up with the lectures and might attend on-site lessons whenever they can or want. Any other student would also benefit from such a personalized approach. However, online lessons are not enough, which is where the use of digital twins comes into play.

A digital twin is a digital replica of a physical entity, and it is created by combining pieces of data from various sources. For instance, Bob’s digital twin might be created with data related to his academic background and studying activities (for instance, past exams, high-school grades, or study time), his behavior on campus (for example, library activities or classroom time), and his digital educational material consumption (for instance, the video lectures, handouts, slideshows, and books he’s consumed and the time he’s spent with those materials). A smart analysis of digital twins might provide their physical, real-world entities with important suggestions and predictions.² Indeed, it might identify knowledge and gaps and then suggest educational material to fill these gaps.

Today’s students face challenges such as work commitments, family obligations, financial constraints, physical impairments, and learning models that do not adequately engage or help them understand core concepts.

Digital twins and AI are essential to building a personalized learning model. Indeed, Bob’s digital twin will end up knowing Bob better than he knows himself. On some level, a digital twin’s ability to suggest and predict might change the world of education. For instance, suppose that Bob has four months to study for the “video communication” module. He wants to earn an excellent grade. Thanks to the analysis of his digital twin, algorithms based on machine-learning techniques might recommend that Bob spend more time at on-site lessons, which handouts he should study, which video lectures to watch, which topics to concentrate on, how much time to devote to studying, sitting in the front rows of the classroom, and more. Moreover, it will be possible to predict Bob’s grade for the exam, to notify him if he is at risk of not passing, or even he is at risk of dropping out.

This example might sound like science fiction, but personalization based on digital twins and AI is already present in many other environments,⁵ and there is no real reason not to use personalization in education systems as well. Personalized learning models might be built upon the infrastructure, data, and presentation layers shown in Figure 1. The infrastructure level must include multimedia and IoT hardware to record activities within university spaces. The data layer must embed intelligent strategies to create the student’s digital twin (using on-site, online, and offline activities such as study time on handouts, viewing time on video lectures, library and classroom time, and evaluation of past exams) and smart approaches to produce enriched digital educational material, starting from on-site lessons and intelligent algorithms to analyze digital twins and to personalize learning experiences. The presentation layer must include accessible and personalized interfaces and services.

Figure 1. Personalized learning models might be built upon the infrastructure, data, and presentation layers. Each layer presents challenges that can be addressed with different approaches and with the cooperation of researchers from different backgrounds.

Back to Top

Multimedia and IoT Infrastructures

Multimedia and IoT infrastructures must record activities performed within university spaces, must use off-the-shelf technologies to keep costs under control (cost-effectiveness), must support expansion without requiring a complete redesign (scalability), and must meet privacy regulations. Furthermore, they must address two main challenges:

Challenge #1: Infrastructure to make professors more comfortable. Once, there was chalk and a blackboard. Today, the teacher’s work is subject to multiple technological constraints, ranging from microphones (wired or wireless) to projectors (VGA or HDMI), computers (personal or classroom-based), and cameras (static or dynamic). The first challenge when building a personalized learning model that integrates on-site and online university students is the use of off-the-shelf technologies to make teachers more comfortable. Indeed, educators should be free to use slideshows, touchscreens, or the good old white/blackboard to lecture. Integration between on-site and online students should be as transparent as possible. For example, the lesson’s pace should not be slowed because on-site students with questions need to pass around a microphone so that online students can hear them. In this scenario, other potential problems can arise, such as missing or broken equipment, not to mention that sharing a microphone might break some hygiene safety rules.

This challenge is quite easy to remedy. In addition to the popularity of massive open online courses (MOOCs), video lectures have, in recent years, become a viable means of delivering knowledge and improving the learning process.^12,20 Of course, through the COVID-19 pandemic, they have been the only means to keeping in touch with students forced to learn from home. As a result, many universities around the world already have classrooms equipped to produce video lectures. In the unfortunate event that a classroom is not equipped with recording technology, off-the-shelf devices are sufficient to build a multimedia infra-structure—for example, a camera pointed at the teacher, a projector and white screen to display slides, a microphone for the teacher, and an audio/video mixer to mix the different signals. If the lesson must be broadcast live, streaming services are available at a relatively minimal cost, if not for free.

The recent widespread use of live broadcast lessons highlights the difficulties of dealing with on-site and online students at the same time. Indeed, a poor oral and visual interaction between the teacher, on-site students, and online students might be detrimental to the learning experience⁸ and the lecture’s effectiveness. Again, off-the-shelf technologies can be of great help. Figure 2 offers a hypothetical look at what the ideal classroom might offer: a video wall (right side of Figure 2) displays remotely connected students to improve the face-to-face learning experience, while the use of ceiling microphones avoids having the teacher repeat questions at a microphone or having to share a microphone among on-site students.

Figure 2. A hypothetical classroom features a video wall to improve the face-to-face learning experience and environmental ceiling microphones to allow on-site students to interact with the teacher and online students.

Challenge #2: Infrastructure to monitor students within university spaces. To monitor and understand student behavior during educational activities within university spaces (for example, attending on-site lectures in classrooms and exercises in labs, reading at the library, studying in the lounge with friends, or relaxing in the campus cafeteria), it is necessary to exploit the potential in existing infrastructures (for instance, smart building facilities, Wi-Fi networks, or surveillance and thermographic cameras), as well as to install some off-the-shelf smart technologies.

For instance, the University of Bologna is already using IoT devices and tinyML strategies in the Cesena campus to monitor classroom occupancy and to provide accessible indoor paths and orientation services to students with disabilities.¹⁵ Similarly, specific hardware, such as surveillance and thermographic cameras, has recently been installed in many university classrooms, study lounges, libraries, and hallways around the world to check body temperature or measure social distancing and people flow.¹⁶

Adopting a frugal approach, these pieces of data might be transformed into precious information about student behavior.³ For instance, indoor position and localization techniques and related devices—such as triangulation, BLE sensors, and thermographic cameras—might provide information about Bob’s presence within the classroom, to determine if he is sitting in the back row during math module lessons. Thermo-graphic cameras might help to derive information about classroom occupancy and distances between the occupants, while Wi-Fi passive tracking and BLE sensors might be exploited to collect information about how long Alice has been working on assignments with her group-project mates in the study lounge. Needless to say, all research efforts must adhere to privacy regulations.

Back to Top

Algorithms to Produce Enriched Educational Material

Video lectures are often produced with a stationary camera pointed at a teacher’s face or with a simple slideshow recording. Unedited video lectures created by simply recording a classroom lesson are very often too long and boring: Bob would surely fall asleep watching such material in the evening after a full day of work. Indeed, filmmaking theory teaches us that a long, fixed shot (that is, more than 20 secs) is not pleasant to watch and that it is necessary to keep changing the movie’s framing to retain the viewer’s interest.²

Challenge #3: Define an engaging video lecture format. Let us assume a camera is pointed at a teacher, who is using a slideshow for a lecture. What video feed should online students see? Is it sufficient to split the screen? Is the slide-show more important than the teacher? We need to address these questions, as the graphical layout greatly affects the learning experience.

Optimally, video lectures would be produced by a human director; however, budgetary reasons may preclude such an approach. Instead, more cost-effective machine-learning techniques can be used to design a virtual director that can produce engaging video lectures (with change of framing, zoom in and out) with minimal human effort.²³ For instance, machine learning can analyze audio/video (A/V) features, such as speech and gestures, to identify when the teacher feed is important (for instance, when relating an anecdote) or when the slideshow feed is important (for example, when explaining a concept on the slide). Similarly, machine learning can be used to calibrate/improve picture settings (a complicated task for teachers without the necessary expertise), as well as in post-production to remove identified gaps (for example, pieces of video with long silence).

Challenge #4: Produce enriched content for on-demand use. Suppose Bob realizes he needs to review some specific topics to prepare for an exam or to follow an upcoming lesson. He uses VCR-like controls to browse entire video lectures, but he feels like he is looking for a needle in a haystack. Suppose Alice is hearing-impaired: She would appreciate subtitles and textual handouts of the lectures. Moreover, they both would be happy to have a video summary of the lecture to meet their personal commitments.

We cannot ask teachers to produce such content from scratch. Therefore, research efforts should be devoted to the automatic production of such content, starting from the recorded video lectures.^13,22 Combining machine-learning analysis on low-level A/V features with digital twins presents endless possibilities: Video pills and summarizations, audio transcriptions, handouts in textual formats, and A/V material with content aimed at people with hearing or visual disabilities are just some examples of enriched and personalized content that can be automatically produced with the combination of machine learning and digital twins—with-out requiring the teacher to spend time or thought on producing such material.

Back to Top

Personalized Learning Experience

Assume now that Bob is having trouble with his various learning activities. He stopped attending classes because he didn’t understand what was being taught. He tried to fill in the gaps with video lectures and the textbook, but that did not work. He is disheartened, and is considering dropping out, when his fitness app reminds him to do 15 pushups and a 10-minute jog to be able to lose 2 kg in the next month. Looking at the message, he thinks about how helpful it would be to have an app that can advise him about what and when to study. Such an app could provide highly personalized suggestions, such as, “Bob, you should watch the following lectures,” “Bob, you should attend this lesson,” “Bob, you should study these handouts before watching this lesson,” and predictions, such as “Bob, be careful, your study time is too short to pass the exam.”

What happened to Bob happens to many university students because current learning models do not consider that the needs of individual students—personal constraints, previous knowledge, psycho-sensorial-motion impairments — can complicate the learning process. For instance, nearly 54% of U.S. students who dropped out of college indicated they were unable to balance work and school.⁷

Personalizing the learning process is currently considered the most promising approach for revolutionizing the world of education.^6,17 Therefore, research efforts should be devoted to personalizing the learning experience, a pathway that involves digital twins and AI. Note that all research efforts must consider privacy regulations.

Challenge #5: Create the student’s digital twin. A digital replica of a student might be created using data related to the student’s behavior on campus, academic background and studying activities, and consumption of digital educational materials (Figure 3). This type of data—for instance, which lessons the student attended, for how long, where the student sat, how much time the student spent in the library or watching online lectures, what type of digital material the student used—represents an important part of a student’s digital twin⁴ and can be collected automatically by various technologies. Data such as background information or about offline activities—for instance, academic background, possible disabilities, study time on books or notes, etc.—can be entered manually by the student.

Figure 3. The university student’s digital twin might be created using data related to the student’s behavior on campus, academic background and studying activities, and use of digital educational materials.

Challenge #6. Design recommender algorithms to personalize learning. To enable the student’s digital twin to offer highly personalized suggestions and predictions, research efforts should be devoted to the design of recommendation algorithms using classic big data analytics and AI techniques, such as classification, clustering, neural networks, and deep learning. Having a digital twin is akin to having a virtual assistant; in education, it manifests as an entity that can help to stimulate the student’s learning process. Other applications outside education already leverage digital twin technology and recommendation algorithms, with very impressive results: music streaming services know which music we like, video streaming services know our TV preferences and suggest the next series to binge, fitness apps know our bodies better than we do and suggest tailored exercise routines, and the list goes on.

Back to Top

Design of Accessible and Personalized Interfaces and Services

A personalized learning experience should have an accessible and usable interface that accounts for the needs of the individual user. The Universal Design approach¹⁸ is the most promising candidate to developing such an interface; it puts users at the center of the design activities and identifies their needs early on. Although not novel, the Universal Design concept has rarely been applied, due to both cultural and cost reasons, but this approach can benefit students, in terms of better results and greater satisfaction, as well as the teachers and universities that offer these services.

Research efforts should focus on designing interfaces that support different types of students, that support teachers when lessons are simultaneously given to on-site and online students, and that meet the requirements of current privacy regulations—for example, the GDPR in the European Union.

Challenge #7: How to support different user types. To understand students’ needs and to define the most suitable functionalities and interactions, the Universal Design approach suggests the use of individual interviews and focus groups to define system requirements and the data that must be stored within the digital twin. This task is multidisciplinary and requires the involvement of computer scientists, cognitive psychologists, and statisticians, who will define interview and focus group topics and analyze the results to understand the students’ needs. This knowledge and insight will be used to build the interfaces. This process should not be limited to visual interfaces but should also support students with special needs — for instance, by designing chatbots or conversational interfaces based on AI technology. The interfaces should also be sure to present educational content in an engaging way that motivates students to fill their educational profile with information related to their academic backgrounds and studying activities—for example, study time on books or on notes. Studies on gamification and incentive mechanisms are likely to offer interesting insights on ways to engage and motivate students to enter their data.¹⁴

Challenge #8: How to support teachers when the lesson is attended simultaneously by on-site and online university students. The simultaneous presence of on-site and online students is likely to become the new normal in the coming years, so it is necessary to provide the teacher with a simulated face-to-face experience. This is not a trivial task due to the high number of online students who do not allow the use of a simple computer screen. A video wall is necessary, but not sufficient. Indeed, research efforts must be devoted to understanding effective ways to display online students within the video wall (for instance, digital twin data can be used to display students who should be more involved in the learning process) and effective ways to allow interactions (for instance, how and when online students can interject to either ask or answer questions). Again, this is multidisciplinary research that requires computer scientists to work with cognitive psychologists.

Back to Top

What Will Be the Benefits of Personalized Learning?

Personalized learning models based on AI and digital twins have the potential for deep social impact. They can reduce student dropout rates, boost student success, improve the integration of online and on-site students, better support teachers in mixed-teaching modalities, and enhance support for and integration with students with disabilities. In short, these models can improve the overall quality of both the learning process and teaching process.

At the same time, personalized learning models will be able to meet Sustainable Development Goals (SDGs) set by the United Nations General Assembly.²¹ In particular, such models meet:

• SDG 4: Quality education—Ensure inclusive and equitable quality education and promote university learning opportunities for all.
• SDG 10: Reduced inequality—Reduce inequalities among on-site and online students and limit digital barriers to students with disabilities.

Finally, personalized learning models will have a deep economic impact, as the progress of a nation is strongly tied to the education level of its citizens. By minimizing dropout rates and equipping students with knowledge that enhances their academic success, personalized learning models will provide economic benefits for the entire community as well as the individual. This will increase the community’s global educational and cultural levels and enhance the quality and the quantity of professionals.

In conclusion, one might argue that personalized learning is still far from becoming mainstream, as it requires a considerable upfront investment, but that would be a very shortsighted view and an excuse to maintain the status quo. Indeed, on one hand, digital twins and AI—the proposed pillars on which to build personalized learning—are a reality in many other sectors and are built upon infrastructures that will last many years. On the other hand, student dropouts caused by the status quo are resulting in annual tuition losses of approximately $16.5 billion to U.S. universities,⁷ not to mention the social cost of dropouts. It is, therefore, time to press down on the innovation accelerator, as the industry is doing with the Industry 4.0 plan, because the social and economic benefits of personalized learning models are enormous.

Figure. Watch the authors discuss this work in the exclusive Communications video. https://cacm.acm.org/videos/digital-twins-and-ai