How Is History Shaping My Ph.D. Research? Unexpectedly!

Introduction by Orit Hazzan:

In line with my (Orit Hazzan’s) co-authored blog on The Triad of Graduate Studies, Yael Erez’s Ph.D. deals with an authentic topic that has the potential to have real impact beyond the knowledge created in the research. As authentic research, it is influenced by events that happen in its environment. As the following fascinating description illustrates, since Yael started her Ph.D. studies two years ago, we have witnessed several crises that have not only led us to change the course of Yael’s research, but also to improve our understanding of its very subtle details. In fact, Yael’s research has been impacted and shaped by events on four levels: the worldwide level, the national level, the institutional level, and the course level. Specifically, these events are, respectively: the COVID-19 pandemic, political events in Israel, a major cyberattack on our institution – the Technion – Israel Institute of Technology, and a crash of a computational tool used in the course that is the setting of Yael’s research.

The rest of the blog is written in first person by Yael, as she tells the story of her doctoral research.

Background

When I started my Ph.D. in computer science education, I didn’t have a clue that several seemingly unrelated historic events will make it what it is today. I simply wanted to advance my professional development by studying for a Ph.D. degree, and thought it would be natural that my research focuses on the course I was (and still am) teaching at the Technion: Introduction to Computer Science I.

My Ph.D. thesis deals with competency-based and automatic (CBA) assessment in courses that can be defined as massive on-campus programming-oriented for practitioners (MOPP) courses. In our case, the practitioners are undergraduate students. The research target is to investigate the adoption process of CBA assessment methods in MOPP courses that are in line with the changes taking place in higher education and in industry. Accordingly, the research questions are: 1. Is it possible to design a process for the adoption of CBA assessment in MOPP courses? and 2. If it is, what are the characteristics of such a process and what guidelines can be given for its implementation?

Each year, more than 1,000 Technion computer science, electrical engineering, and data science undergraduates enroll in the Introduction to Computer Science I course, making it a MOPP course. In such populous courses, it is almost impossible to evaluate and grade paper-based exams in a consistent and fair manner. Even though this observation is well-known, until I started my Ph.D., the final exam in that course (which I teach), as in many other MOPP courses worldwide, was written and evaluated manually. Moreover, as we shall see below, the exam format was not changed in one stroke through a pre-determined set of activities, but rather was reshaped gradually by global and local events.  

The description of this process is presented in four parts according to the timeline of the events, as summarized in the following table:

Date	Level	Event	Lessons learned /research insights
March 2020	Worldwide	COVID-9 pandemic	CBA assessment in general and in executable exam in particular has become the research focus.
December 30, 2022	Course	Crash of a computational tool during the exam	The concept of the diffusion of technological innovation is relevant for the adoption process of CBA assessment.
February 14, 2023	Institutional	Cyberattack on our institution	The new exam format is anchored, and the institutional risk management approach should include digitized online exams.
March 26, 2023	National	Political events in Israel	The high-tech industry’s perspective becomes more relevant than before.

 

Worldwide level: The COVID-19 pandemic

A very short time after having several meetings with my supervisor, Orit Hazzan, in which I mentioned the discrepancy between our paper-based exam and the commonly used software development methods implemented in the industry, the COVID-19 pandemic stormed into our lives and forced students to take all of their final exams remotely, online. This change in the exam format applied also to the exam in the Introduction to Computer Science I course, which I was teaching (and still teach). Due to the limited time we had to adjust the exam format to the new setting, we decided to keep the paper-based exam and to ask the students to scan it and submit it through our LMS. During the exam, however, it was easy (and ridiculous) to observe that the students were using the computer for everything but programming: the students connected to a video call for proctoring purposes, downloaded the exam from the LMS, scanned it, and finally uploaded it back to the LMS. Indeed, it looked bizarre and led us to the realization that the exam format really should be changed. Consequently, the focus of our research became CBA assessment with a special focus on executable exams.

Course level: Crash of a computational tool during the exam

Over the past two years of research, the format of the final exam in my course was gradually changed to an executable format. When taking an executable exam, students not only use computers to write their code, but can also use the programming environment to execute and test their programs during the exam itself. Everything is digital, automatic, and online. Although this looks very promising, it does have one major disadvantage, which we encountered when the special LMS plugin crashed during the mid-semester exam of the Winter 2022-23 semester. Following this incident, lessons were learned and adjustments were made to ensure the system works properly during the final exam. Nevertheless, this event provided important data for my research. It was clear that the adoption of the new exam format must be examined not only from learning and teaching perspectives, but also from a technical perspective, as a technological innovation.

Institution level: Cyberattack on our institution

Just before the final exam of the Winter 2022-23 semester, it seemed as if the new exam format was stable and reliable. Alas, our institution was then attacked by hackers, and all digital systems were shut down, including our dedicated digital programming environment created for the exam. At that time, it was clear to everyone that we were not going to return to the paper-based exam format, and the exam date was postponed to enable us to prepare an executable exam that can be administrated without using the special LMS plugin and even, if needed, without an Internet connection. Again, this event provided very good data for my research. It is clear that the change in the exam format proved to be anchored and is here to stay. It is also clear that the institutional risk management approach should include digitized online exams.

National level: Political events in Israel

The current attempts of the new Israeli government to implement revolutionary reform in the country’s judicial system without broad consensus is creating a deep rift in Israeli society. The night before my candidacy exam, the Prime Minister decided to fire the Minister of Defense. This action led to a wave of strong and emotional reactions, as hundreds of thousands of people flowed into the streets to protest, all night long. As a result, the Technion, along with all other leading universities in Israel, stopped all academic activities until further notice. Although examiners and I got very little sleep that night, we decided to conduct the exam online. I passed the exam, and later that day the government declared it would suspend its attempts to proceed with the reform. One lesson we learned from this event is that since the Technion is closely connected to the society in which it is embedded and cannot disregard even political events, my research should also examine the environment outside of the academic arena. Accordingly, our next step will be to study the perspective of the hi-tech industry on CBA assessment in MOPP courses.

I am now waiting to see what will come next…

Epilogue

My decision to call my reflection “How Is History Shaping my Ph.D. Research?” reflects two important lessons:

1. Unexpected events can be viewed as good opportunities to gain new insights about the research topic.

2. Authentic research topics should be attentive to what is going on in the research environment, responsive to those events and reflective of them.

 

Yael Erez is a lecturer at the Technion’s Faculty of Computer Science and a staff member at Ort Braude’s Department of Electrical Engineering. She is currently a doctoral student at the Technion’s Department of Education in Science and Technology, under the supervision of Orit Hazzan.