ChatGPT in Computer Science Education: Freshmen’s Perceptions

In her January 23, 2023 blog, ChatGPT in Computer Science Education, Hazzan (first author) showed that computer science teachers tend to emphasize the benefits of ChatGPT for computer science education and to highlight the opportunities it presents for computer science education over the potential threats it poses.

In this blog, we further demonstrate this perspective, exploring freshmen’s conceptions ofthis topic. The computer science, electrical engineering, and data science freshmen, whose conceptions we analyzed, were enrolled in the CS1 course at the Technion – Israel Institute of Technology, in the Winter 2023 semester. As we discovered, they too see mainly the positive impact of ChatGPT on the way the course is studied and, furthermore, suggest that the teaching of the course be revised.

Before we delve into the details, we note that:

1. We asked the students to express their perspectives on the impact of ChatGPT on the learning of the course in January 2023. At this time, ChatGPT was essentially the only famous large language model (LLM) conversational agent available for wide usage; nevertheless, as is illustrated later on in this blog, the perspectives of the students (as well as those of the teachers) can be applied to other LLM-based tools as well.

2. In this blog we talk about the impact of LLM-based tools on basic computer science courses and, specifically, on programming-oriented courses. In other words, we do not address advanced computer science courses, in which the impact of ChatGPT and other similar LLM-based tools may be expressed differently. Nevertheless, although the students’ answers focused on the learning that takes place in the CS1 course, their perspectives can be generalized easily to apply to learning processes in other programming-oriented courses and maybe even to more advanced courses, especially project courses in which, in addition to programming, students must occasionally learn new application domains.

We will now delve into the details.

Of the 956 Technion freshmen who were enrolled in a CS1 course, 756 (79%) completed the questionnaire that included the question: “In your opinion, how will ChatGPT impact learning in the course?” Table 1 shows the distribution of the responses, classified according to the way the respondents feel that ChatGPT will impact learning processes in the CS1 course.

Table 1: Computer science freshmen’s conceptions of how ChatGPT will impact learning and teaching in the CS1 course.

Following, we present several illustrative excerpts from students’ answers in each category. In the first four categories, it seems that students presumed that the way the course is taught would not change; the fifth category includes statements that deliver the message that the teaching method of the course should be changed.

A. ChatGPT will have a positive impact: The main message delivered in this category is that ChatGPT promotes and supports learning processes and cognitive abilities and can, therefore, be of assistance in a variety of ways. Here are some excerpts that illustrate this:

“In a very significant way. At the moment it has a very advanced ability to understand human languages and it can be of great help in the programming of short functions.”

“It can serve as an auxiliary tool to check errors in very basic code, excellent self-practice vis-à-vis the machine.”

“It helps understand the course material in a focused manner, instead of reviewing all of the lectures, you can ask it and it gives an answer.”

B. ChatGPT will have no impact: Arguments in this category reflect the message that since the course teaches the basics of programming, ChatGPT will not influence learning. Several students explained this perspective as follows: Since the basics of programming taught in this course should be learned mainly by repetition and practice, students who use ChatGPT, harm their own studies. The following excerpt demonstrates this perspective:

 “I don’t think that it will affect learning. Students who decide to use this tool will only hurt themselves. Practice is important in this course.”

One implication of this argument (i.e., that ChatGPT is not useful for tasks that require practice and repetition), is that ChatGPT is useful for tasks that require creativity; in the solution of such tasks, ChatGPT can be used to do the routine work while the human does the creative work.  The following quote illustrates this viewpoint:

“The add-on will affect learning in general (not only in this course). There is a system that can write solutions to almost any question, and so the essence will be comprehension and creativity, not memorization. I think that in basic courses, like our course, the effect will be minimal. The course focuses on learning the basic language, rather than on very complex problems.”

Other students argued that the basic knowledge learned in the course was available online prior to the emergence of ChatGPT and other LMM-based tools and, therefore, it will not influence the way the course is learned. The following quote illustrates this argument:

“I think that the information was available on the Internet before the Chat, and so it would not have a significant effect.”

Several students argued that ChatGPT will not influence learning in the course since it is not yet mature enough. For example:

“I think that it doesn’t work well enough yet, and it gives code with errors and bugs, which unexperienced and novices in the course will not always succeed in fixing.”

C. ChatGPT will have a negative impact: This category contains statements that suggest that ChatGPT can disrupt learning processes since it may encourage students not to think of the solutions themselves and to ask the Chat for the answer, like they currently ask Google or copy solutions of friends. The following statements illustrate this perspective:

“People use it as a search engine, which is very bad since it is not accurate and sometimes is simply very incorrect. The danger is that people rely on it heavily and do not check the validity of the answers it offers.”

“It will enable to solve homework automatically. People will not learn anything.”

Naturally, the students also addressed the ethical aspect of ChatGPT use, claiming that “It will increase the extent of homework copying” and “It will be harder to keep tests fair.” 

D. ChatGPT’s impact depends on how it will be used: Statements in this category reflect the idea that the way students use ChatGPT determines its impact. Here are several quotes that illustrate this perspective:

“Like any other tool… it can help and it can not help, depending on how people use it.”

“On one hand it can solve the homework questions. On the other hand, we have to develop this foundation and these basic programming abilities in order to progress and deal with more complex issues.”

“It depends on the students, whether they use it wisely in order to help them cope with issues they have difficulty understanding. Yet, to the same extent, the students can copy-paste without learning anything.”

E. The teaching approach of the course should be changed: The fifth category includes statements that suggest that the way the course is taught should change. As the following quote reflects, the students suggested that ChatGPT will expand the teaching methods and enable conceptual and philosophical topics to be addressed as well:

“I think that this tool can be used to present the students with more complex algorithms to which they can aspire, to address the question of the importance of the programmer in the programming process, to see whether there are programs that can do the programming instead of us, and to compare different codes that ChatGPT can produce with codes that we produce.”

Some of the students further referred to the fact that the teaching approach of the course should be changed since the profession of computing, in general, and programming, in particular, is expected to change as a result of LLM-based technologies, as is illustrated by the following quotes:

“The Chat will very much affect the course, just like it will affect all junior programming positions in the world.”

“I think that in general, the courses studied in undergraduate studies (not research) should prepare the students as much as possible for industry. Therefore, in general, if the industry adopts this tool (like it seems to be happening), there is no reason the university should not use the tool as well, possibly with certain adaptations.”

“I think amazing things can be done with it. Just like there are software testers in reality, it can be used to check bugs in code. I think that if it is decided to integrate it into learning, it has great potential. It prepares students for the real world.”

Discussion

Both computer science teachers (as described in Hazzan’s blog) and freshmen in the CS1 course (as presented in this blog) clearly see the potential contribution of ChatGPT to computer science teaching and learning processes and how it can improve computer science education.

One way to explain this potential impact of ChatGPT on computer science education is to examine the type of conversation it enables, through prompts (that is, the question/request presented to ChatGPT) formulated in natural language in a way that increases the chances to receive meaningful answers. This type of conversation with the computer takes place on a higher level of abstraction than that which takes place in traditional IDEs. Specifically, in computer science learning processes, computer feedback provided by IDEs is limited to determining whether a learner’s program is syntactically and semantically correct. Conversational LLM-based agents (like ChatGPT) offer more freedom to discuss related topics beyond program correctness, such as alternative solutions and explanations based on the learner’s background. One of the students expressed this idea in the following statement:

“Wow!!! A strong tool that improves the course in an amazing way. In my opinion it helps a lot to improve the course since it functions as a conversation.”

The importance of the feedback that the computer provides in learning processes has been discussed since the early days of integration of computers in educational environments. Professor Seymour Papert was one of the pioneers in using computers for educational purposes, and particularly in the realm of programming. Papert encapsulated his pedagogical perspective in the concept of “constructionism,” which emphasizes the importance of the learners’ active involvement in constructing their own knowledge through hands-on experiences that are supported by feedback provided by the computer (Harel and Papert, 1991).

Now, with conversational agents such as ChatGPT, which enable to receive feedback from the computer on a higher level of abstraction, the attention given to the rigid formality of programming languages (whose purpose in most cases serves the compiler or interpreter) can be shifted and directed towards problem solving and thinking processes. One of the students expressed this idea as follows:

“I think the course should embrace the Chat, and should actually teach the students how to think in computer science and focus less on programming-related issues.”

This kind of conversation with the computer has the potential to increase diversity in computing. Specifically, since the formality of the computing process will be alleviated when programming is done using LLM tools and the conversation with the computer is conducted in a natural human language, underrepresented populations in computing, for whom this rigid formality was traditionally a barrier, will join the profession of programming, thus increasing its  diversity. Furthermore, this wider and more diverse population may contribute its verbal and other capabilities to aspects related to the conversation with the AI-based tools. One of the students claimed that:

“A very significant tool that can assist students who have no background and help them understand the material better, especially if they have nobody to help them or if the pace is way too fast for them.”

We note that ChatGPT itself highlighted an important difference between its feedback and the feedback provided by traditional IDEs. When asked “So, let’s focus on computer science education. What is the difference between the feedback that ChatGPT provides and the feedback provided by regular IDEs to students who develop computer programs?” it answered that “The feedback provided by ChatGPT and regular Integrated Development Environments (IDEs) to students who develop computer programs differs in several ways.” Then, it presented the following differences and explained each one of them briefly. Specifically, on the one hand, it mentioned the nature of feedback (in the form of natural language responses), contextual understanding and flexibility in interaction (beyond code-specific issues), and learning support (beyond code correction) and on the other hand, it mentioned two significant differences. First, while regular IDEs offer real-time code execution, enabling students to immediately see the results of their code and debug issues directly within the development environment, ChatGPT does not have direct execution capabilities and cannot evaluate code in real time; second, while regular IDEs offer a comprehensive environment for writing, testing, and deploying code, ChatGPT does not provide the same level of features for all these actions. Furthermore, it summarizes: While ChatGPT can offer more interactive and conceptual support, regular IDEs excel in providing direct, code-specific feedback and an environment tailored to the coding process.

Based on the above perspectives and additional arguments, it appears that generative AI, in general, and LLM-based conversational agents, specifically, have the potential to be a disruptive technology for computer science education. In one of our upcoming blogs, we will elaborate on this assertion.

 

References

Harel, I. and Papert, I. (1991). Constructionism, Praeger.

 

Orit Hazzan is a professor at the Technion’s Department of Education in Science and Technology. Her research focuses on computer science, software engineering, and data science education. For additional details, see https://orithazzan.net.technion.ac.il/. Yael Erez is a lecturer at the Technion’s Faculty of Computer Science and a staff member at the Department of Electrical Engineering at the Braude College of Engineering in Karmiel. She is currently a doctoral student at the Technion’s Department of Education in Science and Technology, under the supervision of Orit Hazzan.