Prutor: An Intelligent Learning and Management System for Programming Courses

Programming education in India faces an uphill task of educating two-plus million students who enroll in degree programs with coding as a core skill.¹⁰ Fulfilling this demand presents unique challenges due to inadequate infrastructure and the unavailability of technical content in regional languages⁶ with unfortunate outcomes: more than 90% of Indian graduates have coding skills inadequate for IT roles, and more than 37% struggle to write code that even compiles.¹³

Studies indicate a steep decline in coding skills between graduates from top-100 colleges and the rest.¹² While concerning, since only a tiny minority of students enroll in top-tier colleges, this is not entirely surprising. A recent study indicates that even experienced instructors at non-top-tier colleges in India struggle to write code.¹⁰ Offering high-quality instruction (monitoring student progress, giving feedback, uniform grading) thus becomes doubly challenging when instructors teach large class sizes with little to no teaching support.

---

More than 90% of Indian degree program graduates have coding skills inadequate for IT roles, and more than 37% struggle to write code that even compiles.

---

It is notable that instructors at top-tier colleges also face large classes but usually have the luxury of employing senior students as teaching assistants (TAs). However, an exclusive reliance on TAs who are full-time students themselves has drawbacks of its own such as delays in doubt clearance and exposing students to variability in TA expertise, leading to non-uniform grading and unintentional bias.^11,18

Back to Top

Can Intelligent Tutoring Systems Provide a Solution?

Prutor^a is a browser-based tutoring system⁷ offering an integrated development environment (IDE) with a text editor, linter, interactive access to a remote compiler, and a debugger (see Figure 1). Prutor supports both manual and automated test case-based grading. Its browser-based nature allows students to focus on coding by ensuring a consistent experience independent of the client OS, compiler, or runtime versions. Since its release in 2014, Prutor has been used by more than 45 programming courses at over five institutes. Indeed, Prutor is used with an introductory computer science course offered to 1,000-plus students every year at IIT Kanpur alone.

Figure 1. Prutor provides a familiar, easy-to-use, and dynamic platform for both students and course administrators, allowing practice or laboratory sessions, assignments, and multisession examinations to be conducted and graded over a secure and scalable platform.

Although Prutor was created to ease the coordination of coding labs and examinations, it soon became apparent that data collected by its telemetry modules opened entirely new research and product directions that could significantly improve student and instructor experience. Prutor captures code being written by students at regular intervals. These snapshots of code at various stages of completion offer a ringside view of the problem-solving approaches used by students, help understand difficulties faced by them, and reveal concepts where they make the most mistakes. However, could this data also be used to automate mechanical and time-consuming tasks performed by instructors and TAs? Apart from reducing manual effort, this could also improve instruction quality in resource-strapped situations.

Our research group at IIT Kanpur has used Prutor data to develop AI tools targeting core pedagogical aspects of a course, such as question creation, doubt clearing, and grading (see Figure 2). Prutor data has also enabled researchers at IISc Bengaluru, National University of Singapore, Queensland University of Technology, and Innopolis University to develop solutions for error repair and improved feedback generation resulting in work appearing in several top-tier publications.^{2,3,4,5,8,9,11,15,17,18}

Figure 2. A schematic depicting the workflow of AI-powered pedagogical tools being developed for the Prutor platform.

Back to Top

Current Work and Future Directions

Automated compilation-error repair: Suggesting fixes for student code that fails to compile is a significant component of TA’s workload but essential for beginners who may find compiler error messages cryptic and unhelpful (see Figure 3). Error-repair systems^2,5 created using tools from Web-scale recommendation give state-of-the-art repair accuracies. They also offer pedagogically valuable features such as giving only repair hints, instead of the actual repair, to encourage the student to ruminate.^14,16

Figure 3. Intelligent tutoring systems such as TRACER⁷ offer a unique way to augment classroom teaching.

A search engine for programming problems: Creating assignments and exam problems is a time-consuming exercise. Searchable problem banks^14,16 created using Prutor data from historical course offerings allow instructors to search for problems with a certain difficulty level or those that require use of specific constructs (if, for, while), types (arrays, pointers), or programming techniques (recursion, DP).

Code clustering: Identifying student submissions with a similar syntactic or semantic structure has several applications. This includes plagiarism detection, rapid manual grading with reduced grade variance for similar solutions (by marking submissions in the same cluster together), program repair (by identifying differences between structurally similar but semantically different submissions), and test case creation to help students spot mistakes in their code¹ (by comparing correct and incorrect solution clusters).

Natural language feedback: Existing tools offer repair suggestions in the form of code snippets that may be less helpful for students who are already struggling with programming language constructs.^4,18 Our current research is focusing on generating natural language feedback by mapping auto-generated repairs to simple natural language descriptions.

We see immense potential in AI applications built using data collected by interfaces such as Prutor for decreasing the sharp divide in instruction quality across institutions, reducing manual effort in conducting large courses, and presenting compelling research challenges that fuel the next generation of product innovations.

Acknowledgments. The authors thank all their collaborators for their contributions to the various Prutor-driven projects. Purushottam Kar thanks Microsoft Research India and Tower Research for research grants.