Research and Advances
Computing Applications India Region Special Section: Hot Topics

Prutor: An Intelligent Learning and Management System for Programming Courses

Posted
code from a display screen reflected onto computer user
  1. Introduction
  2. Can Intelligent Tutoring Systems Provide a Solution?
  3. Current Work and Future Directions
  4. References
  5. Authors
  6. Footnotes
code from a display screen reflected onto computer user

Programming education in India faces an uphill task of educating two-plus million students who enroll in degree programs with coding as a core skill.10 Fulfilling this demand presents unique challenges due to inadequate infrastructure and the unavailability of technical content in regional languages6 with unfortunate outcomes: more than 90% of Indian graduates have coding skills inadequate for IT roles, and more than 37% struggle to write code that even compiles.13

Studies indicate a steep decline in coding skills between graduates from top-100 colleges and the rest.12 While concerning, since only a tiny minority of students enroll in top-tier colleges, this is not entirely surprising. A recent study indicates that even experienced instructors at non-top-tier colleges in India struggle to write code.10 Offering high-quality instruction (monitoring student progress, giving feedback, uniform grading) thus becomes doubly challenging when instructors teach large class sizes with little to no teaching support.


More than 90% of Indian degree program graduates have coding skills inadequate for IT roles, and more than 37% struggle to write code that even compiles.


It is notable that instructors at top-tier colleges also face large classes but usually have the luxury of employing senior students as teaching assistants (TAs). However, an exclusive reliance on TAs who are full-time students themselves has drawbacks of its own such as delays in doubt clearance and exposing students to variability in TA expertise, leading to non-uniform grading and unintentional bias.11,18

Back to Top

Can Intelligent Tutoring Systems Provide a Solution?

Prutora is a browser-based tutoring system7 offering an integrated development environment (IDE) with a text editor, linter, interactive access to a remote compiler, and a debugger (see Figure 1). Prutor supports both manual and automated test case-based grading. Its browser-based nature allows students to focus on coding by ensuring a consistent experience independent of the client OS, compiler, or runtime versions. Since its release in 2014, Prutor has been used by more than 45 programming courses at over five institutes. Indeed, Prutor is used with an introductory computer science course offered to 1,000-plus students every year at IIT Kanpur alone.

f1.jpg
Figure 1. Prutor provides a familiar, easy-to-use, and dynamic platform for both students and course administrators, allowing practice or laboratory sessions, assignments, and multisession examinations to be conducted and graded over a secure and scalable platform.

Although Prutor was created to ease the coordination of coding labs and examinations, it soon became apparent that data collected by its telemetry modules opened entirely new research and product directions that could significantly improve student and instructor experience. Prutor captures code being written by students at regular intervals. These snapshots of code at various stages of completion offer a ringside view of the problem-solving approaches used by students, help understand difficulties faced by them, and reveal concepts where they make the most mistakes. However, could this data also be used to automate mechanical and time-consuming tasks performed by instructors and TAs? Apart from reducing manual effort, this could also improve instruction quality in resource-strapped situations.

Our research group at IIT Kanpur has used Prutor data to develop AI tools targeting core pedagogical aspects of a course, such as question creation, doubt clearing, and grading (see Figure 2). Prutor data has also enabled researchers at IISc Bengaluru, National University of Singapore, Queensland University of Technology, and Innopolis University to develop solutions for error repair and improved feedback generation resulting in work appearing in several top-tier publications.2,3,4,5,8,9,11,15,17,18

f2.jpg
Figure 2. A schematic depicting the workflow of AI-powered pedagogical tools being developed for the Prutor platform.

Back to Top

Current Work and Future Directions

Automated compilation-error repair: Suggesting fixes for student code that fails to compile is a significant component of TA’s workload but essential for beginners who may find compiler error messages cryptic and unhelpful (see Figure 3). Error-repair systems2,5 created using tools from Web-scale recommendation give state-of-the-art repair accuracies. They also offer pedagogically valuable features such as giving only repair hints, instead of the actual repair, to encourage the student to ruminate.14,16

f3.jpg
Figure 3. Intelligent tutoring systems such as TRACER7 offer a unique way to augment classroom teaching.

A search engine for programming problems: Creating assignments and exam problems is a time-consuming exercise. Searchable problem banks14,16 created using Prutor data from historical course offerings allow instructors to search for problems with a certain difficulty level or those that require use of specific constructs (if, for, while), types (arrays, pointers), or programming techniques (recursion, DP).

Code clustering: Identifying student submissions with a similar syntactic or semantic structure has several applications. This includes plagiarism detection, rapid manual grading with reduced grade variance for similar solutions (by marking submissions in the same cluster together), program repair (by identifying differences between structurally similar but semantically different submissions), and test case creation to help students spot mistakes in their code1 (by comparing correct and incorrect solution clusters).

Natural language feedback: Existing tools offer repair suggestions in the form of code snippets that may be less helpful for students who are already struggling with programming language constructs.4,18 Our current research is focusing on generating natural language feedback by mapping auto-generated repairs to simple natural language descriptions.

We see immense potential in AI applications built using data collected by interfaces such as Prutor for decreasing the sharp divide in instruction quality across institutions, reducing manual effort in conducting large courses, and presenting compelling research challenges that fuel the next generation of product innovations.

Acknowledgments. The authors thank all their collaborators for their contributions to the various Prutor-driven projects. Purushottam Kar thanks Microsoft Research India and Tower Research for research grants.

    1. Agarwal, N. and Karkare, A. LEGenT: Localizing errors and generating testcases for CS1. L@S, New York, 2022.

    2. Ahmed, U.Z., Kumar, P., Karkare, A., Kar, P. and Gulwani, S. Compilation error repair: for the student programs, from the student programs. ICSE, 2018.

    3. Ahmed, U.Z., Sindhgatta, R., Srivastava, N. and Karkare, A. Targeted example generation for compilation errors. ASE, 2019.

    4. Ahmed, U.Z., Srivastava, N., Sindhgatta, R. and Karkare, A. Characterizing the pedagogical benefits of adaptive feedback for compilation errors by novice programmers. ICSE, 2020.

    5. Chhatbar, D., Ahmed, U.Z. and Kar, P. MACER: A modular framework for accelerated compilation error repair. AIED, 2020.

    6. Cutrell, E. et al. Massively empowered classroom: Enhancing technical education in India; https://bit.ly/3OtW6fC

    7. Das, R. Ahmed, U.Z., Karkare, A. and Gulwani, S. Prutor: A system for tutoring CS1 and collecting student programs for analysis, 2016; arXiv:1608.03828.

    8. Gupta, R., Kanade, A. and Shevade, S.K. Deep reinforcement learning for syntactic error repair in student programs. AAAI, 2019.

    9. Gupta, R., Pal, S., Kanade, A., and Shevade, S. DeepFix: Fixing common C language errors by deep learning. AAAI, 2017.

    10. Kumar, V. and Amey Karkare, A. Instructor performance on progressively complex programming tasks. ITiCSE, 2021.

    11. Parihar, S. et al. Automatic grading and feedback using program repair for introductory programming courses. ITiCSE, 2017.

    12. SHL. National Programming Skills Report, 2017; https://bit.ly/3Qz3qIA.

    13. SHL. National Employability Report for Engineers, 2019; https://www.shl.com/en/india/employability-reports/.

    14. Shaikh, F.M. Advancements in AI-assisted compilation error repair and program retrieval, Master's thesis. IIT Kanpur, 2021.

    15. Sharma, S., Agarwal, P., Mor, P, and Karkare, A. TipsC: Tips and corrections for programming MOOCs. AIED, 2018.

    16. Sharath, HP. Real-world deployments of AI-assisted tools for compilation error repair and program retrieval, Master's thesis. IIT Kanpur, 2021.

    17. Verma, S. and Roy, S. Synergistic debug-repair of heap manipulations. ESEC/FSE, 2017.

    18. Yi, J., Ahmed, U.Z., Karkare, A. Tan, S-H, and Roychoudhury, A. A feasibility study of using automated program repair for introductory programming assignments. ESEC/FSE, 2017.

Join the Discussion (0)

Become a Member or Sign In to Post a Comment

The Latest from CACM

Shape the Future of Computing

ACM encourages its members to take a direct hand in shaping the future of the association. There are more ways than ever to get involved.

Get Involved

Communications of the ACM (CACM) is now a fully Open Access publication.

By opening CACM to the world, we hope to increase engagement among the broader computer science community and encourage non-members to discover the rich resources ACM has to offer.

Learn More