Skip to content

Experience in CS1010S

We share our experience of deploying the ITS in CS1010S during the fall semester of the 2023-2024 academic year, which involved 571 students.

Setup

CS1010S covers topics such as recursion, higher-order functions, abstract data types, basic data structures, sorting algorithms, and object-oriented programming through nine weekly assignments. Students can make multiple attempts to revise their solutions before the deadline. After the deadline, TAs manually review incorrect submissions and write personalized feedback to students who submit incorrect solutions.
As a first step towards complete deployment, this setup restricted the generated feedback visible only to the human tutors.

  • We enabled ITS for 20+ programming assignments spanning six weeks of assignments.
  • We excluded the first two introductory weeks and the final week, which involves object-oriented programming that the current ITS does not support.
  • Coursemology forwards feedback generation requests to ITS only for finalised submissions that fail pre-defined test cases.
  • Correct submissions from past semesters were used as reference solutions to enhance the alignment component.

Results

Throughout the semester, Coursemology collected 3,117 incorrect solutions from 571 students.

  • The deployed ITS successfully generated correct patches (i.e. pass all test cases) for 1,758 (56.4%) incorrect submissions.
  • Among those 1,758 correct patches, 77.2% of the ITS-generated feedback are semantically equivalent to TAs feedback.

The main bottleneck is that the current ITS focuses on offering last-mile help to students. It is not capable of generating step-by-step feedback for students who are stuck at the beginning or middle of a task. To address this limitation, we are planning to develop APR techniques for incomplete programs. Fortunately, the modular design of the ITS allows us to easily integrate such new APR techniques into the system once available.