Abstract This article presents a continuous assessment methodology for a computer programming course supported by an automatic assessment tool, applied to the practical programming exercises performed by the students. The interaction between the students and the assessment tool was studied through quantitative analyses. In particular, the solutions proposed by the students (computer programs) were analyzed using the verdicts given by the automatic assessment tool: correct solutions or incorrect solutions. In the case of incorrect solutions, the types of programming errors were studied. Additionally, interaction was also studied by analyzing the students’ success rate. This rate is the percentage of correct solutions among the total number of attempts (correct and incorrect). Moreover, the relationship between success rate and academic performance was analyzed. Furthermore, this research examines the students’ perceptions toward the assessment tool through interviews. The results of this study help understanding the benefits and perceptions of the students with respect to the use of an automatic assessment tool in a computer programming course.