In 2005, Colombia’s Ministry of National Education developed a nationwide program to improve the teaching and learning of English. As one of this project’s initial steps, the Ministry developed two English tests included in the Common European Framework of Reference (CEFR). This paper presents the findings from a study that examined the validity of one of these tests, the ECAES English Exam. Data for this study was gathered using content evaluation sessions with teachers and think-aloud protocols with students. Our findings suggest that this test may not be a valid measure of the students’ general English language proficiency.