One of the most challenging processes in the rural world is the one related to rural abandonment. Rural abandonment has drawn attention worldwide, particularly in Europe and Asia without neglecting the importance in other regional contexts such as Latin America. Even though the literature about rural abandonment drivers is vast, these pieces of literature do not identify the drivers perceived by the people who live in rural areas. Therefore, in this paper, we search for the understanding of the perceived drivers of rural abandonment in the Colombian coffee region. The specific objectives were: (1) to identify socioeconomic characteristics of the respondents; (2) to analyze local attitudes toward the countryside and farming, and (3) to identify perceived factors influencing rural abandonment. We surveyed 320 households in 12 villages around the city of Manizales (Colombia), and we conducted 27 key informant interviews, and three focal groups. Our results show that the rural population faces several socio-economic constraints; however, despite these constraints, most people are willing to remain in the countryside (85%). We discussed the variables affecting these preferences. Furthermore, we found several drivers of rural abandonment perceived by the sampled population, where they highlight the lack of State support and rural education. We concluded that our results help to deepen into the understanding of the contemporary rural dynamics related to countryside abandonment.