(1) Background: Due to the pandemic caused by COVID-19, mandatory confinement was declared, which generated a decrease in the practice of physical activity (PA). Based on this problem, it was proposed to study the associations between PA in relation to depressive symptoms, quality of sleep, and the quality of life of middle-aged people who work in the university context during compulsory confinement as a result of COVID-19. (2) Methods: A total of 336 middle-aged people (48 ± 6.67) participated in this analytical cross-sectional study. The variable levels of PA, quality of sleep, symptoms of depression, and quality of life were measured with the International Physical Activity Questionary (IPAQ), the Pittsburgh Sleep Quality Index (PSQI), the Zung Self-Rating Depression Scale (ZSDS), and the SF-12v2 questionnaire, respectively. (3) Results: A logistic regression model was used to analyze the relationships between the level of PA and depressive symptoms (OR = 2.053), total sleep duration (OR = 0.495), sleep disturbances (OR = 2.414), quality of sleep (OR = 2.471), use of sleep medication (OR = 0.348), daytime dysfunction (OR = 1.809), general health (OR = 0.949), and physical functioning (OR = 0.987). (4) Conclusions: In middle-aged people, during compulsory confinement, being insufficiently active is a risk for depressive symptoms and disturbances in sleep quality.