The automatic detection of diseases and gastrointestinal tract anomalies is a challenge for medical experts, affecting patient treatment decisions. Deep learning emerges as a new tool in interpreting medical images for the diagnosis, disease prediction, and clinical treatment analysis. Several works proposed in the literature rely on convolutional neural networks to classify medical images accurately. However, the state-of-the-art methods for gastrointestinal anomalies classification have complex architectures, which require multiple parameters to be trained. Then, there is a scope for developing a light and easily replicable deep-learning-based method that maintains the high precision of more complex models. This work proposes a workflow to classify diseases and anomalies of the gastrointestinal tract using image processing and a transfer learning strategy. Our proposed method is tested on the Kvasir-V2 dataset, containing 8000 endoscopic images divided into eight classes. Our proposed approach achieves more than 98% accuracy during testing by only using the fifth part of trainable parameters compared to the state-of-the-art methods we compare our approach in the experiments.