Digital Lensless Holographic Microscopy (DLHM) is a phase imaging modality that omits the use of lenses or other bulky hardware to recover information from microscopic objects. Deep learning models have been recently used to substitute traditional DLHM reconstruction algorithms and classify samples from the reconstructed amplitude and phase images. In this work, we have investigated using these models to classify diatom samples, circumventing the whole reconstruction process altogether. We have validated our approach using a simulated DLHM dataset by comparing the performance of three typical image-processing learning-based models: AlexNet, VGG16, and ResNet-18.