Single-pixel camera (SPC) is a low-cost compressive imaging architecture that obtains random projections of the scenes using binary coded apertures. After the acquisitions, image reconstructions are usually obtained by nonlinear and relatively expensive optimization-based algorithms. Recent works have focused on designing the binary coded apertures to improve the speed of the reconstruction algorithms and the sampling complexity of the compressed sensing systems. However, it has been shown that image recovery is not necessary for image classification from compressive measurements, where only specific features of the images are required. This work proposes a deep learning approach for image classification directly from SPC measurements. In this approach, a neural network is trained to simultaneously learn the linear binary sensing matrix and the non-linear classification parameters, considering the constraints imposed by the SPC. Specifically, the first layer learns the sensing matrix, and subsequent layers perform the classification directly on the compressed measurements. Simulation results from two image datasets validate the proposed method, which provides the best classification accuracy along with a binary sensing matrix.