In this paper we consider the application of the integral transformations for image encryption through optical systems, a mathematical algorithm under Matlab platform using fractional Fourier transform (FrFT) and Random Phase Mask (RPM) for digital images encryption is implemented. The FrFT can be related to others integral transforms, such as: Fourier transform, Sine and Cosine transforms, Radial Hilbert transform, fractional Sine transform, fractional Cosine transform, fractional Hartley transform, fractional Wavelet transform and Gyrator transform, among other transforms. The encryption scheme is based on the use of the FrFT, the joint transform correlator and two RPMs, which provide security and robustness to the implemented security system. One of the RPMs used during encryption-decryption and the fractional order of the FrFT are the keys to improve security and make the system more resistant against security attacks.