Abstract In this paper, the fractional Fourier operators, the random phase masks and the nonlinear operations of amplitude and phase truncations are utilized to encrypt and decrypt images. We use the following fractional Fourier operators in the image encryption-decryption system: the fractional Fourier transform, the fractional traslation and the fractional correlation. The proposed encryption system uses nonlinear operations, such as phase encoding and truncation operations, in order to increase the security of the encrypted image. The encryption-decryption system has the following security keys: one fractional order of the fractional Fourier transform, two random phase masks and two pseudorandom code images. When all the proper security keys are used in the decryption system, the obtained decrypted image is a replica of the image to encrypt.