Abstract We use the fractional convolution, nonlinear operations and random phase masks (RPMs) in order to encrypt images. The definition used in this paper of the fractional convolution is based on the fractional Fourier transform (FrFT). The amplitude and phase truncation are nonlinear operations that allow to select a specific information of a complex-valued image. The encryption and decryption systems are based on the double random phase encoding (DRPE). At the beginning of the encryption process, the image to encrypt is encoded in phase. The rest of the encryption and decryption process use two RPMs, the fractional convolution and the amplitude and phase truncation operations in a sequential way in order to get the encrypted and decrypted image. The amplitude and phase truncation and the phase encoding are nonlinear operations that improving the security of the encrypted image, because the obtained key space of the proposed encryption process is very larger. The fractional order of the FrFT introduces a new key for the security system and the encrypted image of the proposed encryption system is a real-valued image. The encryption-decryption system has five security keys. All these five keys with their correct values are necessary in the decryption process with the purpose of obtaining the original image that was encrypted in the encryption process.