We propose for the first time, to the best of our knowledge, the use of optimized random phases (ORAPs) in a double random phase encryption scheme (DRPE). In DRPE schemes the convolution between two random phase functions encrypts the information to be secured. However, in actual encryption applications, this convolution of random phases also results in unwanted effects like speckle noise. In this Letter we show that under certain conditions this noise can be drastically reduced. These conditions can be easily achieved by using ORAPs. These ORAPs, besides containing information about the parameters of the optical system and maintaining all the security properties of a random phase function, ensure that the encrypted data is a phase-only function. This leads to a great increase in system performance, with decryption quality similar to the reconstruction of a phase-only hologram generated with the Gerchberg-Saxton algorithm. We show both numerical and experimental results confirming the validity of our proposal.