A new optical security system for image encryption based on a nonlinear joint transform correlator (JTC) and Fresnel transform is proposed. A lensless optical encryption system, which is a simplified version of previous JTCbased encryption systems, produces a complex distribution in the Fresnel domain, whose intensity is captured by a power-law sensor and nonlinearly modified to yield the real-valued encrypted image. The nonlinearity plays an essential role in the decryption system which, in turn, contains an optical fractional Fourier transform to retrieve the primary image. The security system proposed in this work is a novel extension of the conventional JTC-based encryption system from the Fourier domain to the Fresnel domain. Additional advantages of the proposed system are: simplification of the optical setup, alleviated alignment requirements, and an additional key (the propagation distance) that improves security. Numerical simulations verify the validity of this new optical security system.