Abstract This article proposes the design of a hybrid transmission system using wavelength division multiplexing and subcarrier under the frequency frame for 5G. The proposal is evaluated through modeling based on simulation of the generation of optical carriers using a broadband source in combination with an optical filter configured with different pass bands and transmission at different distances. The simulation model evaluated the signal quality levels in terms of the Q factor, taking into account the frequency framework for 5G (5 GHz, 30 GHz and 60 GHz) and the optical fiber length (5 km, 10 km, 15 km, 20 km, 25 km, 30 km). The results obtained show that the best performance of the Q factor compared to distances was that of the 30 GHz frequency, both in the access network and in the metro network. It was also possible to demonstrate that through a broadband light source and a waveguide grating array it is possible to obtain optical carriers for the transmission of signals in a hybrid wavelength division multiplexing and subcarrier transmission system.