This paper presents the electron and photon energy calibration obtained with the ATLAS detector using about 36 fb$^{-1}$ of LHC proton-proton collision data recorded at $\sqrt{s}=13$ TeV in 2015 and 2016. The different calibration steps applied to the data and the optimization of the reconstruction of electron and photon energies are discussed. The absolute energy scale is set using a large sample of $Z$ boson decays into electron--positron pairs. The systematic uncertainty in the energy scale calibration varies between 0.03% to 0.2% in most of the detector acceptance for electrons with transverse momentum close to 45 GeV. For electrons with transverse momentum of 10 GeV the typical uncertainty is 0.3% to 0.8% and it varies between 0.25% and 1% for photons with transverse momentum around 60 GeV. Validations of the energy calibration with $J/\psi \rightarrow e^+e^-$ decays and radiative $Z$ boson decays are also presented.