We present measurements of the atmospheric depth of the shower maximum $X_\mathrm{max}$, inferred for the first time on an event-by-event level using the Surface Detector of the Pierre Auger Observatory. Using deep learning, we were able to extend measurements of the $X_\mathrm{max}$ distributions up to energies of 100 EeV ($10^{20}$ eV), not yet revealed by current measurements, providing new insights into the mass composition of cosmic rays at extreme energies. Gaining a 10-fold increase in statistics compared to the Fluorescence Detector data, we find evidence that the rate of change of the average $X_\mathrm{max}$ with the logarithm of energy features three breaks at $6.5\pm0.6~(\mathrm{stat})\pm1~(\mathrm{sys})$ EeV, $11\pm 2~(\mathrm{stat})\pm1~(\mathrm{sys})$ EeV, and $31\pm5~(\mathrm{stat})\pm3~(\mathrm{sys})$ EeV, in the vicinity to the three prominent features (ankle, instep, suppression) of the cosmic-ray flux. The energy evolution of the mean and standard deviation of the measured $X_\mathrm{max}$ distributions indicates that the mass composition becomes increasingly heavier and purer, thus being incompatible with a large fraction of light nuclei between 50 EeV and 100 EeV.