By encoding two 16-dimensional qudits in the time and frequency degrees of freedom of a heralded single photon, we realize a deterministic photonic two-qudit SUM gate operating on a 256-dimensional Hilbert space.