The excitation energy transfer dynamics in the Fenna-Matthews-Olson complex is quantified in terms of a non-Markovianity measure based on the time evolution of the trace distance of two quantum states. We use a system description derived from experiments and different environmental fluctuation spectral functions, which are obtained either from experimental data or from molecular dynamics simulations. These exhibit, in all cases, a nontrivial structure with several peaks attributed to vibrational modes of the pigment-protein complex. Such a structured environmental spectrum can, in principle, give rise to strong non-Markovian effects. We present numerically exact real-time path-integral calculations for the transfer dynamics and find, in all cases, a monotonic decrease of the trace distance with increasing time which renders a Markovian description valid.