Measurements of fragmentation functions for jets associated with an isolated photon are presented for the first time in pp and PbPb collisions. The analysis uses data collected with the CMS detector at the CERN LHC at a nucleon-nucleon center-of-mass energy of 5.02 TeV. Fragmentation functions are obtained for jets with p$_\mathrm{T}^\text{jet} >$ 30 GeV in events containing an isolated photon with p$_\mathrm{T}^\gamma>$ 60 GeV, using charged tracks with transverse momentum p$_\mathrm{T}^\text{trk} >$ 1 GeV in a cone around the jet axis. The association with an isolated photon constrains the initial p$_\mathrm{T}$ and azimuthal angle of the parton whose shower produced the jet. For central PbPb collisions, modifications of the jet fragmentation functions are observed when compared to those measured in pp collisions, while no significant differences are found in the 50% most peripheral collisions. Jets in central PbPb events show an excess (depletion) of low (high) p$_\mathrm{T}$ particles, with a transition around 3 GeV. This measurement shows for the first time the in-medium shower modifications of partons (quark dominated) with well defined initial kinematics. It constitutes a new well-controlled reference for testing theoretical models of the parton passage through the QGP.