Anisotropies in the initial energy density distribution of the quark-gluon plasma created in high energy heavy ion collisions lead to anisotropies in the azimuthal distributions of the final-state particles known as collective flow. Fourier harmonic decomposition is used to quantify these anisotropies. The higher-order harmonics can be induced by the same order anisotropies (linear response) or by the combined influence of several lower order anisotropies (nonlinear response) in the initial state. The mixed higher-order anisotropic flow and nonlinear response coefficients of charged particles are measured as functions of transverse momentum and centrality in PbPb collisions at nucleon-nucleon center-of-mass energies $\sqrt{s_\mathrm{NN}} =$ 2.76 and 5.02 TeV with the CMS detector. The results are compared with viscous hydrodynamic calculations using several different initial conditions, as well as microscopic transport model calculations. None of the models provides a simultaneous description of the mixed higher-order flow harmonics and nonlinear response coefficients.