The azimuthal anisotropy of charged particles produced in $\sqrt{s_{\mathrm{NN}}}=8.16$ TeV $p$+Pb collisions is measured with the ATLAS detector at the LHC. The data correspond to an integrated luminosity of $165$ $\mathrm{nb}^{-1}$ that was collected in 2016. Azimuthal anisotropy coefficients, elliptic $v_2$ and triangular $v_3$, extracted using two-particle correlations with a non-flow template fit procedure, are presented as a function of particle transverse momentum ($p_\mathrm{T}$) between $0.5$ and $50$ GeV. The $v_2$ results are also reported as a function of centrality in three different particle $p_\mathrm{T}$ intervals. The results are reported from minimum-bias events and jet-triggered events, where two jet $p_\mathrm{T}$ thresholds are used. The anisotropies for particles with $p_\mathrm{T}$ less than about $2$ GeV are consistent with hydrodynamic flow expectations, while the significant non-zero anisotropies for $p_\mathrm{T}$ in the range $9$-$50$ GeV are not explained within current theoretical frameworks. In the $p_\mathrm{T}$ range $2$-$9$ GeV, the anisotropies are larger in minimum-bias than in jet-triggered events. Possible origins of these effects, such as the changing admixture of particles from hard scattering and the underlying event, are discussed.