A study is presented of anomalous $HVV$ interactions of the Higgs boson, including its $CP$ properties. The study uses Higgs boson candidates produced mainly in vector boson fusion and gluon fusion that subsequently decay to a pair of $\ensuremath{\tau}$ leptons. The data were recorded by the CMS experiment at the LHC in 2016 at a center-of-mass energy of 13 TeV and correspond to an integrated luminosity of $35.9\text{ }\text{ }{\mathrm{fb}}^{\ensuremath{-}1}$. A matrix element technique is employed for the analysis of anomalous interactions. The results are combined with those from the $H\ensuremath{\rightarrow}4\ensuremath{\ell}$ decay channel presented earlier, yielding the most stringent constraints on anomalous Higgs boson couplings to electroweak vector bosons expressed as effective cross section fractions and phases: the $CP$-violating parameter ${f}_{a3}\mathrm{cos}({\ensuremath{\phi}}_{a3})=(0.00\ifmmode\pm\else\textpm\fi{}0.27)\ifmmode\times\else\texttimes\fi{}{10}^{\ensuremath{-}3}$ and the $CP$-conserving parameters ${f}_{a2}\mathrm{cos}({\ensuremath{\phi}}_{a2})=(0.0{8}_{\ensuremath{-}0.21}^{+1.04})\ifmmode\times\else\texttimes\fi{}{10}^{\ensuremath{-}3}$, ${f}_{\mathrm{\ensuremath{\Lambda}}1}\mathrm{cos}({\ensuremath{\phi}}_{\mathrm{\ensuremath{\Lambda}}1})=(0.0{0}_{\ensuremath{-}0.09}^{+0.53})\ifmmode\times\else\texttimes\fi{}{10}^{\ensuremath{-}3}$, and ${f}_{\mathrm{\ensuremath{\Lambda}}1}^{Z\ensuremath{\gamma}}\mathrm{cos}({\ensuremath{\phi}}_{\mathrm{\ensuremath{\Lambda}}1}^{Z\ensuremath{\gamma}})=(0.{0}_{\ensuremath{-}1.3}^{+1.1})\ifmmode\times\else\texttimes\fi{}{10}^{\ensuremath{-}3}$. The current dataset does not allow for precise constraints on $CP$ properties in the gluon fusion process. The results are consistent with standard model expectations.