A search for the pair production of heavy fermionic partners of the top quark with charge 5/3 (X$_{5/3}$) is performed in proton-proton collisions at a center-of-mass energy of 13 TeV with the CMS detector at the CERN LHC. The data sample analyzed corresponds to an integrated luminosity of 35.9 fb$^{-1}$. The X$_{5/3}$ quark is assumed always to decay into a top quark and a W boson. Both the right-handed and left-handed X$_{5/3}$ couplings to the W boson are considered. Final states with either a pair of same-sign leptons or a single lepton are studied. No significant excess of events is observed above the expected standard model background. Lower limits at 95% confidence level on the X$_{5/3}$ quark mass are set at 1.33 and 1.30 TeV respectively for the case of right-handed and left-handed couplings to W bosons in a combination of the same-sign dilepton and single-lepton final states.