A search for resonances in the mass range 12-70 GeV produced in association with a b quark jet and a second jet, and decaying to a muon pair, is reported. The analysis is based on data from proton-proton collisions at center-of-mass energies of 8 and 13 TeV, collected with the CMS detector at the LHC and corresponding to integrated luminosities of 19.7 and 35.9 fb$^{-1}$, respectively. The search is carried out in two mutually exclusive event categories. Events in the first category are required to have a b quark jet in the central region ($|\eta|\le$ 2.4) and at least one jet in the forward region ($|\eta|>$ 2.4). Events in the second category are required to have two jets in the central region, at least one of which is identified as a b quark jet, no jets in the forward region, and low missing transverse momentum. An excess of events above the background near a dimuon mass of 28 GeV is observed in the 8 TeV data, corresponding to local significances of 4.2 and 2.9 standard deviations for the first and second event categories, respectively. A similar analysis conducted with the 13 TeV data results in a mild excess over the background in the first event category corresponding to a local significance of 2.0 standard deviations, while the second category results in a 1.4 standard deviation deficit. The fiducial cross section measurements and 95% confidence level upper limits on those for a resonance consistent with the 8 TeV excess are provided at both collision energies.