Results of a search for gluino pair production with subsequent R-parity-violating decays to quarks are presented. This search uses 36.1 fb$^{-1}$ of data collected by the ATLAS detector in proton-proton collisions with a center-of-mass energy of $\sqrt{s} = 13$ TeV at the LHC. The analysis is performed using requirements on the number of jets and the number of jets tagged as containing a $b$-hadron as well as a topological observable formed by the scalar sum of masses of large-radius jets in the event. No significant excess above the expected Standard Model background is observed. Limits are set on the production of gluinos in models with the R-parity-violating decays of either the gluino itself (direct decay) or the neutralino produced in the R-parity-conserving gluino decay (cascade decay). In the gluino cascade decay model, gluinos with masses below 1850 GeV are excluded for 1000 GeV neutralino mass. For the gluino direct decay model, the 95% confidence level upper limit on the cross section times branching ratio varies between 0.80 fb at $m_{\tilde{g}}$ = 900 GeV and 0.011 fb at $m_{\tilde{g}}$ = 1800 GeV.