A search for non-resonant Higgs boson pair production, as predicted by the Standard Model, is presented, where one of the Higgs bosons decays via the $H\rightarrow bb$ channel and the other via one of the $H \rightarrow WW^*/ZZ^*/ττ$ channels. The analysis selection requires events to have at least two $b$-tagged jets and exactly two leptons (electrons or muons) with opposite electric charge in the final state. Candidate events consistent with Higgs boson pair production are selected using a multi-class neural network discriminant. The analysis uses 139 fb$^{-1}$ of $pp$ collision data recorded at a centre-of-mass energy of 13 TeV by the ATLAS detector at the Large Hadron Collider. An observed (expected) upper limit of 1.2 ($0.9^{+0.4}_{-0.3}$) pb is set on the non-resonant Higgs boson pair production cross-section at 95% confidence level, which is equivalent to 40 ($29^{+14}_{-9}$) times the value predicted in the Standard Model.