A search for long-lived particles decaying into an oppositely charged lepton pair, $\mu\mu$, $ee$, or $e\mu$, is presented using 32.8 fb$^{-1}$ of $pp$ collision data collected at $\sqrt{s}=13$ TeV by the ATLAS detector at the LHC. Candidate leptons are required to form a vertex, within the inner tracking volume of ATLAS, displaced from the primary $pp$ interaction region. No lepton pairs with an invariant mass greater than 12 GeV are observed, consistent with the background expectations derived from data. The detection efficiencies for generic resonances with lifetimes ($c\tau$) of 100-1000 mm decaying into a dilepton pair with masses between 0.1-1.0 TeV are presented as a function of $p_T$ and decay radius of the resonances to allow the extraction of upper limits on the cross sections for theoretical models. The result is also interpreted in a supersymmetric model in which the lightest neutralino, produced via squark-antisquark production, decays into $\ell^{+}\ell^{'-}\nu$ ($\ell, \ell^{'} = e$, $\mu$) with a finite lifetime due to the presence of R-parity violating couplings. Cross-section limits are presented for specific squark and neutralino masses. For a 700 GeV squark, neutralinos with masses of 50-500 GeV and mean proper lifetimes corresponding to $c\tau$ values between 1 mm to 6 m are excluded. For a 1.6 TeV squark, $c\tau$ values between 3 mm to 1 m are excluded for 1.3 TeV neutralinos.