We present the first measurement of the inclusive three-jet differential cross section as a function of the invariant mass of the three jets with the largest transverse momenta in an event in pp¯ collisions at s=1.96TeV. The measurement is made in different rapidity regions and for different jet transverse momentum requirements and is based on a data set corresponding to an integrated luminosity of 0.7fb−1 collected with the D0 detector at the Fermilab Tevatron Collider. The results are used to test the three-jet matrix elements in perturbative QCD calculations at next-to-leading order in the strong coupling constant. The data allow discrimination between parametrizations of the parton distribution functions of the proton.