Spatial filtering techniques are used in the analysis of interferograms and off-axis digital holograms to obtain the phase information from an optical field. The masks applied for the selection of the virtual image order in the frequency space usually have regular shapes and are located by hand. Therefore, they create artifacts that hide some details in the obtained phase, especially when holograms from objects with sharp edges are reconstructed. In this work, a novel algorithm that automatically calculates and locates the mask separating the spectral orders is presented. This new method uses a distance criterion between the maximum values in the amplitude spectrum as a clustering parameter. The values for the distance parameter are changed and the results are analyzed for a simulated image-plane hologram. As an example of the algorithm application, a digital hologram obtained from one USAF-1951 test target is reconstructed and the phase of the test target element is obtained.