Despite the benefits of mammography investigations, some studies have shown that X-ray exposure from the mammography screening itself can statistically cause breast cancer in a small fraction of women. Therefore, a dose reduction in mammography is desirable. At the same time, there is a demand for a higher spatial resolution in mammographic imaging. The most promising way to achieve these goals is the use of advanced photon-processing semiconductor X-ray detectors with optimum sensor materials. This study addresses the investigation of the optimum semiconductor sensor material for mammography in combination with the photon-processing detector Medipix3RX. The influence of K-shell fluorescence from the sensor material on the achievable contrast-to-noise ratio is investigated, as well as the attenuation efficiency. The three different sensor materials, CdTe, GaAs, and Si are studied, showing advances of CdTe-sensors for mammography. Furthermore, a comparison of the contrast-to-noise ratio between a clinical Se-detector and Medipix3RX detectors with Si- and CdTe-sensors is shown using a self-produced mammography phantom that is based on real human tissue.