Rock mineralogy is critical in determining the adhesion quality and moisture susceptibility of asphalt-aggregate systems in asphalt mixtures. This work aims at identifying the chemical properties of aggregates that influence the quality and durability of these systems. To accomplish this goal, three lithologies with an ample mineralogical and chemical spectrum (i.e. quartzite, granodiorite and serpentinite) were selected. Asphalt-aggregate specimens fabricated with these lithologies were tested using a new pull-off experiment in dry condition, and after 1, 3 and 7 days of being submerged in water. The results show that rocks of mafic composition (e.g. serpentinite) have better resistance to moisture damage than felsic rocks (e.g. quartzite and granodiorite), and that certain specific aggregate oxides promote or prevent moisture degradation in these systems.