Colorectal cancer is one of the leading death-related diseases worldwide, usually induced by a multifactorial and complex process, including genetic and epigenetic abnormalities and the impact of diet and lifestyle. In the present study, we evaluated the biological impact of two of the main coffee polyphenols, chlorogenic acid (CGA) and caffeic acid (CA), as well as two polyphenol-rich coffee extracts (green coffee extract and toasted coffee Extract) against SW480 and SW620 colorectal cancer cells. First, the total phenolic content and the antioxidant capability of the extracts were determined. Then, cytotoxicity was evaluated by MTT and SBR. Finally, a wound healing assay was performed to determine the impact on the cell migration process. The results showed a cytotoxic effect of all treatments in a time and dose-dependent manner, which decreased the viability in both cell lines at 24 h and 48 h; likewise, the migration capability of cells decreased with low doses of treatments. These results suggest the potential of coffee to modulate biological mechanisms involved in colorectal cancer development; however, more studies are required to understand the mechanistic insights of these observations.