Colorectal cancer is the third most incidence cancer world-around. Colonoscopies are the most effective resource to detect and segment abnormal polyp masses, considered as the main biomarker of this cancer. Nonetheless, some recent clinical studies have revealed a polyp miss rate up to 26% during the clinical routine. Also, the expert bias introduced during polyp shape characterization may induce to false-negative diagnosis. Current computational approaches have supported polyp segmentation but over controlled scenarios, where polyp frames have been labeled by an expert. These supervised representations are fully dependent of well-segmented polyps, in crop sequences that always report these masses. This work introduces an attention receptive field mechanism, that robustly recover the polyp shape, by learning non-local pixel relationship. Besides this deep representation is learning from a weakly supervised scheme that includes unlabeled background frames, to discriminate polyps from near structures like intestinal folds. The achieved results outperform state-of-the-art approaches achieving a 95.1% precision in the public CVC-Colon DB, with also competitive performance on other datasets. Clinical relevance—The work address a novel strategy to support segmentation tools in a clinical routine with redundant background over colonoscopy sequences.