Polyps are the main biomarker to diagnose colorectal cancer. These polyps are protuberant masses that namely appear around intestinal tract. Currently, the automatic polyp detection is a challenging task because the high shape and appearance variability. Additionally, the videos captured during colonoscopies are namely distorted because the clinical procedure, presenting strong camera motions and illumination changes. This work introduces an automatic strategy to detect polyps by using a pixel level characterization of orientation and curvatures, which are coded using a dense Hough transform. Because the complexity of polyp modelling, an additional appearance model was herein implemented, allowing to filter and to find potential polyp regions. The proposed strategy was evaluated over real sequences of ASU-Mayo Clinic Colonoscopy Video dataset, reporting in average 81% of precision on polyp detection task. This strategy is also able to online track polyps in long sequences, requiring only a first frame delineation.