Full visual scene understanding has always been one of the main goals of machine perception. The ability to describe the components of a scene using only information taken by a digital camera has been the main focus of computer vision tasks such as semantic segmentation and instance segmentation. Where by using Deep Learning techniques, a neural network is capable to assign a label to each pixel of an image (semantic segmentation) or define the boundaries of an instance or object with more precision than a bounding box (instance segmentation). The task of Panoptic Segmentation tries to achieve a full scene description by merging semantic and instance segmentation information and leveraging the strengths of these two tasks. On this report it is shown a possible alternative to solve this merging problem by using Convolutional Neural Networks (CNNs) to refine the boundaries between each class.