Often, it is necessary to construct training sets. If we have only a small number of tagged objects and a large group of unlabeled objects, we can build the training set simulating a data stream of unlabelled objects from which it is necessary to learn and to incorporate them to the training set later. In order to prevent deterioration of the training set obtained, in this work we propose a scheme that takes into account the concept drift, since in many situations the distribution of classes may change over time. To classify the unlabelled objects we have used an ensemble of classifiers and we propose a strategy to detect the noise after the classification process.