Emotion recognition is a challenging research problem with a significant scientific interest. Most of the emotion assessment studies have focused on the analysis of facial expressions. Recently, it has been shown that the simultaneous use of several biosignals taken from the patient may improve the classification accuracy. An open problem in this area is to identify which biosignals are more relevant for emotion recognition. In this paper, we perform Recursive Feature Elimination (RFE) to select a subset of features that allows emotion classification. Experiments are carried out over a multimodal database with arousal and valence annotations, and a diverse range of features extracted from physiological, neurophysiological, and video signals. Results show that several features can be eliminated while still preserving classification accuracy in setups of 2 and 3 classes. Using a small subset of the features, it is possible to reach 70% accuracy for arousal and 60% accuracy for valence in some experiments. Experimentally, it is shown that the Galvanic Skin Response (GSR) is relevant for arousal classification, while the electroencephalogram (EEG) is relevant for valence.