Most studies of emotional responses have used unimodal stimuli (e.g., pictures or sounds) or congruent bimodal stimuli (e.g., video clips with sound), but little is known about the emotional response to incongruent bimodal stimuli. The aim of the present study was to evaluate the effect of congruence between auditory and visual bimodal stimuli on heart rate and self-reported measures of emotional dimension, valence and arousal. Subjects listened to pleasant, neutral, and unpleasant sounds, accompanied by videos with and without content congruence, and heart rate was recorded. Dimensions of valence and arousal of each bimodal stimulus were then self-reported. The results showed that heart rate depends of the valence of the sounds but not of the congruence of the bimodal stimuli. The valence and arousal scores changed depending on the congruence of the bimodal stimuli. These results suggest that the congruence of bimodal stimuli affects the subjective perception of emotion.