Emotion recognition has become a hot research topic in the field of human-computer interaction (HCI), while the recognition accuracy is still not adequate for real applications. In this paper, a new emotion recognition framework based on multi-channel physiological signals including ECG, EMG and SCL using the dataset of Bio Vid Emo DB was proposed. A series of feature selection methods and fusion methods had been evaluated, through which wavelet transform features and SVM classifier were adopted. An improved accuracy of 94.81% was achieved by fusing the classifiers of ECG and EMG, which was adequate for real applications and better than relevant studies.