Utilization of Color-depth Combination Features and Multi-level Refinement CNN for Upper-limb Posture Recognition

Upper-limb posture recognition is of great value to rehabilitation and assessment of stroke patients. In this paper, we propose a novel method for upper-limb posture recognition. Convolutional neural network (CNN) cascade is applied to reduce the training difficulty of the algorithm. Information of depth and color is combined to eliminate the influence of complex background and illumination variation. Kinect is used to automatically acquire a large number of upper limb posture labels. The principle of coarse-to-fine runs through the whole algorithm. The overall network architecture consists of 3 levels with a total of six CNNs. First of all, color and depth images are aligned to obtain a RGB-D quad channels image. The quad-channel image is sent to level-1 cascade network to obtain a bounding box containing the upper limb cropped from the entire body. Then, the resulting bounding box is brought into level-2 cascade network and 4 sets of rough upper-limb joints coordinates are obtained. Finally, zoom in the visual field to local area of 4 key points, 4 sets of accurate coordinate is obtained by level-3 cascade network. Experimental results show that upper-limb posture is calculated by the proposed algorithm that has strong stability to both illumination and background problems.