This paper presents a novel solution toward the accurate and automatic cartilage segmentation with multi-contrast MR images based on pixel classification. The previous pixel classification based works for cartilage segmentation only rely on the labeling by a trained classifier, such as support vector machines (SVM) or k-nearest neighbors. However, these frameworks do not consider the spatial information. To incorporate spatial dependencies in pixel classification, we explore a principled framework of pixel classification based on the convex optimization of an SVM-based association potential and a discriminative random fields (DRF) based interaction potential for our task of cartilage segmentation. The local image structure based features as well as the features based on geometrical information are adopted as the features. We finally perform the loopy belief propagation inference algorithm to find the optimal label configuration. Our framework is validated on a dataset of multi-contrast MR images. Experimental results show that the combined features compare favorably to the two types of separate features and our pixel classification framework outperforms the conventional frameworks based solely on SVM or DRF for cartilage segmentation in subject-specific training scenario.