Modality is a key facet in medical image retrieval, as a user is likely interested in only one of e.g. radiology images, flowcharts, and pathology photos. While assessing image modality is trivial for humans, reliable automatic methods are required to deal with large un-annotated image bases, such as figures taken from the millions of scientific publications. We present a multi-disciplinary approach to tackle the classification problem by combining image features, meta-data, textual and referential information. We test our system's accuracy on the ImageCLEF 2011 medical modality classification data set. We show that using multiple kernel based classification, where the kernels are carefully selected for the different features, significantly increases the classification accuracy. Moreover, we demonstrate that by using linear support vector machine with explicit feature maps [35] of the selected kernels one can achieve comparable results to the (non-linear) kernel based one. Our best method achieves 88.47% accuracy and outperforms the state of the art.