Accuracy, robustness and efficiency are three general requirements for the online visual tracking. Although the correlation based trackers achieved competitive results on both accuracy and efficiency, they still suffer from time-consuming scale searching and are prone to drifting. In this paper, an Extreme Learning Machine (ELM) augmented adaptive correlation filter is proposed to address these issues. We enhance the correlation based tracker in four aspects. Firstly, a keypoints tracking based scale estimation method is proposed to handle the target scale variation during tracking. Secondly, the powerful and efficient features including intensity, Fast Histogram of Oriented Gradient (FHOG) and Discriminative color Descriptors (DD) are integrated to further boost the tracking performance. Thirdly, dynamic updating mechanism is introduced to better adapt the correlation filter to the appearance changes. Moreover, to handle the drift brought by severe occlusion, we train an online ELM based classifier to re-initialize object in case of tracking failure. Extensive experimental results on the Object Tracking Benchmark (OTB) dataset show that our proposed tracker performs favorably against state-of-the-art methods in terms of accuracy, efficiency, and robustness.