We propose a method that uses the visibility feature extraction module and support vector machine (VFEM SEM) classification algorithm for visibility estimation. First, five statistical image features are extracted to fit a multivariate Gaussian (MVG) model; then, the visibility indexes and Mahalanobis distance are calculated from the MVG model and the reference samples, which are selected from the highest and lowest visibility levels, respectively. Finally, the SVM classifier is trained for visibility classification. Due to the lack of real scene foggy datasets, we collect a dataset called foggy highway from highway surveillance scenes. To evaluate the proposed algorithm’s performance fairly and objectively, we synthesize the foggy virtual KITTI image dataset with continuous annotations using the atmospheric scattering model and use another public synthetic dataset called “foggy road sign images.” Comprehensive experiments are conducted under three datasets. The experimental results show that the proposed method has achieved better classification accuracy compared with the other traditional image processing methods. Moreover, compared with deep learning-based methods, the proposed method has also achieved competitive performance and requires much less computational resources and image data. All of these advantages make the proposed algorithm more suitable for mobile device deployment.