Previous studies have suggested that vitamin D is associated with cardiovascular disease (CVD), however, the relationship between vitamin D levels and CVD risk is still unclear. The purpose of this study was to assess the relationship of serum concentration of 25-hydroxyvitamin D (25(OH)D) with CVD in rural residents of Henan province of China. Basic information and medical history were gathered through face-to-face surveys from July 2013 to August 2015, and biochemical indicators were gathered in a laboratory setting. Logistic and restricted cubic splines regression analyses were used to estimate odd ratios (ORs) and 95% confidence intervals (95%CI) of CVD. A total of 1078 participants were included, the mean serum 25(OH)D concentration was determined to be 25 ± 18 ng/ml, with 54.45% of the participants presenting vitamin D deficiency [25(OH)D < 20 ng/mL]. Moreover, the prevalence of CVD was 59.28% in the vitamin D deficient group, which was higher than in the insufficient (48.55%) and sufficient groups (52.78%). After adjusting for potential confounders, compared with the deficient group, the ORs (95%CI) of CVDs were 0.68 (0.50, 0.91) in the insufficient group and 0.81 (0.56, 1.16) in the sufficient group. A nonlinear (U-shaped) association was observed between the risk of CVD and 25(OH)D concentration. Further research suggested that the risk of CVD was higher in males than in females. In conclusion, a U-shape association between serum levels of 25(OH)D and the risk of CVD was identified in our study, suggesting a nonlinear relationship between vitamin D with the prevalence of CVD.