Laser Induced Breakdown Spectroscopy (LIBS) is attracting more and more attention in geology fields because of its unique advantages of on-line and in situ analysis and the availability of portable even handheld instruments due to the development of laser sources and mini-spectrometers. However, parameters such as accuracy and precision of the instrument are still essential for field application. In this paper, two algorithms to determine the concentrations of five main elements (Si, Ca, Mg, Fe and Al) in sedimentary rock samples are proposed based on support vector regression (SVR) and partial least squares regression (PLSR). The proposed comparison demonstrates that the SVR model performed better with more satisfactory accuracy and precision under the optimized conditions. For SVR quantitative analysis, the spectral features (20 lines) without principal component analysis (PCA) were selected as the input variables. The optimized penalty parameter C and the key parameter of the radial basis function (RBF)-σ obtained by genetic algorithm (GA) were 4.63 and 0.9159, respectively. Also, the best number of the principal components of PLSR was optimized to be 8 by 10-fold cross-validation (CV) testing. Furthermore, the accuracy corresponding to the average relative standard deviations (RSDs) and the precision related to the root mean square error (RMSE) were calculated according to the performance of the two regression models. A significant enhancement of accuracy, of up to 43.50 times, and of precision, of 7.19 times, for the SVR model was obtained, which can eliminate the self-absorption of plasma efficiently compared with the linear machine learning method PLSR. In conclusion, the chemometric method of SVR with better accuracy and precision can be successfully applied for the quantitative analysis of complex geological samples using the LIBS technique.