A New Approach for Missing Data Imputation in Big Data Interface

Abstract

The three-stage approach for missing data imputation in Big data interface is proposed in the paper. The first stage includes designing the Big data model in the task of missing data recovery, which enables to process the structured and semistructured data. The next stage is developing the method of missing data recovery based on functional dependencies and association rules. The estimating the algorithm complexity for missing data recovery is provided at the last stage. The proposed method of missing data recovery creates additional data values using a based domain and functional dependencies and adds these values in available training data. The performing the analysis of data different types is possible too. The correctness of the imputed values is verified on the classifier built on the original dataset. The proposed method performs in 12% better than the EM and RF methods for 30% missing data and enables the parallel execution in distributed databases. The acceleration for m=41 attributes is larger in 12.5 times for 20 servers (processors) comparing with the non-parallel modeю