EMeD-Part: An Efficient Methodology for Horizontal Partitioning in Data Warehouses

Nowadays, data warehouses store Peta-bytes of data. Queries defined on data warehouses are generally complex. Several techniques are used for optimizing queries in data warehouses such as indexes, partitioning and materialized views. Selecting the best configuration of indexes, or partitions or materialized views are all NP-hard. Here, we focus on the horizontal partitioning problem in data warehouses. Several approaches were proposed for solving horizontal partitioning problem in data warehouses including genetic algorithms using a small set of query workload in general. We present a new methodology based on data mining and particle swarm optimization for solving the horizontal partitioning problem in data warehouses using relatively large query workload. First, we compute attraction between predicates followed by a hierarchical clustering of predicates. In the second step, we use discrete particle swarm optimization for selecting the best partitioning schema. Several experiments are performed to demonstrate the effectiveness of the proposed approach and the results are compared to the best well known method so far, the genetic algorithm based approach. The proposed approach is found to be faster and more effective than the genetic algorithm based approach for solving the data warehouse horizontal partitioning.