Phronesis: Efficient Performance Modeling for High-dimensional Configuration Tuning

Spark is a distributed, in-memory computing framework that supports varied applications in data analysis such as machine learning and graph processing [2]. Spark defines a Resilient Distributed Dataset (RDD), an abstraction of distributed memory [73], and provides programmers with composable operators to compute on it. An application defines a sequence of RDD operators and specifies dependencies between them in a directed acyclic graph. Operators are organized into stages, each with a set of parallel tasks that operate on RDD partitions. The Spark run-time system schedules tasks on executors distributed across worker machines. Executors are Java Virtual Machines that execute tasks, which are Java threads, and store the partitions of the RDD.

Applications are sensitive to the Spark configuration. Spark provides more than 100 parameters. Configuration parameters specify computational resources (i.e., CPU time and memory) for software processes distributed over worker machines. Second, workload parameters configure both thread-level (i.e., tasks.cpu) and data-level parallelism (e.g., default.parallelism, files.maxPartitionBytes). Third, parameters configure data compression (e.g., io.*) and serialization (e.g., kyroserializer.*) for transfers through the disk and network. Other parameters (e.g., shuffle.*) determine how data is shuffled between executors to synchronize intermediate results. Finally, memory management parameters (e.g., memory.*) configure the layout of Java virtual machines, which is important when storing large in-memory objects.

Table 1 shows representative parameters of these important categories for performance tuning. The default values of these parameters are often far from optimal for varied applications, and tuning them can improve performance by up to 5\(x\) [72]. Although these parameters are Spark-specific, many of them represent common parameters in other cluster computing frameworks [1, 10, 5, 11].

We make several observations about these parameters. First, these parameters constitute a parameter space of approximately \(10^{25}\) configuration points, presenting challenges in generating heuristic search based on expert knowledge and intuitions. Second, parameters interact in complex ways. The effect of shuffle.file.buffer depends on memory.fraction, which controls the fraction of heap space allocated for computing and temporal storage; default.parallelism depends on executor.core, which controls per-machine parallelism for the application. Identifying and modeling interactions manually is difficult, motivating modeling strategies that discover and account for these interactions automatically.

Third, some important parameters represent trade-offs in resource and performance. For example, tuning driver.cores and executors.cores trades off core resources and computation performance, whereas tuning executor.memory trades off in-memory resources and data-loading performance. Spark users may have different optimization targets that weigh resource and performance differently. For example, users running Spark on private clusters may prefer maximizing performance, whereas users running on public clouds may prefer minimizing cost with a performance constraint. Building accurate performance models enables flexible optimization targets.

Finally, an application’s computational characteristics determine the parameters that affect performance most. Applications that frequently communicate data between machines are affected more by network and shuffle parameters. Applications with massive parallel computations are more sensitive to workload parallelism and CPU resource parameters. Applications that need to store large in-memory objects are more sensitive to the memory parameters. As a result, it is difficult to specify a single set of important parameters for all Spark applications.

Previous research has proposed various machine learning techniques to predict software configuration performance and tune configurations with those predictions. These techniques include tree-based regression models [17, 19, 22, 29, 72], support vector machines [47, 71], multivariate linear regressions [59, 63], and neural networks [39, 46, 64]. Although these works demonstrate the potential of using machine learning for accurate models, they neglect several practical issues.

First, training performance models require a substantial amount of training data. We generate training data by sampling configurations from the space and measuring their performance as the application runs. This procedure is time-consuming as application runs require from minutes to hours. In this setting, reducing the number of configurations for training a performance model is essential. Most works neglect these costs and use simple random sampling to generate configurations for training.

A few works propose more advanced techniques to improve sample efficiency. Latin-Hypercube Sampling can better cover the configuration space, compared with random sampling, with the same number of sampled configurations [17, 19]. However, maximizing coverage of the configuration space does not necessarily identify the most useful configurations for reducing a model’s prediction error. Adaptively selecting configurations that can most improve model accuracy may help, but prior system studies are limited to specific classes of models—linear regression [63] and Delaunay Triangulation [23]—and neither class of models can effectively model performance from high-dimensional configurations.

Second, training requires specifying model hyperparameters, which determine the model’s complexity and impact its prediction accuracy. Prior works are unable to specify hyperparameter values automatically. They rely on either manual tuning [17, 64, 72] or prior knowledge [63] to set hyperparameter values.

In machine learning, error decomposition provides insight in analyzing a performance model’s expected generalization error. Given a workload’s performance model, we can predict execution time \(\hat{y}\) and measure actual execution time \(y\) for Spark configuration \(x\). The expected squared error \(\textrm {E}[(y-\hat{y})^{2}|x]\) can be decomposed as

The first term is noise, the difference between measured and expected performance for a Spark configuration. The second term is variance, the difference between the model’s actual and expected prediction. The third term is bias, the difference between the expected prediction and the expected measurement. Because noise refers to the randomness inherent in the data generation procedure and is non-reducible from the perspective of performance modeling strategy, we focus on how to efficiently reduce bias and variance.

Variance accounts for prediction errors related to randomness in estimated model parameters, which in turn arises from randomness in the training procedure and data collection. First, model training often includes non-deterministic elements. Neural network training randomly initializes a model [28] and samples batch data [24]. Training a tree-based model randomly selects thresholds for splitting a node [35].

Second, randomness arises in system measurements and data collection. To generate a dataset for training, system architects sample randomly from the configuration space and measure each sample’s performance. Models may over-fit to the performance of specific samples and therefore produce high variant predictions when fitted with different training samples.

A formal definition of model variance places a distribution on model parameter \(\theta\) that is conditional on the training data \(\mathbf {X}_{tr},\mathbf {Y}_{tr}\):

For a given training set, we can construct such a distribution by training multiple instances of models with bootstrapped samples from the training data. For a given input \(x\), the prediction variance \(\textrm {Var}_{\theta }(y)\) is the variance of \(y\)’s estimate with respect to the variance in \(\theta\). For example, variance in a simple linear model \(y={W}x\) is given by the variance of linear combinations:

In practice, performance models with higher prediction variance will make less accurate predictions. System architects reduce prediction variance by sampling more configurations for training.

On the other hand, Bias describes errors produced by models despite having been trained with enough data. This type of error represents a model’s inability to capture complex relationships in the data, usually due to simplified model structures. For example, a linear model cannot fit a cubic function regardless of training time and data. Model complexity is often determined by hyperparameters, such as regularization terms and model structure. Neural networks, regression splines, and tree models offer a large space of models defined by neural architectures, polynomial degrees, and tree depth, respectively, as shown in Table 2.

Bias versus Variance. For a given class of models, increasing model structure and complexity leads to a trade-off between higher bias and higher variance. A simpler model (e.g., lower-degree polynomials, fewer interaction terms, shallower neural networks) uses fewer model parameters and produces less variant predictions. But a simpler model’s predictions may not become more accurate as the size of the training dataset increases; its simple structure cannot characterize the complicated interactions among configuration parameters and results in higher prediction bias.

In contrast, a complex model (e.g., higher-degree polynomials, more interaction terms, deeper neural networks) produces better predictions with more model parameters when more training data becomes available. But its predictions might be unstable and sensitive to specific configurations due to over-fitting [4], and this results in higher prediction variance.

Figure 1 compares errors from neural networks that predict performance for Spark’s logistic regression workload. The networks differ in model structure and training dataset size. The smaller network with layer size (16, 8) performs better when the number of sampled configurations is small (<1,000). But it cannot efficiently utilize more training data due to its simpler model structure and larger bias. The larger network suffers from high variance and cannot predict performance accurately when trained with small datasets. However, it gradually outperforms the smaller neural network as dataset size increases beyond 1,000 sampled configurations.

Cross-validation and Generality. \(K\)-fold cross-validation [43] is a popular, successful method for evaluating a model structure’s generalized prediction ability considering bias and variance. Cross-validation divides data into \(K\) groups. For the \(k\)th group, it takes group \(k\) for test data and uses the remaining groups for training data. Cross-validation uses the average test score across the \(k\) groups to estimate the model structure’s generalized prediction ability.

Our experiments show that cross-validation produces different evaluation results when the amount of training data changes. Table 3 shows 10-fold cross-validation scores as model structure, for neural networks and random forests, and the amount of training data vary. When training data is relatively scarce (e.g., 50 to 150 configurations), cross-validation scores are higher with smaller models. When data is abundant, scores are higher with larger models. This insight, combined with observations from Figure 1, shows that simple models are preferred when there is insufficient data to balance variance and bias.

Training performance models is expensive and requires measuring outcomes for varied configurations. For a given workload-input pair, the time to train performance models depends on the time required to evaluate a configuration and the number of configurations that must be evaluated. Reducing evaluation time is difficult and constrained by the time required to read and process the dataset. As a result, we seek to reduce the number of configurations needed to train an effective model.

For efficient data collection, the system architect should identify configurations for which model predictions are likely to be inaccurate. Although the architect has not measured the performance of an unknown configuration, he or she can estimate the model’s prediction variance for that configuration. Intuitively, the performance model makes inaccurate performance for the configuration for which it has high prediction variance and less confidence in the performance prediction.

Figure 2 quantifies this intuition, comparing prediction variance and prediction error as neural networks predict the performance of the Spark logistic regression workload. Configurations with higher prediction variance are associated with higher prediction errors. And updating the model with performance data from these configurations is more likely to improve predictive accuracy. Thus, the system architect should identify configurations with high prediction variance to improve a model.

Active Learning. Active learning [58] applies this idea in collecting the smallest possible dataset to achieve target accuracy. This technique initializes performance models with a small amount of uniformly sampled data points for training. It then strategically acquires new data samples by locating points with high prediction variances.

Figure 3 illustrates the benefits of active learning for three classes of machine learning models: neural network, random forest, and regression splines. For each model class, the figure reports errors given the same number of measured configurations (i.e., training data). Active learning trains models with lower error by measuring configurations with higher prediction variance. It outperforms passive learning, which samples configurations uniformly at random. Active learning is particularly appealing when modeling distributed analytic applications, where data can be expensive to acquire, because it accurately predicts performance with fewer system profiles.

Dynamic Model Growth. Complementing data acquisition strategies that reduce variance, model growth strategies reduce bias. In statistical learning theory, the capacity and complexity of a model should grow linearly with the amount of training data in order to bound generalization error [32].² If updates to model structure increase capacity too quickly, a model may over-fit the training data because of its high expressiveness and may be sensitive to model variance. If updates increase capacity too slowly, a model may exhibit larger approximation uncertainty and bias between model predictions and true observations.

Dynamic modeling requires techniques that continuously collect data and periodically update not only model parameters but also model structure. The technique should start with a simple, minimally defined model architecture and dynamically grow the model as training data is collected.

Given our understanding of variance and bias, we propose Phronesis, a machine learning framework that accurately predicts performance with less data. It strategically collects training data with active learning and cautiously grows models to increase their capacity for accurate predictions.

Prediction variance determines whether performance models need additional data from particular regions of the parameter space. Measures of prediction variance guide data acquisition and reveal unobserved configurations that, if measured, could most improve model accuracy.

Estimating Prediction Variance. Phronesis estimates variance with ensemble learning. First, Phronesis initializes an ensemble of \(m\) models with distinct, random parameter values. It then independently samples training data and produces a different dataset to train each model in the ensemble. This bootstrapping technique trains \(m\) distinct models and \(m\) distinct performance predictions for each system configuration. Phronesis approximates the prediction variance for an unknown configuration by calculating the average deviation of each model’s prediction \(\hat{g}_i\) from the median prediction.

For neural networks and regression splines, Phronesis maintains an ensemble of models and calculates variance in the ensemble’s outputs as illustrated in Figure 5. We choose m = 10 to balance estimation accuracy and computation efficiency. For random forests, Phronesis maintains one instance of random forest, which is already an ensemble of decision trees, and calculates variance in the outputs from individual trees.

Acquiring Configuration Data. Phronesis scores each configuration in the parameter space with measures of prediction variance. A batch of configurations with high scores are evaluated to generate new training data. First, finding the configuration in a high-dimensional space that maximizes the acquisition score is difficult. Second, strictly searching for configurations with the highest scores may yield similar configurations for training. Two configurations with high scores can lie close to each other in the parameter space, and sampling both is redundant for training. Solving these issues would permit us to evaluate multiple configurations at a time yet avoid evaluating similar, redundant configurations when one is sufficient [57].

Search heuristics, such as the Genetic Algorithm (GA), can be used to search the configuration with the highest prediction variance. However, GA has two disadvantages. First, GA is quite computationally expensive and time-consuming. Our experiments show that GA typically takes tens of minutes to converge, which could match up with the time to evaluate unknown configurations. This is not desired in an iterative procedure in which we need to search for new configurations when the model is updated repeatedly. Second, GA normally converges to a single solution, but we want to acquire a batch of configurations at one time.

Phronesis addresses these challenges by seeking to maximize the mutual distances between configurations and maximizing the prediction variance in the training set. The distance approximates how similarly two configurations perform. By maximizing distance, Phronesis obtains configurations that are dissimilar to each other and also contain useful information for improving the model.

To calculate distances among configurations, Phronesis vectorizes configuration files and calculates vector distances. For continuous/numerical parameters, Phronesis records its value. For categorical parameters, it maps categories into multiple dummy parameters using one-hot encoding [68]. Furthermore, Phronesis re-scales the continuous parameters such that each parameter is in \([0,1]\). Re-scaling ensures each parameter is considered equally in the similarity measure between two configurations. Then, for two vectorized configurations \(c^{1}\) and \(c^{2}\) of length \(n\), Phronesis calculates the average Manhattan distance [7]:

Phronesis seeks to maximize the mutual distance of sampled configurations to get a diverse set of configurations for training. It uses a two-step procedure. First, it samples \(N=5,000\) configurations uniformly at random from the space and ranks samples by their estimated prediction variance. Second, it initializes an empty set \(S\) and iterates over ranked configurations. In each iteration, it adds a configuration if its average distance to those already in set \(S\) and its average distance to the existing training set \(S_{train}\) are both greater than a pre-defined threshold. The procedure halts when the set \(S\) reaches some budget (e.g., 15). If it cannot find enough configurations to meet the budget, it re-samples another \(N=5,000\) configurations and repeats the procedure. We define the cut-off threshold to be the average distance between configurations in training set \(S_{train}\). Phronesis evaluates the set of configurations, deploying the workload and measuring each configuration’s performance.

Phronesis initializes its ensemble of performance models with a starting dataset, determining an appropriate model architecture (from a pool of 200 candidates randomly sampled from the structure space) using 10-fold cross-validation [33]. The ensemble grows dynamically each time a large batch of new configurations (e.g., 150) is evaluated and added to the training set. Such dynamic growth gradually adjusts predictive structure, improving models’ capacity to learn from new data and balancing between prediction variance and bias. Phronesis uses different model growth strategies for each machine learning model category, as summarized in Table 4, tailoring the strategy to each model’s specific structure and hyperparameters.

Neural Networks. Algorithm 1 outlines the procedure for neural network growth. Phronesis modifies the incremental Neural Architecture Search Algorithm [32], which was originally designed for convolutional neural networks, for fully connected neural networks. Phronesis grows models within a space of neural architectures, restricting the model to a maximum of three layers.

The architecture is described by a three-value vector \((i*b,j*b,k*b),\) where the value in each position specifies the layer size as a multiple of the smallest granularity value \(b\). The algorithm increases model size until model growth no longer reduces the loss function. In each iteration, Phronesis first tries to increase layer and network size (line 6). It then increases the ratio among the network’s three layers (lines 7–10). Phronesis considers multiple candidates and chooses the one with the lowest cross-validation loss. If the loss does not fall from one iteration to the next, the algorithm terminates.

This growth strategy ensures that a neural network’s model complexity, measured by VC-dimension, does not increase faster than the training datasets. The VC-dimension of a neural network is bounded by \(O(\hat{L}W\log (U)),\) where \(\hat{L}\) is the average number of parameters per layer, \(W\) is the total number of parameters, and \(U\) is the total number of neurons [25, 67]. By adjusting each layer individually, Phronesis grows \(\hat{L}W\log (U)\) sub-quadratically to ensure the rate of model growth is similar to that of training datasets.

Random Forests. Phronesis uses coordinate ascent to grow random forests incrementally. Phronesis increases the number of trees or each tree’s depth until the generalized error does not improve. Specifically, consider a random forest defined by (\(nt,td\)), in which \(nt\) denotes the number of trees, and \(td\) denotes the tree depth. Phronesis creates two new models, (\(nt+10,td\)) and (\(nt,td+1\)). It then determines the best model from three candidates: (\(nt,td\)), (\(nt+10,td\)), (\(nt,td+1\)). Phronesis evaluates each candidate using the 10-fold cross-validation score on the training set.

Regression Splines. Phronesis uses Multivariate Adaptive Regression Splines (MARS) to automatically determine non-linearities and interactions between parameters in the spline model [30]. MARS uses a greedy two-phase approach to determine the appropriate splines for regression. In the forward pass, MARS greedily adds a pair of splines that result in the maximum reduction in the training loss. In the backward pass, MARS prunes the model, removing the least effective splines recursively until it identifies the sub-model with the highest generalized predictability. MARS uses generalized cross-validation (GCV) to score the generalized prediction ability of a regression spline where \(RSS\) is the prediction error on the training set and \(N\) is the size of the training set. GCV balances training errors and spline complexity:

Effects of Ensembles Alone. Figure 6 compares model errors as the amount of training data increases, leading to a number of conclusions for three model categories. Ensemble learning improves prediction accuracy (SM+PL versus EL+PL) for training neural networks (Figures 6(a)–6(e)). Ensembles reduce the portion of prediction error due to model variance, the variance in parameter estimation.

However, ensembles do not consistently improve accuracy for regression splines when modeling performance for the five Spark applications (Figures 6(k)–6(c)). This observation suggests that splines initialized with small training data have large biases such that more effective variance analysis and data acquisition alone are also insufficient. We must also re-adjust splines during data collection.

Effects of Active Learning Alone. Active learning reduces error (EL+AL versus EL+PL), but benefits are observed inconsistently for different models and tend to be greater when data is scarce. For neural networks (Figures 6(a)–6(e)), active learning benefits PageRank, TeraSort, Logistic Regression, and WordCount more when the system has a modest amount of training data (e.g., fewer than 750 configurations). For random forest (Figures 6(f)–6(j)), active learning consistently outperforms passive learning for Logistic Regression. It also reduces errors and improves data efficiency for Pagerank, TeraSort, and Kmeans when training data is modest. It reduces errors for WordCount after more than 1,500 configurations are sampled. Active learning has a more significant impact on regression splines (Figures 6(k)–6(o)). Both active and passive learning converge quickly to stable prediction error in all experiments, but active learning results in substantially lower errors.

Effects of Dynamic Model Growth Alone. Dynamic model growth reduces error primarily after the system has acquired a significant amount of training data (e.g., more than 1,500 configurations) when training neural networks. This observation suggests we require large training sets to effectively understand a neural network’s bias and identify more effective model structures. One notable exception arises when we train a neural network for Kmeans (Figure 6(d)). Here, the model growth strategy significantly reduces error early in the training procedure when data is relatively scarce, which may reflect poorly initialized model structure. Initially, we perform 10-fold cross-validation to identify the most promising network architectures from a set of 200 sampled candidates. However, the 200 sampled candidates may not contain a good architecture.

Model growth has a more significant impact on prediction errors when training random forests (Figures 6(f)–6(j)) and regression splines (Figures 6(k)–6(o)). This result suggests that initialized model structures offer significant room for further improvement. We cannot rely on initial model selection alone because cross-validation may not be able to identify the best of candidate models given only modest training datasets.

Benefits of Phronesis. Phronesis systematically outperforms other training strategies by combining active learning and dynamic model growth. Neural networks for PageRank, TeraSort, and Logistic Regression workloads benefit from active learning’s more informative configurations early in the training procedure and from dynamic model growth’s better model structure later. In contrast, neural networks for the Kmeans workload benefits more from dynamic model growth at the beginning and then continues to train efficiently with active learning.

For regression splines, the benefits of Phronesis mainly come from dynamic model growth. For random forests, Phronesis benefits more from dynamic model growth in PageRank and Logistic Regression and more from active learning in TeraSort and Kmeans. Note that Phronesis may initially perform slightly worse than active learning alone, possibly because the dynamic growth strategy identifies an overly complex model at the start.

Finally, note that neural networks and random forests perform better than regression splines for different applications at different numbers of training configurations. This suggests that neural networks and random forests effectively model the complex interactions among model parameters.

Effect of Scaling Training Data. We use up to 2,500 training configurations for each method in our comparison studies. We observe diminishing marginal benefits from scaling up the size of the training dataset for the varied methods. For example, Figure 7 shows the result of increasing the number of training configurations to 10,000 when using Phronesis to construct a neural network model for the Logistic Regression workload. We observe \(\lt \!\!1\%\) reduction in prediction error as data size increases from 2,500 to 10,000. Therefore, it is sufficient to report Phroensis results for dataset sizes up to 2,500.

We also compare Phronesis’s active learning strategy with two other representative sampling approaches used for modeling the performance of system and framework configurations. These sampling alternatives are Latin Hypercube sampling (LHS) [17, 19] and Non-Invasive Modeling for Optimization (NIMO) [59].

LHS is a statistical technique for generating quasi-random sequences in the high-dimensional space. LHS divides the high-dimensional space into small hypercubes. A hypercube is the generalization of squares in the two-dimensional space to high-dimensional spaces. LHS samples configuration such that each hypercube contains at most one sampled configuration. Prior work adopts LHS for performance modeling because it often provides better coverage of the parameter space than random sampling with the same number of sampled configurations.

NIMO is representative of approaches based on experiment design. It uses the Plackett-Burman method [70] to determine the importance of each parameter in determining application performance. Plackett-Burman generates \(N+1\) configurations for an \(N\)-dimensional parameter space to study the dependence of measured performance on each independent variable.

After determining each parameter’s importance, NIMO then samples from a small configuration space that contains only the most important parameter. NIMO sequentially adds a new parameter to the configuration space, in order of importance, until there is no further reduction in prediction errors from more sampled configurations.

The original NIMO implementation discusses two approaches to sample efficiently from the subspace when a new parameter is added. The first uses multi-level, factorial design to explore only values of the newly added parameter. The second uses another Plackett-Burman design to explore combinations of different parameter values for the newly expanded space. Our implementation uses both approaches to generate new configurations.

Two other active learning approaches have been proposed for modeling the performance of data analytic applications. Ernest uses experiment design to optimize the number of configurations collected to train a linear regression model [63]. d-Simplexed uses Delaunay Triangulation to model the performance function of a Spark workload and adaptively samples to reduce the number of configurations collected for training [23]. These active learning strategies are designed for specific models, linear regression and Delaunay Triangulation. Because neither model can capture complex parameter interactions in high-dimensional configuration spaces effectively, we do not include them in our evaluation.

Figure 8 evaluates sampling methods for varied models and applications. We find the advantage of LHS over random sampling is model dependent. LHS is more efficient for regression splines (Figures 8(k)–8(o)) and is similarly efficient for neural networks and regression splines. This result suggests that maximizing the coverage of sampled configurations cannot consistently improve prediction accuracy across different models in the high-dimensional configuration space.

Second, we find that NIMO is often more accurate than any other method early during the training process. However, its advantage vanishes very quickly as the number of samples increases beyond 50 configurations. In most cases, the accuracy curves flatten out or even go reverse.

NIMO benefits from exploring multiple values of the most important parameters early in the training procedure. Focusing on the most important parameters quickly reduces prediction error. However, it suffers from an increasing number of biased samples generated later in training. As NIMO spends most of its time exploring sub-regions of the space, it generates samples that are biased toward a few parameters and cannot efficiently model the interactions between different parameters as training continues. Despite this disadvantage, NIMO remains valuable for data-efficient training. Phronesis could use NIMO’s sampling strategy for an initial training set and then use its active learning strategy for subsequent sampling.

Finally, note that Phronesis’s active sampling approach consistently outperforms alternatives for different combinations of models and applications. Phronesis’s advantage from active sampling is more significant when applied to random forest and linear regression, demonstrating that a wide class of models benefit from its active learning strategy.

To better understand the benefits of model growth, we compare dynamic model growth against pre-defined model structures—either simple or complex model—for training. We manually specify simple and complex models for each model class. For the neural network, we specify a small neural network with two layers ((16,8)) and a large neural network with three layers ((512,512,64)). For the random forest, we specify a simple forest with two shallow trees (tree depth = 2) and a large forest with 200 deep trees (tree depth = 20). For regression splines, we specify a simple linear model and a complex polynomial with up to 20 degrees.

Figure 9 illustrates the benefits of model growth over pre-defined model structures. Simple models tend to reach limits at the early stages and their training curves flatten quickly. In contrast, complex models’ predictive abilities gradually improve as we accumulate more data, but their accuracies improve slowly when compared against accuracies from dynamic model growth. In addition, a complex model may exhibit strong bias at the beginning of training, leading to extremely high prediction error as shown for regression splines; see Figures 9(l)–9(n).

Phronesis requires much less data and time to train effective models when compared against state-of-the-art methods. We compare Phronesis against HMTree [72], the state of the art in building performance models for optimizing high-dimensional Spark parameters. HMTree constructs a tiered ensemble of trees such that trees in higher tiers have greater influence on the final output, whereas our ensemble methods construct models with equal influence on the final output. HMTree collects training configuration randomly and does not apply any strategic data collection technique.

Moreover, HMtrees are affected by three critical hyper-parameters: the number of trees, tree depth, and learning rate. The authors of HMtree determine these parameters by manual tuning with a sufficient amount of training data. We implement HMTree and manually tune hyper-parameters with 3,000 training configurations, constructing 1,000 trees with the maximum tree depth of 10 and learning rate of 0.05.

Figure 10 reports the number of hours required to collect training data for varied methods to reach an error target on the test data. We compare HMTree with Phronesis for neural networks (Phronesis-NN) and random forests (Phronesis-RF) because regression splines cannot reach the reported error target in our training. Both Phronesis-NN and Phronesis-RF outperform HMtree at all error targets. Phronesis’s advantages are greater when error targets are lower. Overall, Phronesis reduces time for evaluating Spark configurations for training data by up to 80 hours, a 57% reduction, for an error target, and by 25 hours, a 37% reduction, on average. Note that we consider varied error targets for each application because different applications incur varied costs for evaluating a Spark configuration.

Phronesis outperforms HMtree for two reasons. First, Phronesis actively seeks configurations with high prediction variance, which efficiently reduces output variance of our ensemble models, whereas HMtree randomly selects configurations. When training data is scarce, active learning reduces the accuracy gap between our neural network ensemble and HMtrees.

Second, Phronesis dynamically adjusts model architectures tuning hyperparameters such as the number of neural network layers, tree numbers, and tree depth. With these adjustments, Phronesis finds a good architecture that can best exploit newly acquired data, whereas HMtree requires manual tuning to pre-determine the best hyperparameters. Because the original study does not provide insight into automatically tuning these hyperparameters and brute-force search for the best hyperparameters is slow, adjusting HMtree hyperparameters for new training data is impractical.

Next, we evaluate whether Phronesis improves the efficiency of model-based autotuning methods. We construct a tuning pipeline that incorporates Phronesis and evaluates its efficiency. The pipeline first uses Phronesis to train a random forest that predicts a configuration’s expected performance for a given application. Then it applies heuristics to optimize the surrogate model’s output and find a good configuration. We evaluate a random forest because its predictions are most accurate for the five Spark applications. We evaluate a genetic algorithm (GA) because the heuristic can escape local optima and converge quickly a good configuration [45, 51].

We compare our approach with state-of-the-art, model-based tuning methods for optimizing high-dimensional Spark configurations [72]. This baseline constructs the HMtree model using random sampling and uses GA for optimization. In both model-based tuning apps, the evaluated GA has population size of 2,000, mutation rate of 0.01, and crossover rate of 0.5. In addition, we compare against two search-based tuning methods: Random sampling and Genetic Algorithm based on actual system evaluations (Real-System-GA). Random sampling randomly picks a set of configurations from the configuration space. Real-System-GA performs genetic algorithms based on the real system evaluation instead of model predictions. For Real-System-GA, we reduce the population size from 2,000 to 20 because it is extremely time-consuming to evaluate 2,000 configurations with real-system execution in one iteration. We use the same mutation rate and crossover rate as in the model-based tuning. These two search-based methods are representative of those in previous frameworks [15, 16, 65]. For example, opentuner [15], one of the most popular frameworks, implements random search and genetic algorithms [15, 6].

We report the best performance of configurations identified by these approaches when these approaches measure 100, 300, and 500 configurations. For each application, we report execution time of the identified configuration normalized to that of the default configuration. To account for randomness, we repeat each method five times and report both average and standard deviation in five experiments.

Figure 11 compares four tuning methods for five applications. Phronesis-RF-GA reduces execution time by an average of 15% and by up to 35% when compared against HMtree-GA and two search-based methods. Phronesis-RF-GA consistently outperforms HMtree-GA when tuning three applications—Pagerank, Terasort, and Logistic Regression—at all three evaluation budgets.

Phronesis-RF-GA benefits from different model training strategies. For Logistic Regression and Terasort, Phronesis-RF-GA’s advantage mainly comes from active learning in the early stages of training (i.e., 100 sampled configurations). After profiling only 100 configurations for logistic regression, Phronesis produces a random forest with less than \(20\%\) prediction error using active learning. In comparison, HMtree’s prediction error is larger than \(30\%\). On the other hand, for Pagerank, Phronesis-RF-GA benefits mostly from dynamic model growth after 300 or 500 configurations have been profiled.

On the other hand, HMtree-GA occasionally achieves comparable or better performance for Kmeans and Wordcount. For Kmeans, HMtree is more accurate up to the first 200 smapled configurations, and this accuracy translates into better performance tuning. However, as the number of sampled configurations increases to 500, Phronesis-RF-GA discovers good random forest structure that is more accurate than HMtree’s. For WordCount, Phronesis-RF and HMtree’s prediction accuracies are similar at 300 and 500 configurations. The methods’ accuracies differ by less than 1% and tuned performance is comparable.

We note that Real-System-GA generally under-performs when compared with model-based tuning methods when the number of measured configurations is greater than 300. At this point, model-based methods start to benefit from improved accuracy derived from more training data. Performance models provide an efficient way for search algorithms to explore the space, and they can have unlimited budgets to explore until they converge to an optimal configuration. In contrast, Real-System-GA is limited by the search budget and cannot converge to a final solution with up to 500 explored configurations.

Phronesis supports three kinds of machine learning models. System architects can choose the model class based on their needs. Architects who want to understand parameter interactions could choose regression splines for performance modeling. We show how Phronesis’s regression splines reveal non-linear interactions between input parameters.

Figure 12 shows Phronesis’s regression splines for predicting Pagerank and TeraSort execution time. We examine individual terms in the regression to understand the impact of each parameter and interactions between parameters. For Pagerank, data parallelism (default.parallelism) has the greatest performance impact. This impact is highly non-linear as the spline specifies an exponent of up to 5 for variable default.parallelism. For TeraSort, performance is primarily affected by the interactions between I/O compression (io.compression.snappy.blockSize), data parallelism (default.parallelism), and the number of cores available for each task (task.cpus).

The importance of each parameter depends on the program’s computational characteristics. Pagerank executes a large number of distributed join and reducebykey operations, whose performance is sensitive to the number of data partitions (i.e., default.parallelism). Terasort is an IO-intensive program that must perform intensive disk reads and writes at the beginning and end of execution. It also must perform large amounts of network IO when shuffling data. Thus, its performance is affected by I/O compression parameters and I/O parallelism, which is determined by task.cpus.

Computational Time Overhead. We characterize computational time for two key Phronesis components—model fitting and model growth—and calculate the average across five applications and three model classes. We observe a monotonic relationship between the amount of training data and the time spent on these components for three models, as shown in Figure 13. Computational costs are at most a few minutes per training iteration, which is negligible compared to the hours saved in data collection. A third Phronesis component—active sampling—incurs computational costs on the order of 100 ms, which are negligible compared to the time spent fitting and growing models.

Memory Overhead. We use the htop command to monitor Phronesis’s memory usage during execution. Profiles indicate Phronesis’s peak memory usage is around 1.5GB when it must explore multiple network architectures in parallel. This memory usage comes from storing parameters of different neural network ensembles. On the other hand, the memory usage for training random forest and regression spline is relatively small—normally tens of megabytes.