Resource-demand Estimation for Edge Tensor Processing Units

The Google Coral Edge TPU [44] is a hardware accelerator for neural network inferences. It is designed for embedded systems with limited power supply (e.g., smartphones), and it promises high power efficiency. The version considered in this article is attached via USB.

This TPU is designed to compute inferences of TensorFlow Lite (TFLite) [39] compatible models. Neural networks can be developed, for example, with TensorFlow in combination with Keras [43] or similar interfaces. This step creates standard TensorFlow “models,” which can be translated to a TFLite-compatible format using the TFLite converter [41]. Afterwards, the “Edge TPU compiler” [40] prepares the TFLite-compatible neural network for the TPU. For example, the neural network is converted to an 8-bit fixed-point number format (i.e., “quantisation”). At runtime, the compiled neural network is passed to an “interpreter” object, which delegates the execution to one of the available processors. This interpreter encapsulates and thus hides communication details. For example, if the neural network is small enough to fit into the 8 MiB memory of the TPU, then it is fully loaded when computing the first inference. If, however, the neural network is too large, then the compiler and interpreter transparently use “off-chip memory” and apply a “streaming” mechanism as a work-around.

While some TPU architectures are described in the literature [27], this specific hardware platform is poorly documented [45]. In addition, the compiler and the interpreter are mostly closed-source, and the neural network is encoded in an undocumented binary file format. Unfortunately, this lack of documentation is typical for commercially available neural network accelerators.

The absence of comprehensive public documentation makes it impossible to create timing and power models by construction. Instead, our approach creates these models by learning, that is, by monitoring the resource demand and deriving models using statistical methods.

Figure 1 visualises the structure of our energy measurement setup, and Figure 2 shows a picture of the actual hardware setup. The host system submits neural networks to the Coral Edge accelerator [44] for execution.¹Precious intercepts the power supply of the accelerator at its USB connection. It uses a shunt resistor together with an LTC2991 voltage and current monitoring sensor [15], which has a 14-bit ADC to measure the power draw. A microcontroller (“sampling chip”) polls the power measurement values periodically via I²C, aggregates them, and eventually communicates them to the host system via USB.

This setup allows Precious to create detailed power traces, as shown in Figure 3. According to our measurements, the accelerator’s idle power draw is between 280 mW and 300 mW. Even before the actual inference begins, we observe significant fluctuations in power demand that go hand-in-hand with the creation of the TFLite interpreter object. This software object is necessary for the execution of the model [41] and switches the accelerator to a state of increased power demand. During the first two seconds of the measurement, we observe power fluctuations ahead of the actual inference execution (at around three seconds into the measurement). This means that there exists a pre-processing stage before the first execution of an inference.

The pre-processing stage contains a low-power substage and a high-power substage. Both the duration of the low-power substage and the power draw (\(\le\)300 mW) during this time are independent of neural network properties. For the high-power substage, the power draw is independent of the neural network (\(\approx\)1,000 mW), but its duration varies—there is a positive correlation to the network size (Spearman correlation coefficient \(\approx\)0.5). However, during this phase, the neural network is not transmitted to the TPU, as documented in Reference [42] and confirmed by our USB communication traces. The most plausible explanation is that the host system prepares the neural network for execution on the TPU during this stage, while the TPU is in an active state, ready to receive the neural network for execution.

The TPU executes inferences only during the middle part of the power trace (in Figure 3, beginning at around three seconds until ca. six and a half seconds). During the first inference, the neural network gets loaded onto the TPU, so it lasts longer than all other inferences [42]. Therefore, its measurement results are excluded from further processing in Precious. The subsequent inferences yield repeatable measurement results from which Precious can derive its power and time models.

In Figure 3, the measured values of several inferences of the same convolutional and fully connected neural networks were superimposed to visualise the power course of an inference. The superimposed visualisation allows a detailed analysis of the power draw over time, even though a single inference executes within a few milliseconds. This is due to the intentionally deterministic execution behaviour of the TPU [27]. The measurements show clear differences between fully connected (“dense”) and convolutional networks. Convolutional networks have clearly repeating “hills” and a higher power draw of 1,200 mW to 1,400 mW. Our communication analysis (cf. Section 5.5) shows that the power drop between hills is caused by data transmissions between the TPU and host. In comparison, dense networks show only minor oscillations near 1,150 mW, as shown in Figure 3. For both network types, the observed behaviour is repeatable.

In summary, both the execution time and the power draw during the inferences stage depends strongly on the neural network. As Figures 5 and 6 show, there are correlations between properties of the neural network and the resource demand for its execution. Therefore, Sections 4 and 5 describe the design, implementation, and evaluation of resource-demand prediction models for the inference stage.

After the inferences stage, the TPU remains in an active high-power state for approximately 3.5 seconds before returning to a low-power idle state. This post-processing stage has a higher power draw that is similar to the pre-processing stage and higher than the TPU’s initial idle state. We assume that the pre- and post-processing stage correspond to different USB transmission and power states of the Cortex-M0+ microprocessor that operates on the TPU stick. The duration of the post-processing stage appears to be constant, and in particular, independent of the properties of the executed neural network. This post-processing stage was only observable in TensorFlow version 2.0; the behaviour changed when we updated our host system to version 2.1.² In the newer version, the TPU remains in a high-power state (\(\approx\)1,000 mW), rather than switching to a low-power idle state (\(\le\)300 mW).

In the first phase, Precious uses the machine learning framework TensorFlow 2.1 with Keras [43] and Python 3.6 to generate randomised neural networks. It supports three different network types—convolutional (“conv2d”), fully connected (“dense”), and heterogeneous networks [30]. Generated neural networks are either homogeneous or heterogeneous. First, homogeneous networks consist of either only convolutional or only fully connected layers. Second, heterogeneous networks contain both convolutional and fully connected layers. As of now, the implementation supports “relu” as an activation function and imposes restrictions on the dimensions of convolutional networks. For each layer, the input dimension is the same as the output dimension. All generated homogeneous networks consist of layers of the same type and dimension (for convolutional layers this implies the use of the “same” padding). Convolutional layers always have a square-dimension input with a depth of 3 and 3 filters with the dimension \((3,3)\). Our system varies the number of layers (between 2 and 250) and the dimensions (between 100 and 1,024), randomly. Generated heterogeneous networks contain first convolutional layers, a “maxPool2d” and a “flatten” layer, followed by dense layers. The number of layers varies between 10 and 1,024, and the range of layer dimensions is equal to the generation of homogeneous networks. For all network types, all network-internal parameters (i.e., weights) are initialised to random values. The neural networks remain untrained, as we use Precious to examine the resource demand of different network types and configurations and not the resource demand for a specifically trained network.

To execute a neural network on the accelerator, it must first be converted to a TensorFlow Lite (TFLite) model. In addition, all parameters must be in an 8-bit fixed-point number format. The conversion to this format is either achieved during training (i.e., “quantisation-aware training”) or afterwards (i.e., “post-training quantisation”). Since Precious only uses randomised instead of meaningfully trained weights, it uses the latter technique to translate the randomised parameters to 8-bit values. Furthermore, this technique also allows to convert existing pre-trained neural networks and execute them on the accelerator. This neural network translation process is executed on the host system using the Edge TPU compiler [40].

In the second phase, Precious analyses several properties of the neural network. This analysis is entirely static, that is, all information is available without executing the neural network.

First, the TensorFlow format already offers information about the number of layers and the layer dimensions. For the generated neural networks in Precious, these properties are randomly chosen. For applications that utilise neural networks, these properties are selected by the application developer and potentially adjusted during or after the training process. Second, Precious derives the number of (internal) neural network parameters, such as weights and biases, from the number and dimension of the neural network layers. Furthermore, the number of multiply-accumulate (MAC) operations is computed (i.e., the number of MAC operations needed per neural network execution). Third, the Edge TPU compiler provides the network size (i.e., memory demand) of the neural network. The memory demand is important, because small neural networks fit into the 8 MiB on-chip memory, but larger neural networks transparently use a “streaming” mechanism where parts of the neural network are fetched on-demand during inferences. Finally, the file system tells the file size of the neural network in the binary format used by the accelerator after compilation. In summary, these easily accessible properties later serve as features for the training phase that creates a model that maps from the neural network properties to their resource demand. The labels (i.e., the resource demand) needed for the training phase are collected in the neural network execution phase.

Figure 5 summarises the influence of the number of MAC operations on the execution time and power draw of fully connected and convolutional networks. Some correlations are clearly visible. Similarly, Figure 6 visualises the influence of the memory demand of neural networks on the resource demand. The figures look similar, because the number of MAC operations and the memory demand correlate positively. However, both properties individually are not suitable as resource-demand predictors, since both figures show a large amount of noise. The figures look noisy, but as the resource demand measurements are repeatable, the noise cannot be run-to-run variation. Instead, the shown variance is caused by other neural network properties that are not visible, as the many-dimensional property space is reduced to two-dimensional representations. In consequence, no single property is sufficient to accurately estimate power draw and execution time. Instead, Precious has to consider multiple properties in the estimation models.

In the third phase, Precious measures the energy demand of neural networks. As described in Section 3, Figure 1 visualises the structure of our energy measurement setup, and Figure 2 shows a picture of the actual hardware setup. For the power and execution time measurements, the host system submits neural networks to the Coral Edge accelerator [44] for execution. The input data for the execution is randomised, but used repeatedly for each neural network, and no batching is applied.

Our embedded TPU [44] can be configured in two variants, a standard variant (STD) and a second variant (MAX) that operates with the maximum operating frequency [42]. Both the power demand and execution time of the TPU depend on this configuration. We use the former (i.e., slower) variant for our implementation of Precious, because the latter suffers from documented thermal problems [42]. To avoid overheating, Precious nevertheless makes idle periods during the evaluation to maintain a constant temperature. Section 5.4 looks at the self-induced warm-up in detail.

Each neural network executes five iterations. Each iteration repeatedly executes inferences of the neural network and lasts at least 30 seconds. The power draw and execution time are averaged during each iteration, ignoring the first inference, as it takes longer because the neural network is loaded onto the TPU. We then combine the five iterations by computing the median. In general, power draw and execution time show little run-to-run variance—for example, the detailed power trace in Figure 3 combines measurements from multiple inferences. The obtained power and time measurements serve as labels in the training phase.

The training phase applies supervised learning where the neural network properties (cf., Section 4.2) are the features and the resource demand (cf., Section 4.3) is the label.

The total dataset contains 2,992 homogeneous and 127 heterogeneous neural networks that are partitioned into training (80%), validation (10%), and evaluation (10%) sets. Each regressor trains a model using the training set, only. Hyperparameters were fine-tuned using randomised parameter optimisation (RPO) with 100 iterations using the validation set. The hyperparameter optimisation was only conducted for the homogeneous neural network dataset and the results were reused for the heterogeneous neural networks. Consequently, the validation dataset were merged into the evaluation dataset for heterogeneous networks. Importantly, the regressors never access the evaluation set—that set is only used for the evaluation in Section 5.2. We create, in summary, 10 estimators—for time and power, and for small dense (i.e., without off-chip memory), large dense (i.e., with off-chip memory), convolutional neural networks, and for small (i.e., without off-chip memory) and large (i.e., with off-chip memory) heterogeneous networks.

The models are trained by, in total, eight regressors³ with 18 configurations (denoted as (*)) in the non-heterogeneous case. Models are either linear [14] or ensemble [13], that is, models that internally combine different machine learning techniques. Table 1 gives an overview of the regressors and the models they create. Regressors for ensemble models are configurable by the error metric that is minimised during training, such as the Friedman mean squared error (FMSE), the mean absolute error (MAE), or the mean squared error (MSE). For comparison, an additional dummy regressor computes the mean (MEA) or the median (MED) of the labels, ignoring all features. This dummy regressor serves as a baseline for the evaluation of linear and ensemble models. Based on the results for homogeneous networks, we used the ETR(MAE) regressor to implement the models for heterogeneous networks.

We intentionally did not train models that use deep learning or neural networks internally—neural networks are only the subject of modelling, not the implementation technique. The relatively simple regressors used in this article require significantly lower training efforts, but the evaluation shows that they provide accurate estimations.

In the final phase, the developers of deep learning applications apply the trained models to estimate the resource demand of their neural networks. The intended use-case is that phases 1 to 4 are executed once, for example, by the hardware vendor, and the resulting model is distributed. In contrast to cycle-accurate simulators, such measurement-based resource-demand models can be published without the risk of leaking intellectual property, as they are derived from user-observable properties only. Using these models, deep-learning application developers can determine the resource demand of different network architectures—without power measurement infrastructure and before training—and restrict the training process to models that satisfy all resource constraints. Further potential application scenarios are discussed in detail in Section 2.

In general, the properties derived by static analysis carry information on the resource-demand of the neural network. We analyse the importance of individual properties by computing the Spearman correlation coefficient between a feature and the power draw (\(r_p\)) and between the feature and the execution time (\(r_t\)). A high correlation (i.e., values near 1 or \(-1\)) indicates a strong dependency between the respective property and the resource demand. In consequence, such a property is a useful feature for a resource-demand estimator. Table 2 summarises the Spearman correlation coefficients for homogeneous (i.e., convolutional, small dense, and large dense) and heterogeneous (i.e., small and large) neural network properties used by Precious. Input Features (Homogeneous Networks). Each property has a high correlation in at least one case, indicating that each property contributes to the estimation accuracy of the resource-demand models. In most cases, the number of MAC operations correlates strongly with the resource demand, in particular, considering the execution time data visualised in Figure 5. For large dense models with parameter streaming, the execution time correlates extremely strong with the amount of off-chip data, as indicated in Figure 6. However, Figures 5 and 6 also demonstrate that neither property alone is sufficient to estimate the resource demand. Instead, the combination of properties is needed for an accurate estimation.

Input Features (Heterogeneous Networks). For the resource-demand estimation of heterogeneous networks, we analysed the same input features as used for homogeneous networks. Additionally, we analyse the number of MAC operations disaggregated for convolutional and dense layers, respectively. We deliberately refrain from using more complex representations of neural network topologies (e.g., order of layer types), but continue to focus on metrics, which are easy to statically determine and easy to represent, for example, the number of parameters or MAC operations. This allows us to still use machine learning techniques with relatively little overhead.

The Spearman correlation coefficients show that the memory demand for small heterogeneous networks is only lightly correlated with the resource demand. Due to their heterogeneity, networks of the same size can have very different topologies and hence resource demand. The off-chip memory demand for large heterogeneous networks, however, determines the overhead for parameter streaming and is hence more strongly correlated.

In contrast to homogeneous networks, the number of layers and parameters are only lightly correlated, as different layers and parameters potentially yield very different behaviour, depending on the network topology. Furthermore, to fit on the TPU, small heterogeneous networks only have a limited potential number of layers. On the other side, the number of MAC operations proved again to be strongly correlated with the resource demand and remains one of the most important input features. Furthermore, the number of MAC operations for convolutional layers significantly outweighs the number of MAC operations for dense layers.

Based on these analyses, we selected the off-chip memory demand, total memory demand, and the three metrics regarding the number of MAC operations as inputs to the resource-demand estimators for heterogeneous networks.

Figure 7 summarises the estimation accuracy of all models trained to estimate the resource demand for homogeneous networks. We measure the resource demand for the evaluation dataset and compare this “true” value with the estimation based on the trained models. We use the mean absolute percentage error (MAPE) as error metric, because it normalises the error to the measurement, and the individual resource demand can vary (network-to-network) significantly. The graphic is cut off at 15% to maintain readability. The evaluation demonstrates the following:

The dummy regressor performs badly at modelling the execution time. This demonstrates that execution-time modelling is not trivial and neural networks properties must be considered.

The ABR(*) regressors cannot model the execution time of neural networks accurately. For dense and convolutional networks, the estimation error is larger than all other ensemble and linear models. This relatively bad accuracy might be caused by an insufficient number of hyperparameter tuning iterations. The accuracy of the outlier-robust HR regressor is consistently worse than a simple linear regressor.

The execution time of dense networks without off-chip memory (i.e., without streaming) can be generally well approximated. Linear and ensemble models achieve similar accuracy. This means that the relation between the chosen neural network properties and the execution time is linear (as indicated by Figure 6). For the execution time of dense networks with off-chip memory (i.e., with streaming), the linear models perform worse than ensemble models. This means that, in contrast to small networks without off-chip memory, the relation is no longer strictly linear. There is no single supreme regressor, as ensemble models (ETR(*), RFR(*)) achieve similar accuracy. The execution time of convolutional networks is harder to predict than dense networks, as the remaining MAPE is higher. Similar to dense networks with off-chip memory, ensemble models achieve a higher accuracy than linear models. As an exception, the ABR regressors converge too slowly during our experiments.

The power draw of dense networks without off-chip memory is trivial to predict. Linear, ensemble, and dummy models have similar accuracy. This is indicated in Figure 3, as the power draw of dense networks is almost constant, with only minor oscillations. For dense networks with off-chip memory, the power draw is more difficult than for small networks without off-chip memory. Linear and ensemble models achieve similar accuracy, and the dummy regressor is slightly worse. For convolutional networks, the power can be approximated by linear models as well. Linear models and ensemble models are similarly accurate, and both are more accurate than the dummy regressor. Only the HR regressor achieves a bad accuracy.

In summary, regressors can achieve an error below 1% for homogeneous networks and for both power draw and execution time. This is due to the deterministic characteristics of neural network executions in general and the deterministic hardware platform in particular. As our approach is not restricted to the TPU used in our evaluation, estimators for similar accelerators (e.g., a Kendryte K210) can be trained without changes to our approach and we expect similarly low errors.

Based on the results for homogeneous networks, we trained four additional estimators to predict the execution time and power demand for small and large heterogeneous networks. Therefore, we utilised the ETR(MAE) regressor, as it showed consistently good behaviour for all homogeneous networks.

The mean absolute percentage error (MAPE) to estimate the power demand and execution time is 1.5% and 8.6% for small heterogeneous networks, respectively. For large heterogeneous networks, the MAPE for the power demand and execution time is 2.2% and 9.2%, respectively. Hence, the estimation error for small heterogeneous networks is slightly smaller, which is due to the more predictable behaviour if no parameter streaming is required.

Even though the accuracy is lower compared to the estimators for homogeneous networks, the heterogeneous-network estimators achieve lower or comparable accuracy results compared to state-of-the-art estimators without access to cycle-accurate simulators (see the related work in Section 6). Furthermore, we achieve these results with statically determinable features and low-overhead prediction methods.

However, the network topology may comprise information relevant for the execution time, which is not accessible for the regressors in our implementation. Using more sophisticated implementation techniques (e.g., neural networks) could allow us to use more expressive network representations and utilise such information to achieve potentially higher accuracies. However, using more sophisticated modelling techniques would also come with increased overheads for training and execution. The following sections will put the accuracy of linear and ensemble models into context by comparing their estimation errors to the influence of self-induced heat and data transmissions.

The power draw of transistor-based logic circuits generally depends on the temperature, which is influenced by the ambient temperature and also self-induced heat. The ambient temperature varies over time for most embedded systems, and also the self-induced warm-up is typically not precisely controllable. In consequence, the power draws of identical workloads differ between executions. Models trained with only statically available data—such as the models trained by Precious—are oblivious of temperature changes during runtime and therefore cannot capture this run-to-run variance. We evaluate the influence of the device temperature on its power draw—it constitutes a lower bound for the achievable estimation error of any power-demand model under realistic conditions.

To obtain temperature values, we utilise a BME280 temperature sensor from Bosch Sensortec. We sample temperature values at \(40 \,\mathrm{Hz}\) with 1°C accuracy and 0.01°C precision. During measurements, the temperature sensor is attached to the surface of the TPU casing.

Figure 8 shows the power demand and surface temperature of the TPU executing a convolutional network. The TPU surface temperature starts at ambient temperature (24.6°C) and rises within 20 inferences by 5.8°C. Around inference 20 the temperature stabilises at 30.4°C. The power demand shows the expected correlation with temperature and rises by \(39.0 \,\mathrm{m}\mathrm{W}\) and stabilises at 1,332.1 mW after 20 inferences, which is an increase of 2.9%. In comparison, most power models discussed above achieve an accuracy better than 2.0%, which is only possible because the device temperature is controlled in our experiments. Therefore, linear and ensemble models achieve sufficient accuracy, and more elaborate modelling techniques cannot improve the accuracy further under realistic conditions.

One important factor for the efficient usage of an external accelerator is the communication between the accelerator and host system, that is, the USB communication in our case. As already described in Section 3, the TPU has a pre-processing, inference, and post-processing stage. This section further analyses the USB traffic during the inference stage and its influence on the TPU’s efficiency.

Figure 9 shows the power demand during a convolutional network inference and the respective USB traffic from and to the host system. The power data consists of 210 superimposed inferences of the same network and the same input data to improve the level of detail. During the reception and transmission of data, the power demand of the TPU significantly decreases from approximately \(1.450 \,\mathrm{m}\mathrm{W}\) to \(1.150 \,\mathrm{m}\mathrm{W}\). Hence, the data transmission phases are one reason for the very characteristic power traces of the TPU and thus influence the energy demand and efficiency of the TPU.

USB traffic during the inference stage usually occurs due to one of two reasons:

Due to the reduced power demand during USB transmission phases, we assume that the execution units of the TPU are not or not fully utilised and the TPU waits for additional data to arrive or send. Thus, we observe that USB traffic negatively affects the TPU’s utilisation. Figure 10 shows the average utilisation (i.e., MAC operations per second) for different input-data dimensions for convolutional and fully connected (“dense”) networks, respectively. For dense networks, the network size depends on the number of layers and the layers’ dimensions. Consequently, for higher input dimensions, the network may at some point be too big to fit into the on-chip memory and must be continuously streamed during execution, which significantly decreases the utilisation and, therefore, the TPU’s efficiency (left plot in Figure 10). However, dense networks fitting into the on-chip memory achieve a higher utilisation compared to the convolutional networks in our dataset (right plot in Figure 10).

Most convolutional networks in our dataset fit into the on-chip memory, as the convolutional networks’ sizes are independent of the input-data dimension, but only depend on the number and dimension of the filters. Accordingly, the utilisation is relatively constant for different input dimensions (right plot in Figure 10; pink graph), although lower compared to dense networks fitting into the on-chip memory. This reduced utilisation is presumably due to the bigger input-data sizes for our convolutional networks and the corresponding USB traffic. One interesting finding is that the incoming traffic decreases if the outgoing traffic is reduced (by reducing the output dimension). For a subset of the convolutional networks, we executed identical convolutional networks with the same data and only one additional small layer to harmonise the output size to a constant size of three filters with dimension 14x14 (right plot in Figure 10; yellow graph). We found that not only the output traffic, but also the input traffic was reduced by the additional layer, and consequently this traffic reduction led to an increased utilisation. We assume that the reduced output complexity also reduced the complexity of the execution instructions and hence decreased the traffic from the host.

In summary, reducing the USB traffic significantly increases the TPU’s utilisation and efficiency. Especially streaming the network on-the-fly for networks exceeding the on-chip memory decreases the TPU’s utilisation. However, the same holds for the input and output data transmissions. In general, the influence of the USB communication on the TPU’s utilisation demonstrates that, in many cases, the TPU’s performance is communication-bound. In this case, an accurate execution-time model has to take communication timings into account—however, Precious’ models do not consider them, because the exact timings are not statically analysable, in particular, with bus contention. Nevertheless, the USB traffic size is one example of a property that can be easily optimised by a neural network developer with a resource-prediction model at hand. These differences are further increased for different execution frequencies presented in the following section.

The TPU supports two different power modes, that is, the STD and MAX mode. This section analyses the impact on the power demand, execution time, and energy demand of both power modes. The TPU runs with the full frequency in MAX mode, whereas in STD mode the TPU is throttled. However, the MAX mode may not be applicable, depending on the deployment scenario due to a significantly increased heat dissipation [42].

Figure 11 shows the execution time and power demand differences between both power modes for dense and convolutional networks, respectively. The difference is calculated by subtracting the STD execution time/power demand values from the MAX mode values. Enabling the MAX power mode leads to decreased execution times for both network types. However, the gains for the convolutional networks are significantly higher, because the dense networks’ execution times are more affected by data transmissions (cf. Section 5.5). As expected, the power demand increases when using the MAX power mode for both network types. The power difference between STD and MAX mode is relatively constant if the TPU is fully utilised (as seen with the convolutional networks). If the network is too small to fully utilise the TPU’s on-chip memory, for example, the first dense networks with less than \(2 \cdot 10^7\) MAC operations in Figure 11, then the power differences rise linearly with the number of MAC operations.

Figure 12 shows the energy demand difference between both power modes for the dense and convolutional networks, respectively. Again, the difference is calculated by subtracting the STD energy demand from the MAX mode energy demand. Dense networks with less than approximately \(2 \cdot 10^7\) MAC operations can be stored entirely on the on-chip memory and need no additional data streaming. For these networks, the energy demand is slightly lower in MAX mode than in STD mode. For bigger dense networks (i.e., with more MAC operations), additional traffic is required and the increased power demand is not compensated by the reduced execution time. Hence, for these bigger networks, the energy demand is increased in MAX mode. Most convolutional networks in our dataset fit into the on-chip memory and do not require additional traffic. Consequently, for all convolutional networks, the energy demand is reduced in MAX mode.

In summary, for networks fitting into the on-chip memory, the energy demand is significantly reduced in MAX mode due to the decreased execution time. For all other networks, the energy demand is increased due to the increased power demand and only slightly decreased execution time. Hence, for scenarios where the additional heat dissipation can be handled and the network fits into the on-chip memory, the MAX mode can help to reduce the energy demand.

In summary, the results indicate that the generated models estimate the resource demand sufficiently accurate, despite relatively simple regressors. However, the results are obtained for only one hardware platform. For an extension to further hardware platforms, Precious needs only minor adjustments. First, Precious has to replace TPU-specific aspects, such as the off-chip memory usage. However, other important neural network properties, like the number of MAC operations and the number of neural network parameters, can be reused. Second, for the execution phase the hardware platform has to be replaced. Depending on the accelerator, it may be challenging to accurately measure the power draw if it is tightly integrated in the system-on-chip. Considering the achieved accuracy of resource-demand models, other hardware platforms may need more complex regressors or provide worse results, as TPUs are specifically designed for deterministic execution [27]. Papers that examine alternative hardware platforms in detail are discussed in the following section.