Virtual machine scheduling and migration management across multi-cloud data centers: blockchain-based versus centralized frameworks

Efficiently managing virtual resources in the cloud is crucial for successful recourse utilization. Scheduling is a vital technique used to manage Virtual Machines (VMs), enabling placement and migration between hosts located in the same or different data centers. Effective scheduling not only ensures better server consolidation but also enhances hardware utilization and reduces power consumption in data centers. However, scheduling VMs across a Wide Area Network (WAN) poses considerable challenges due to connectivity issues, slower communication speeds, and concerns around data integrity and confidentiality. To enable informed scheduling decisions, it is critical to facilitate the exchange of real-time and accurate status information between cloud data centers, ensuring optimal resource allocation and minimizing latency. To address this, we propose a novel distributed cloud management solution that utilizes blockchain technology to facilitate efficient sharing of VM characteristics across multiple data centers. BigchainDB platform has been used as a blockchain-based ledger database to effectively share information required for VM scheduling and migration across different data centers. The proposed framework has been validated and compared with a Virtual Private Network (VPN)-based centralized management solution. The proposed model utilizing blockchain-based solution achieves 41.79% to 49.85% reduction in number of communication messages and 2% to 12% decrease in total communication delay comparing to the centralized model.

The use of cloud computing is rapidly growing, with virtualization playing a key role in cloud data centers. Virtualization allows multiple VMs to run concurrently, improving hardware utilization [1]. Due to the dynamic nature of cloud resources, advanced scheduling methods are essential to allocate tasks efficiently, meet application needs, and optimize resource use [2]. Scheduling techniques focus on improving metrics like reliability, response time, availability, energy efficiency, cost, and resource utilization [3]. Additionally, extended cloud resources such as Fog and Edge computing, supporting IoT devices, consider resource location for migrating VMs, containers, or data between nodes [4, 5]. Inadequate scheduling can lead to resource overutilization or underutilization, resulting in wasted resources or degraded performance [6].

To address this, advanced scheduling algorithms using heuristic, meta-heuristic, and hybrid methods have been developed. Dynamic resource allocation techniques also aim to optimize resource usage while considering multiple objectives and constraints [7]. However, conflicting application demands and workload complexities make resource allocation a persistent challenge [8]. With increasing customer demands and limited resources, developing effective models and strategies remains critical, driving ongoing research into innovative solutions.

VM migration is a widely used method for managing virtual machines by transferring them between hosts within or across data centers. VM migration is essential for optimizing resource utilization, ensuring high availability, and balancing workloads across hosts in cloud environments. It typically occurs when a host experiences resource overload or underutilization, or when a VM’s resource needs exceed the host’s capacity. Migration also happens during planned maintenance, fault tolerance events, or to improve energy efficiency by consolidating VMs on hosts with lower power consumption [9]. Ensuring minimal service disruption during migration is critical, and live VM migration techniques have been developed to address this, enabling seamless transfers with minimal downtime.

The migration process begins with assessment and planning, where the system evaluates resource usage and selects the appropriate destination host. The migration phase involves transferring the VM’s state, including memory, CPU, and storage, using methods like pre-copy [10] or post-copy [11] . During this phase, network configurations are updated to reflect the new host. Once migration is complete, the VM runs on the destination host, and the source host’s resources are released. Continuous monitoring ensures smooth operation and resolves any issues post-migration.

Effective virtual resource management and scheduling are crucial for optimizing data center utilization while reducing costs and meeting market demands [12]. Maintaining data center stability requires a clear understanding of system status and the use of intelligent scheduling techniques. By analyzing global and local migration data, more precise decisions can be made regarding VM scheduling and migration. Accurate, real-time status information is essential for efficient, secure, and timely decisions, leading to better server consolidation, load balancing, and optimized resource distribution.

In centralized cloud management, a global manager collects status data from all data centers, either through periodic pulls or local pushes, storing it in a central database. Scheduling algorithms then use this data to optimize decisions. However, this model faces challenges like communication bottlenecks, single points of failure, security risks, and contention from transaction updates [13]. These issues require careful evaluation and mitigation to ensure performance in large distributed systems. Distributed cloud management offers a more efficient and secure alternative [14]. Each data center maintains a replica of others’ statuses, synchronized to reflect the latest VM and data center updates. This model reduces communication overhead and enables direct inter-center communication, streamlining scheduling and resource allocation.

Cloud and edge computing have transformed internet-based applications and services but face challenges in data sharing, collaboration, and security. Traditional centralized architectures often fall short of modern digital infrastructure needs. Blockchain, a decentralized and tamper-resistant distributed ledger, offers a trust-based alternative by enabling secure, cost-effective transactions without central authority. Its distributed structure makes it well-suited for cloud/edge computing, addressing data privacy, security, and decentralization challenges while enhancing efficiency in scalable environments. Blockchain’s potential spans various sectors, including IoT, healthcare, electronic voting, and social networking.

For data communication, REST APIs dominate in industrial and academic settings, particularly for log monitoring through on-demand retrieval or periodic pushing. Tools like Prometheus and Grafana handle log collection, storage, and visualization [15]. In systems like Nova [16] and Cloud Foundry [17], compute nodes publish logs via REST APIs for centralized management. However, this centralized approach struggles with security, scalability, and high communication volumes in distributed cloud environments. Blockchain-based distributed management overcomes these limitations, enabling secure and efficient communication among nodes. This reduces errors, improves decision-making, and ensures better scalability in large-scale cloud systems.

The main contribution of this research is to enhance the scheduling and migration of VMs in cloud data centers by enabling effective and secure sharing of VM characteristics between different cloud data centers. Blockchain technology is integrated into a distributed cloud management model. Blockchain nodes are distributed across cloud data centers and act as a secure and efficient registry to data center status information. It streamlines WAN migration by securely and efficiently sharing the status of all data centers. With its immutable nature, it ensures the consistency of VMs and data center status replicas across all data centers. In this type of management model, all data centers work together in VM scheduling across data centers, taking into account both WAN and LAN scheduling considerations. This approach minimizes the potentially adverse effects of WAN scheduling operations. Additionally, we present a VPN-based centralized cloud management model that is based on centralized systems’ definitions. A central database that resides on a centralized global cloud manager is utilized to save the status of all VMs in all data centers. The performance of the two presented models is compared in terms of the number of exchanged communication messages and the total delay of communication to perform migration decisions for certain numbers of VM migrations across data centers.

The remaining of the paper is structured as follows: “Related work” section discusses the related work in this area; “Proposed model architecture for cloud management data sharing” section describes the proposed system model and architecture; the experimental setup is shown in “Experimental setup” section; “Results and discussion” section presents performance evaluation results; “Results validation” section presents a validation for the results; the limitations and future work is presented in “Discussion, limitations, and future work” section; and finally, “Conclusion” section concludes the paper with final remarks and potential future directions for this work.

Sharing and storing data to ensure collaboration in cloud resource scheduling, while guaranteeing data integrity and confidentiality, pose significant challenges in cloud computing. Traditional centralized storage systems offer cost-effective solutions, but they often face scalability issues, performance bottlenecks, and security vulnerabilities. In contrast, decentralized storage systems distribute data across various nodes, providing redundancy, data availability, and improved security. However, decentralized storage introduces its own set of challenges, such as complex data retrieval processes, potential inconsistencies in data versions, and difficulties in ensuring data privacy and integrity in a distributed setup. The combination of blockchain technology with cloud computing has proven to be a promising approach for bolstering cloud transaction security and enabling access to data and applications.

The article in [18] proposes a decentralized storage system that combines cloud-native concepts with blockchain technology to address the challenges of data management in decentralized settings. The proposed design offers enhanced scalability, data security, and privacy. When deployed on a containerized edge infrastructure, this storage system provides faster data-transfer speeds than the interplanetary file system.

Edge computing boasts a swift connection with end devices, but it does come with limited resources that necessitate cooperation and data exchange among various edge clusters and cloud data centers. To address these challenges, numerous research studies have delved into the topic. In particular, articles featured in [19,20,21,22] have explored the potential of blockchain technology as a means of facilitating data sharing management.

The article in [19] proposes a collaborative data-sharing system that leverages blockchain and cloud-edge computing technologies. Its aim is to allow multiple data providers and users to collaborate effectively. The system used a stochastical probability model where authors assumed a homogeneous Poisson point process for the spatial distribution of data providers and users and an independent Bernoulli process for the transaction generation rate at each node. Authors provide analyses for selected performance metrics and evaluate the system’s performance based on their experimental findings.

Authors in [20] discuss the challenges of collaborative edge storage due to the limited capacity of individual edge servers and their operation in a distrusted environment. They introduce CSEdge, a decentralized system that uses blockchain technology to enable collaborative storage among edge servers. The system allows edge servers to submit data offloading requests, which are contested for by other servers based on their reputations. Successful servers are rewarded and their performance is recorded on the blockchain, resulting in CSEdge being an efficient solution for collaborative edge storage.

The approach outlined in [21] enhances the edge demand response (EDR) process by applying Vickrey-Clarke-Groves (VCG) auction theory. This enhancement results in improved distribution of workload among edge nodes, better utilization of edge resources, and decreased energy consumption within the system. To strengthen the incentive and trust system for collaborative edge computing, the researchers integrated blockchain technology. They examined three different scenarios: no collaboration, internal collaboration, and incentive collaboration. The approach takes into account the impact of user task transmission distance on the quality of experience (QoE) and addresses potential forking attacks on the blockchain within collaborative edge computing. By integrating blockchain technology, the approach ensures secure and continuous records of task execution, establishes a trust system among edge nodes, and encourages servers to actively engage in edge collaboration while upholding task execution quality.

A study in [22] proposes a cloud and edge computing system that utilizes blockchain for VM placement in a decentralized manner. This system employs a dynamic threshold-based approach, where each server takes into account its own and other servers’ CPU utilization before migrating VMs. The study shows that this approach is more effective than random peer-to-peer VM placement. Moreover, it ensures secure communication among servers and maintains CPU utilization below the upper threshold limit.

Article in [23] introduces a new cloud scheduler based on blockchain technology. The goal of this scheduler is to enhance security in task allocation, data storage, and transmission among cloud nodes and clusters. The scheduling optimization problem is solved using an asymmetric Stackelberg game model, with schedule execution time as the privileged criterion. The effectiveness of the proposed blockchain cloud scheduler is evaluated through a specially designed simulator named the Blockchain Secure Cloud Scheduler Simulator, BCSchedCloudSim, and its efficacy is confirmed through experimental analysis.

BigchainDB and Amazon Quantum Ledger Database (QLDB) were compared in a study in [24]. The particular use cases for each technology were considered when making a comparison. QLDB, which is Amazon’s proprietary technology, is most suitable for situations where all parties trust a central authority and centralization is not a concern. On the other hand, BigchainDB provides true decentralization across distributed ledgers, making it ideal for situations where there is no central authority and centralization is a concern. Furthermore, BigchainDB benefits from the advantages of contemporary distributed databases and blockchain technology, such as high transaction rates, low latency, indexing, and structured data querying, as well as decentralized control, tamper-proofing, and traceability [25]. According to the findings, while Amazon QLDB may have better overall performance based on CPU and memory usage metrics, BigchainDB’s flexibility and potential for broader application makes it a valuable technology worth considering. These findings highlight the importance of carefully evaluating the specific needs and requirements of a given project when choosing between decentralized and centralized ledger databases.

A literature review comparison is presented in Table 1. Recent studies have shown that there is a gap in the existing literature when it comes to the development of a secure and efficient distributed system that can handle a large volume of transactions for scheduling cloud resources across multiple data centers using blockchain technology. Our research aims to bridge this gap by utilizing the BigchainDB platform, which combines the best of both worlds by offering a distributed, immutable, and secure blockchain along with the scalability and low latency of distributed databases. We are particularly interested in exploring the migration performance of the cloud scheduler, focusing on reducing the impact of VM migration across data centers on the overall scheduling delays of the system. Our findings have significant implications for the future development of cloud resource scheduling systems and can contribute to the ongoing effort to make cloud environments more efficient and secure.

Specialized modules that perform specific functions make the efficient operation of a data center’s cloud manager possible. These modules of a basic cloud manager include the Allocation Manager, Resources Scheduler Manager, Network Manager, VM Migration Manager, and Service Manager as shown in Fig. 1. These modules carry out the critical activities of a data center, making them indispensable for the smooth functioning of a cloud manager. Communication channels refer to all methods of communication between the processes and modules of the cloud manager, including network (wired or wireless), inter-process communication, and shared memory. However, this research focuses on communication overhead between data centers, not the specifics of inter-process communication.

Basic cloud manager internal modules

Full size image

The Resource Scheduler Module is a critical part of the cloud management infrastructure, responsible for efficiently managing resources to optimize data center performance. It allocates VMs to hosts based on resource availability and initiates VM migration to ensure high availability and balanced workloads, reducing host overloading and maintaining optimal performance.

The Allocation Manager works alongside the Resource Scheduler to determine the best timing and method for allocating computing and memory resources to each VM. This process takes into account factors such as VM characteristics, workload priority, and host resource availability. By using optimization techniques, the Allocation Manager ensures efficient distribution of VMs across hosts. Together, the Resource Scheduler and Allocation Manager maximize data center utilization through effective resource allocation.

The Allocation Manager monitors the status of all VMs in the data center, identifying which need to be relocated and which destination hosts are available. This information is provided to the scheduler, which selects the optimal host for minimal migration. The Allocation Manager is responsible for the initial VM placement, while the Resource Scheduler manages VM migration. In practice, the Allocation Manager receives updates from the scheduler, including a list of VMs to be reallocated, along with source and destination host details. It then reserves resources on the destination host and releases them on the source host, ensuring efficient VM deployment and migration.

The Service Manager continuously monitors virtual services to maintain service quality, analyzing response times and resource scheduling. If services fall short of SLA criteria, it triggers migration to optimize performance and ensure compliance with SLA parameters.

The Network Manager is essential for ensuring optimal communication and resource usage within a data center. It continuously monitors bandwidth utilization to identify and resolve congestion, ensuring that VMs and services perform efficiently. The manager also allocates bandwidth to VMs, particularly in data centers with limited network resources and multiple running VMs. Additionally, the Network Manager oversees all external communication links, including internet access, third-party cloud services, and connections to other data centers. By managing these links effectively, it prevents network faults from disrupting operations and ensures sufficient network capacity for VM migrations.

The VM Migration Manager works closely with the Scheduler and Allocation Manager to monitor VM status and generate lists of available hosts for local and WAN migrations. It makes scheduling decisions and executes the migration process by interacting with the destination data center’s cloud manager, accessing VMM resources, and initiating the migration via an API call.

Scheduling techniques can be classified into static scheduling such as Min-Min algorithm [26, 27] and PSO based algorithm [28], Dynamic Scheduling such as the Dynamic Round-Robin algorithm [29,30,31], and Heuristic Scheduling by applying optimization algorithms such as the Genetic Algorithm (GA) and Particle Swarm Optimization [32]. Deep Reinforcement Learning [33,34,35,36] is used in the development of scheduling strategies for VMs in cloud data centers to optimize the scheduling strategies and to reduce their delays.

This research proposes a new distributed cloud management model that employs blockchain as a secure registry for sharing data center information required for VM scheduling across multiple data centers.

Blockchain-based data sharing model for distributed cloud management

Migrating VMs between data centers using traditional management techniques can sometimes result in imprecision and excessive VM movement, reducing the overall efficiency and efficacy of the migration process. To improve resource usage and reduce expenses, a distributed data sharing model based on blockchain is proposed. This model allows secure and adaptive sharing of VM status across all data centers, ensuring that all participating data centers maintain a consistent and up-to-date VM state. Instead of relying on a single global manager, each data center has its own distributed cloud manager that oversees both internal and external operations. In the distributed cloud management model, several cloud data centers are connected via WAN, as shown in Fig. 2. Because each data center has a synchronized copy of the states of all other data centers, the WAN migration decision-making process is expected to be more precise, resulting in fewer VM migrations and eliminating migration errors commonly found in other models.

Distributed management model view

Full size image

The internal modules of the distributed cloud manager in a data center are depicted in Fig. 3. It comprises the five primary modules: Resources Scheduler Manager, Allocation Manager, Service Manager, Network Manager, and VM Migration Manager, along with the Blockchain Service module. The distributed manager follows a specific set of steps, as shown in Algorithm 1, in order to make a migration decision. This results in the movement of VMs either within the same data center or from one data center to another via a WAN.

Distributed cloud manager internal modules

Full size image

Algorithm 1 Distributed manager algorithm

The distributed model is capable of efficiently managing both local migrations within a data center and WAN migrations between data centers, thereby preventing migration errors that can arise with alternative management methods. With only one communication message needed to trigger a VM migration process and update the blockchain register (\(N_m = 1\)), the total number of communication messages exchanged in the distributed model (\(N_{com}\)) is equal to the number of VM migrations. This can be calculated using Eq. 1.

where \(N_{mig}\) represents the total number of VM migrations including both local and external migrations, \(N_{mig\_W}\) represents the total number of WAN VM migrations, and \(N_{mig\_L}\) represents the total number of LAN VM migrations.

Equation 2 is used to calculate the total communication delay (\(D_{total}\)) required to carry out all VM migrations. It is worth noting that this equation takes into account the convergence delay (\(T_{conv}\)) needed to synchronize changes on all blockchain registers. This delay is only applicable when migrating VMs over WANs.

where \(T_{com}\) represents the transmission and processing time for a single communication message. In the distributed model, \(T_{com}\) remains constant for both WAN and local VM migrations (\(T_{com} = T_{com\_W} = T_{com\_L}\)).

Our proposed distributed model aims to enhance the cloud management system by integrating a Blockchain module with the VM Migration Manager. This enables secure sharing of data centers and VM states with other data centers, leading to more efficient distributed management.

The blockchain serves as a decentralized ledger that locally stores the status of each data center in the distributed architecture. Each center reports its status to the blockchain ledger, which then reconciles the statuses with other nodes across other data centers. To accomplish this, the blockchain module leverages the BigchainDB framework [25], which is compatible with decentralized computing platforms and applications. BigchainDB seamlessly integrates the benefits of distributed databases and traditional blockchains by utilizing an enterprise-grade distributed database (MongoDB) [37] and a production-ready consensus engine (Tendermint) [38]. This powerful combination allows for high transaction rates, low latency, indexing, querying of structured data, and incorporates blockchain characteristics such as decentralization, immutability, and owner-controlled assets.

The latest version of BigchainDB, version 2.0, incorporates Tendermint for networking and consensus, ensuring Byzantine Fault Tolerance (BFT). Every node is equipped with its own MongoDB database and communication is facilitated through Tendermint protocols, as depicted in Fig. 4 that illustrates a four-node BigchainDB 2.0 network comprises various interconnected components. The system’s level of security is exceptionally high, as a hack attempt would, at worst, result in the corruption or deletion of data from a single local database. Additionally, Tendermint’s current block proposer changes with each round, creating a decentralized network in which each node is owned and managed by a distinct entity. To prevent single-point failures, it is recommended to distribute nodes across several locations and hosting providers. Remarkably, even if one-third of the nodes become inactive, the network will continue to function without interruption [25].

Four-Node BigchainDB 2.0 network [25]

Full size image

BigchainDB offers a robust solution for safeguarding stored data, rendering it nearly impossible to alter or erase. The data is essentially immutable, with each node containing a comprehensive record of information kept in a specialized MongoDB database. Additionally, cryptographic signatures provide an added layer of protection for transactions, as any changes made to the contents will be detected upon verification. These measures collectively ensure unparalleled safety and security for stored data. The transfer of an asset can only be initiated by the rightful owner(s) possessing the necessary set of private keys. This means that not even a node operator can transfer an asset without the owner’s authorization.

Tendermint consensus protocol has gained notoriety for its remarkable efficiency in transaction processing and commit times. With the ability to manage several thousand transactions per second, it maintains a commitment latency of one to two seconds. Moreover, the protocol has demonstrated its resilience under challenging adversarial scenarios, such as validators crashing or broadcasting maliciously generated votes.

The use of the Tendermint consensus algorithm in BigchainDB networks has been observed to yield exceptional performance, with transactions being added to new committed blocks in mere seconds or less. Additionally, the Tendermint consensus mechanism provides a robust assurance that once a transaction is included in a block, it cannot be reversed or declared defunct in the future, as forking is not supported.

In a BigchainDB 2.0 network, each node is equipped with its own local MongoDB database. This feature provides node operators with complete access to MongoDB’s powerful indexing and querying capabilities. The database supports JSON strings for transactions, assets, metadata, and blocks. Node operators have the flexibility to determine the level of power they want to allocate to external users based on their own preferences. For instance, some node operators may choose to index geospatial data and offer optimal geospatial searches via a REST API, while others may prefer to provide a GraphQL API. While BigchainDB 2.0 automatically generates some MongoDB indexes and the BigchainDB HTTP API comes with a few default endpoints for basic queries, any node operator can add more indexes and query APIs to their node.

When it comes to blockchain networks, there are certain differences in the way they handle the addition of nodes to the network. While some networks, like Bitcoin, allow anyone to add a node to the network, others, such as BigchainDB networks, are managed by the network’s governing organization. This management approach makes it difficult for Sybil attacks to be effective within the network. In the case of Bitcoin, Sybil attacks are not impossible, but they are prohibitively expensive. This is due to the network’s design, which makes it challenging to accumulate enough resources to successfully carry out such attacks. However, in the case of BigchainDB networks, the network management approach means that Sybil attacks are less of a concern altogether.

VPN-based centralized management model

The cloud centralized management model is widely employed in data centers to assess cloud resources and evaluate their orchestration capabilities. In comparing this conventional approach with our proposed model, we aim to manage cloud resources and assess the enhanced significance of cloud management performance and availability. Our chosen centralized approach seeks to minimize management overhead related to data integrity and synchronization by maintaining a single database at a central location, which houses all data center statuses and is regularly updated with new changes.

In centralized management models, a global manager oversees external operations and communication between data centers. As shown in Fig. 5, the global cloud manager connects to various data centers via VPN over a WAN. Within this framework, a local cloud manager is assigned to oversee the daily operations of each data center. Figure 6 illustrates the internal modules of a local cloud manager in a centralized setup, where all critical modules responsible for managing the core functions of the cloud data center are housed. A communication module within the local manager ensures seamless connectivity with the global manager, transmitting registered resource states to facilitate external actions, such as WAN-based VM migration. The global cloud manager obtains information about the data center’s status and saves it in a database. To perform a VM migration from one data center to another, the global manager follows the procedures outlined in Algorithm 2 and initiates the migration over the WAN.

Centralized management model view

Full size image

Local cloud manager internal modules in the centralized management model

Full size image

Algorithm 2 Centralized global manager algorithm

The centralized database is kept up-to-date with partial information regarding VMs that are selected for WAN migrations from their current data center. This updating process is triggered when the list of populated hosts is empty, signifying that the VM must be relocated. To optimize server consolidation, power consumption, and resource utilization, a dynamic algorithm based on priority [39] is employed along with migration facilitator scheduling. The migration facilitator prioritizes and categorizes VMs based on their involvement in host allocation. In this system, VMs with higher migration orders are assigned to hosts with fewer free resources.

In the centralized global manager, a strategy is utilized to make WAN migration decisions based on data center availability. However, as the manager lacks complete access to all data center statuses, decisions may be inaccurate, leading to a series of VM migrations that are necessary to stabilize the system. Additionally, since VM statuses can change locally without the manager’s knowledge during WAN migration decisions, incorrect decisions can be made, resulting in what we refer to as WAN migration errors. On the other hand, in the distributed approach, the blockchain acts as a local register that maintains the status of all data centers. Each center modifies its status on the blockchain register, which is then synchronized with other blockchain nodes across different data centers using the consensus protocol method. This approach eliminates the possibility of errors.

Equation 3 calculates the overall count of communication messages that are exchanged within the centralized model, denoted by \(N_{com}\). Equation 4 calculates the comprehensive delay required to transmit and process all communication messages involved in every VM migration, quantified as \(D_{total}\). It is worth noting that this delay encompasses the duration necessary for transmitting and processing all communication messages, influenced by different factors, including the network speed, the execution speeds of both local and global cloud managers.

where \(N_m\) denotes the required number of messages to complete a single WAN VM migration, \(N_{mig\_W}\) represents the total number of WAN VM migrations, \(N_{mis}\) signifies the total number of VM misplacements in WAN migrations, and \(N_{mig\_L}\) represents the total number of local VM migrations.

where \(T_{com\_W}\) is the transmission and processing time of one communication message transmitted between the local and global managers. \(T_{com\_L}\) is the local transmission and processing time for a local communication message that is done locally within the data center without reaching the global cloud manager.

To validate our proposed models, we used a combination of real-world implementation and simulation methods. We built and ran the experimental environment at the Gina Cody School HPC Facility, known as Speed [40]. The infrastructure comprises 24 nodes with 32 cores each, along with 512 GB of memory and 1 TB of volatile-scratch disk space. Additionally, there are 12 NVIDIA Tesla P6 GPUs with 16 GB of memory and one AMD FirePro GPU with 8 GB. The facility also offers Singularity containers (user-managed containers) and support for a number of machine learning and deep learning frameworks, such as TensorFlow, OpenCV, OpenMPI, OpenISS, MARF, and OpenFOAM.

Multiple containers have been constructed to simulate the operation of data centers, each capable of handling different numbers of hosts and distributed VMs. The simulators are designed to execute both local and cross-data-center VM migrations, with a greater emphasis on scheduling and WAN VM migration as the percentage of WAN migrations increases. For the experiment of this study, we consider a single service provider to guarantee the compatibility and integrity of shared data. But there is nothing preventing the application of the proposed work in multiple providers’ data centers if the distributed platform is set properly.

We utilized containers to implement both the centralized and distributed proposed models. These containers contained all the necessary services, including resource scheduling, allocation management, service management, network management, and migration management. Implementing REST API as internal communications channel between the service components. To simulate LAN and WAN network connections between data centers, we relied on the open-source WAN emulator WANem [41]. WANem is a highly advanced application that can simulate various WAN scenarios, such as network latency, packet loss, corruption, disconnections, reordering, and jitter for data/voice traffic. Additionally, it serves as a transparent application gateway and is licensed under the GPL v2.

The simulation setup for the distributed model is depicted in Fig. 7. BigchainDB platform is utilized for blockchain technology. To establish a distributed system, we employed containers to execute BigchainDB container images as blockchain nodes. Each data center simulator container is connected to a BigchainDB node, which functions as a container image. Container bridging is facilitated through the WANem container, linking each data center to a BigchainDB container.

Distributed management model simulation setup

Full size image

Table 2 shows the extensive examination case scenarios of centralized and distributed models for multiple data centers, hosts, and VMs. Based on statistical measurements of real cloud data centers, where it shows that the number of migrated VMs is between 10% to 30% of the total number of VMs [42]. The migrations were simulated to be approximately 10% of the total number of VMs in all scenarios, with around 90% of these migrations occurring over the WAN.

To simulate the centralized global manager, a container with a MongoDB database is utilized. This database effectively tracks the states of VMs distributed across multiple data centers, updating them as migrations occur. Figure 8 illustrates the configuration of centralized model for simulation. The data center simulators are linked to the global cloud manager through WAN connections, which are mimicked by the WANem container. To lend greater realism to the experiment, all data transfers are emulated as if they were sent via VPN and SSH connections.

Centralized management model simulation setup

Full size image

The suggested distributed model involves selecting a VM to be moved from its current host. The scheduler manager collaborates with the allocation manager to identify available resources on local hosts that can accommodate the potential migration. If there are no available resources, the allocation manager and migration manager work together to conduct allocation optimization techniques. The next step is to determine a suitable destination data center and host for the VM migration through a WAN. The distributed cloud manager then communicates with the cloud manager at the destination data center and initiates the VM migration process by making API calls to the VMM on the destination host through network channels. Finally, the migration process is recorded and updated on the BigchainDB node, and a synchronization mechanism ensures that all BigchainDB nodes receive the modification.

In the centralized approach, if a VM cannot be moved to another host within the same data center, the local cloud management system will send a request for WAN migration to the global manager. The global manager will then gather updates on the status of all data centers, use optimization algorithms to determine the best location for the VM, and communicate this information to both the original and destination data centers to initiate the migration process. In case the destination data center or host experiences any changes during the decision-making process, which may lead to migration errors, the optimization algorithms will be repeated until a suitable destination with adequate resources is found.

Within the centralized management model, the WAN migration requests are received by the global cloud manager. The manager then contacts each data center to assess their local status. Based on the reports received from the data centers, the global cloud manager selects the host and destination data center for the migration. This choice is then communicated to both the source and destination data centers. This communication process is repeated for each WAN migration, resulting in a considerable volume of messages exchanged between data centers and the global manager. Figure 9 illustrates the total number of messages exchanged during VM migrations with various numbers of data centers and hosts under the centralized management model.

Total number of exchanged communication messages in the centralized model

Full size image

Within the distributed management model, the choice to transfer WAN is determined based on the status of data centers available in the local blockchain registry. After this determination, the distributed cloud manager sends a message to the corresponding manager at the destination data center, which initiates the migration process. A single communication message is transmitted to make the appropriate migration decision for each VM migration. Figure 10 illustrates the total number of communication messages required for the distributed management model, as determined by the number of VM migrations.

Total number of exchanged communication messages in the distributed model

Full size image

Figure 11 illustrates the cumulative delay that arises during message exchanges within the centralized model. Successful WAN VM migrations necessitate multiple communication messages between the global manager and data centers. With an increased number of VM migrations, an elevated number of messages are required to manage these operations, leading to prolonged processing times. Furthermore, migration defects can exacerbate this delay, resulting in disruptions in various scenarios. VM misplacements occur when the global manager makes incorrect migration decisions, leading to multiple migrations of the VM before it reaches a stable destination.

Total communication delay in the centralized model

Full size image

In Fig. 12, the correlation between the total number of VM migrations and the overall delay of all communication messages in the distributed model is illustrated. This trend is similar to the centralized model, where varying numbers of VMs are distributed across multiple hosts and data centers. As more VM migrations occur, more communication messages are sent, resulting in longer processing times for all VM migrations. Nevertheless, our research uncovered that the total delay for each scenario is notably less than the centralized model. The entire delay for each VM migration process encompasses the migration decision processing time and the blockchain node’s convergence time, which synchronizes the states of all blockchain nodes deployed across all data centers. Tendermint-based networks, like BigchainDB networks, can add a transaction to a new committed block within seconds or less ([25]). Consequently, the total delay for processing a VM migration is shorter than that of the centralized model, which involves more communication messages, resulting in a longer total delay. Furthermore, the absence of VM misplacements in the distributed management model, as well as the fixed number of communication messages that equals the number of VM migrations, as shown in Fig. 10, signifies that the total communication delay remains undisturbed across different cases. This is in contrast to the centralized management model behavior described in Fig. 11.

Total communication delay in the distributed model

Full size image

Figure 13 compares total communication messages between distributed and centralized approaches. The results show that the centralized architecture requires much more communication messages to accomplish all migrations for a given number of VMs across many hosts and data centers. Compared to the centralized model, the distributed architecture significantly reduces communication messages by 41.79% to 49.85%. This is because the centralized global migration manager must exchange several messages with the data centers to complete each VM transfer.

Total number of exchanged communication messages

Full size image

When comparing the centralized and decentralized VM migration models, it is evident that the former has a much longer total delay due to the increased amount of communication messages sent between data centers and the centralized global migration manager. Figure 14 shows that the distributed model reduces total delay by 2% to 12% compared to the centralized form. It is important to note that the magnitude of this reduction is determined by a variety of parameters, including the total number of WAN migrations and the number of VM misplacements that occur in the centralized management architecture.

Total communication for different number of data centers

Full size image

During WAN migration in the centralized model, any VM misplacement can cause the migration processes to be repeated, which in turn leads to an increase in communication messages that need to be exchanged. This, in turn, results in additional delays to the total processing time for a specific number of WAN VM migrations. Figure 15 illustrates the number of misplacements that arise from various numbers of VM migrations across different data centers and hosts. The occurrence of VM misplacements may be attributed to the inability to access the most up-to-date and comprehensive status of VMs in data centers. This can lead to inaccurate decisions regarding VM migrations, resulting in a series of migrations in order to achieve a stable state. The random nature of these VM misplacements can cause significant fluctuations in the total number of misplacements, as illustrated in Fig. 15. However, as shown in the figure, the average number of VM misplacements generally rises with an increase in the number of VM migrations, as well as with the number of data centers and hosts. These three metrics are crucial in determining the optimal destination for VM migration. When considering a larger number of VM migrations across multiple data centers and hosts, a balanced decision on the final destination for the migrated VM must be made, which takes into account the impact on scheduling decisions.

VM misplacements for different number of VM migrations

Full size image

Figure 16a and b show the impact of VM misplacements on the total number of communication messages and the total communication delay to process all VM migrations. The number of misplacements has a direct impact on the number communication messages, as well as the overall delay, for a specific number of VM migrations, regardless of the number of data centers and hosts involved. In each scenario of VM migrations, the total communication delay and number of messages both increase proportionally with the number of VM misplacements.

Centralized model VM misplacements impact

Full size image

Our innovative solution provides a streamlined and effective approach that reduces management burdens and inefficiencies, outperforming conventional centralized methods. We utilize blockchain technology to facilitate data center information sharing, minimizing the amount of communication messages exchanged and thereby reducing processing delays.

Through the use of statistical and random variable analysis, we have examined the two models based on the central limit theorem. This allows describing the behavior of the two systems by calculating the Z-score of the normal random variable distribution, which accurately predicts and match the characteristics of both the centralized and distributed models.

Our analysis measured the probability of the Z-score for various factors, including the number of VMs migrated, communication message exchanges, and delay. To determine the total delay and number of exchanged messages for each simulation case, we conducted an experiment for each scenario. As per the theorem, if a random sample of size n is taken from a population with a mean of \(\mu\) and a finite variance of \(\sigma ^2\), and X represents the sample mean, then the distribution in Eq. 5 approaches the standard normal distribution as n approaches infinity [43].

If we have two independent populations with means \(\mu _1\) and \(\mu _2\) and variances \(\sigma _1^2\) and \(\sigma _2^2\), and if \(X_1\) and \(X_2\) are the sample means of two independent random samples of sizes \(n_1\) and \(n_2\) from these populations, then the sampling distribution of the Z-statistic is given by Eq. 6:

In statistical hypothesis testing, the Z-statistic, denoted by Z, is a fundamental tool for assessing the significance of observed differences between sample means and population means. The validity of this approach hinges on the application of the Central Limit Theorem (CLT), which provides a framework for approximating the distribution of sample means as normal, given certain conditions. Specifically, when the underlying populations are normal, the sampling distribution of the Z-statistic is known to follow a standard normal distribution, with all the associated properties of mean and variance. This property underpins the use of Z-tests in a variety of statistical applications, including quality control, experimental design, and survey analysis.

We utilized statistical analysis to validate the system by employing a histogram that was mapped to Kernel Density Estimation (KDE). From the output probability density function (PDF), we were able to determine the CDF of the distribution. To estimate the probability density function of a continuous random variable based on a sample, we utilized the KDE formula. This formula utilizes kernels (window functions) to smooth the data and approximate the distribution underlying the sample. Assuming \(X_1\), \(X_2\), …, \(X_n\) are the data points observed from a continuous random variable X, we used Eq. 7 to estimate the probability density function f(x) using KDE.

where: \(\hat{f}(x)\) is the estimated probability density function at point \(x\), \(K(u)\) is the kernel function, \(h\) is the bandwidth, and \(n\) is the number of data points in the sample.

The probability density function, represented by the symbol f(x), is an essential tool for describing the probability distribution of a continuous random variable X. In situations where a specific range is more likely to contain a value for X, the probability is higher, and in turn, f(x) takes on larger values. To calculate the probability that X will fall within a particular range, such as between a and b, we can integrate f(x) from a to b. The probability density function (PDF) for a continuous random variable X is denoted by f(x) and is defined in Eq. 8.

where \(a\) and \(b\) are any two real numbers such that \(a \le b\).

The confidence interval for the two model measurements’ population sampling validation was determined and the interval for the population mean was calculated using Eq. 9, assuming knowledge of the population variance. These findings contribute to a more accurate and robust understanding of the underlying population parameters and can aid in improving the reliability of research outcomes.

where \(\bar{X}\) is the sample mean, \(\sigma\) is the population standard deviation, \(n\) is the sample size, and \(Z\) is the critical value from the Z-distribution corresponding to the desired confidence level.

Figure 17 displays the mean of the total number of communication messages in the distributed model, with its confidence interval. The sampled data is an exact match with the population, resulting in a 100% confidence level for the system. The mean of the total number of communication messages in the centralized model is shown in Fig. 18, with a sampling error falling within the acceptable range of 0.018 to 0.024.

Total number of exchanged communication messages in the distributed model

Full size image

Total number of exchanged communication messages in the centralized model

Full size image

Figure 19 displays the mean total delay of communication messages along with confidence intervals in the distributed model, which has an error range of 0.002 to 0.004. Similarly, Fig. 20 shows the mean total delay of communication messages with confidence intervals in the centralized model, which has an error range between 0.09 to 0.30. The results clearly indicate that the error range in the distributed model is narrower than the centralized model, making it more precise and less uncertain.

Total delay of communication in the distributed model with confidence interval

Full size image

Total delay of communication in the centralized model with confidence interval

Full size image

We gathered data on the processing time of every communication message exchanged in both models and used it to calculate the KDE of the observations for four different cases of VM migrations (100, 200, 300, and 400 migrations). As depicted in Fig. 21a, the KDE density plot for each case of VM migration is strikingly similar, with the majority of observations concentrated on the left side of the graph, indicating that they require less processing time.

Similarly, the centralized model yields the same result. Figure 21b displays the KDE density plot for the same four cases of VM migrations, with all cases exhibiting highly comparable processing delay distributions, again skewed towards the left side of the graph, indicating the need for less processing time.

KDE density diagram for different number of VM migrations

Full size image

Our proposed solution offers a streamlined and efficient approach, reducing management complexities by leveraging blockchain technology for seamless information sharing between data centers and edge nodes. This reduces the number of communication messages and processing delays, enhancing overall system performance.

Experimental results highlight the advantages of a blockchain-based distributed management model, particularly in large-scale cloud environments. Compared to existing technologies, our model excels in minimizing the number of communication messages exchanged and reducing decision-making delays, such as those required for VM migrations. Additionally, the blockchain’s built-in security features eliminate the need for additional security layers, safeguarding the privacy of shared data.

However, the solution faces limitations in low-scale environments, where a higher volume of messages is required for management decisions. Furthermore, the blockchain model is used without customization or control over its low-level functionalities, potentially limiting performance optimization.

Future work will focus on deploying our solution in a real production cloud environment to assess its integration with other cloud management components and to identify potential performance overheads. Additionally, we plan to enhance the system architecture to enable seamless operation across diverse cloud management systems.

In this research paper, we introduced a novel distributed cloud resource management model that utilizes the BigchainDB framework as a blockchain registry. This registry enables secure and efficient sharing of information about data centers, which facilitates quick and reliable scheduling decisions for various cloud data centers. Our proposed distributed model approach requires less management overhead and delay than centralized solutions. The proposed decentralized model decreases the number of exchanged communication messages by 41.79% to 49.85% and reduces total communication delay by 2% to 12% in comparison to the centralized model. These findings showcase the efficacy of the suggested model and its capacity to enhance the effectiveness of cloud systems management. Our main contribution lies in evaluating the performance, particularly in terms of communication delays and the number of communication messages exchanged. We plan to address the security concerns, including real-world examples of log tampering, in our future work, where we will analyze potential threats and develop countermeasures.

No datasets were generated or analysed during the current study.

VM:

Virtual Machine

WAN:

Wide Area Network

VPN:

Virtual Private Network

IoT:

Internet of Things

API:

Application Programming Interface

EDR:

Edge Demand Response

VCG:

Vickrey-Clarke-Groves

QoE:

Quality of Experience

QLDB:

Quantum Ledger Database

VMM:

Virtual Machine Manager

GA:

Genetic Algorithm

CLT:

Central Limit Theorem

KDE:

Kernel Density Estimation

PDF:

Probability Density Function

The authors would like to acknowledge the financial support provided by Natural Sciences and Engineering Research Council (NSERC)’s Discovery Grant, Canada.

The authors would like to acknowledge the financial support provided by Natural Sciences and Engineering Research Council (NSERC)’s Discovery Grant, Canada.

1. Tariq Daradkeh and Anjali Agarwal contributed equally to this work.

Authors and Affiliations

1. Department of Electrical and Computer Engineering, Concordia University, Montreal, H3G 1M8, Quebec, Canada

   Mohammad A. Altahat & Anjali Agarwal

2. Department, Yarmouk University, Irbid, 21163, Irbid, Jordan

   Tariq Daradkeh

Contributions

All authors contributed equally to the research work in this manuscript.

Corresponding author

Correspondence to Mohammad A. Altahat.

Competing interests

The authors declare no competing interests.

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.

Reprints and permissions