Dynamic Adaptation Using Deep Reinforcement Learning for Digital Microfluidic Biochips

The rapid worldwide spread and impact of the COVID-19 virus has created an urgent need for reliable, accurate, and affordable testing on a massive scale. For example, the National Institutes of Health (NIH) has launched the Rapid Acceleration of Diagnostics (RADx) initiative to develop and implement technologies for COVID-19 testing [54]. One of the most promising technologies for realizing this goal is digital microfluidics. A microfluidic biochip (DMFB) manipulates tiny amounts of fluids to automatically execute biochemical protocols for point-of-care clinical diagnosis with high efficiency and fast sample-to-result turnaround [16, 65, 74]. Because of these characteristics, the RADx initiative has awarded grants to several biomedical diagnostic companies to develop microfluidic technologies that could dramatically increase testing capacity and throughput [53, 56]. Other applications of DMFBs include screening of newborn infants [33, 69], drug discovery [38], and clinical diagnostics [8, 62].

A DMFB consists of an electrode array in two dimensions that controls the movement of discrete liquid droplets. Upon actuation by a sequence of control voltages, the electrode array can perform a variety of fluidic operations, such as dispensing, mixing, and splitting [7, 25]. Figure 1(a) shows a DMFB in which two droplets are present on a patterned electrode array. Nanoliter droplets on this platform are transported using the principle of Electrowetting-on-Dielectric (EWOD) [60]. This principle refers to the modulation of the interfacial tension between a conductive fluid and a solid electrode coated with a dielectric layer through the application of an electric field between them. See Figure 1(b).

Illumina commercialized digital microfluidics for sample preparation in 2015 through NeoPrep—a nearly $40K instrument that automates the preparation of up to 16 sequencing libraries at a time [31]. Genmark has also deployed the microfluidic technology for infectious disease testing [59], and Baebies uses this technology to detect lysosomal storage diseases in newborns [26].

However, reliability remains a major concern in DMFB systems. Illumina halted the sale of NeoPrep in February 2017. In its letter to customers, Illumina cited reliability issues in-house and far worse ones in the field. Even though biochips are tested after production, defects such as electrode degradation can occur during system lifetime [13, 71]. As the electrodes are actuated over time, two types of electrode degradation might happen: charge residual and charge trapping. Charge residual is caused by the accumulated charges, which can be mitigated by inserting grounding vectors [58]. Charge trapping is when the charges are trapped in the dielectric insulator, and this phenomenon is irreversible [5]. A consequence of electrode degradation is that droplet movement is impeded [72]. An example of electrode degradation is shown in Figure 2. The figure shows two droplets on the biochip—one located on a degraded electrode. Two electrodes are actuated to move these droplets. However, one of these operations fails because the degraded electrode exerts additional surface-tension force. Detailed analyses of the relationship between electrode defects and fluidic operations can be found in the work of Drygiannakis et al. [14].

In a typical use model for DMFBs [70], a bioassay protocol with fluidic operations is obtained from biologists. Next, a synthesis technique maps these operations to groups of electrodes, referred to as fluidic modules, of a biochip to perform the required operations [4]. A droplet has to be transported from one module to the next. The problem of determining droplet transportation paths between modules is referred to as droplet routing. A number of droplet routing techniques have been proposed in the literature for bioassay applications [73, 82, 86]. Su et. al [73] proposed the first systematic droplet routing approach, which adopted the Lee algorithm and minimized the number of electrodes used for droplet routing. Xu and Chakrabarty [82] proposed a droplet routing aware synthesis tool, which was based on parallel recombinative simulated annealing. Zhao and Chakrabarty [86] proposed an integer linear programming based method to co-optimize droplet routing and pin mapping.

However, these methods overlook the fact that transportation of droplet may fail if the electrodes on the routing path degrade over time.

Example. Figure 3(a) shows a pre-computed routing path. We can see that this route is the shortest path between the start and the destination points. Droplet transportation can be successful because the biochip is healthy (i.e., no electrode degradation has occurred). Conversely, Figure 3(b) shows that droplet transportation to the destination fails because degraded electrodes exist in the associated path. If an online droplet router knows the locations of the degraded electrodes, it can generate another route that involves only healthy electrodes. An alternative route is shown in Figure 3(c); note that this is a shortest path, and it avoids electrodes that are degraded.

In Figure 3, a different color is used to indicate the degraded electrodes. However, in reality, we cannot identify degraded electrodes by simple examination; this is because the degradation process results from charge trapped in the insulator. When routing errors occur, simply replacing the degraded DMFB with a new one will not only increase the cost but also lead to undesirable wastage of biosamples. Droplets that are in the middle of an unfinished operation, such as mixing or diluting, need to be abandoned. The wastage of droplets is particularly undesired in some applications, such as newborn screening [32] and forensic analysis [83], since the bio-examples are limited in volume and availability. For example, in the newborn screening test provided by Baebies Inc., the entire screening test contains 10 to 20 different assays and each assay needs 100 nl of dried blood spot extract [32]. Thus, a newborn screening test needs at least 1,000 nl of dried blood spot extract, which needs 200 to 300 \(\mu\) L (4–6 drops) of whole blood [3]. Prior work has led to synthesis methods that prevent excessive usage of a few electrodes by evenly distributing fluidic operations to multiple electrodes [5, 88]. However, these methods can only postpone the occurrence of electrode degradation, which still happens as electrodes are actuated over time. If such electrode degradation happens during bioassay execution and a route is associated with degraded electrodes, bioassay execution will fail, and it will need to be re-executed on a new biochip [29]. Furthermore, the locations of degraded electrodes may vary from biochip to biochip because the electrode degradation process is affected by geometric variations and different electrode actuation times [24].

Several methods have been proposed to perform error recovery when routing tasks fail [1, 40, 66]. However, these methods are focused on recovery after routing failures, and they do not proactively alleviate the occurrence of erroneous behaviors caused by electrode degradation. Recently, an RL-based routing framework was developed to identify degradation-aware routing strategies for Micro-Electrode-Dot-Array (MEDA) biochips [15]. However, this method cannot be used for DMFBs due to the inherent difference between DMFBs and MEDA biochips: MEDA biochips provide the real-time degradation status of each electrode using built-in sensing circuits. This is not the case of non-MEDA DMFBs. In this work, we adopt RL techniques to respond to the dynamic degradation environments, which is not possible with existing offline routing methods.

Numerous papers have been published in recent years to advance applications that leverage RL theory [9, 11, 49]. Our work aims to introduce RL to a new application—that is, the droplet routing problem on DMFBs. We target an RL formulation for the droplet routing problem to address the dynamic degradation of electrodes. An RL-based droplet router addresses the electrode degradation problem and ensures reliable bioassay executions in three ways. First, it provides real-time decision for droplet routing. Second, it can “learn” from the prior experience associated with electrodes that start malfunctioning. Therefore, the droplet router can generate routing paths that include any healthy electrodes. Third, even though the degradation processes may differ for two DMFBs, the router can generate different, yet reliable, routing paths on distinct DMFBs for the same routing objective.

This article represents one of the first attempts to map RL to clinical microfluidic systems. The main contributions of this work are as follows:

Multiple droplet routing tasks can be parallel executed on a DMFB at the same time. Assume that there are n droplets in total, where \(n \ge 2\). For any two of the droplets \(d_{i}\) and \(d_{j}\), where \(1 \le i,j \le n\) and \(i \ne j\), assume that their positions at timestep t are \(e_{x^{t}, y^{t}}\) and \(e_{k^{t}, m^{t}}\), respectively. The following are the fluidic constraints that should be satisfied [6]:

With the preceding constraints, the objective of the multi-droplet routing problem is to minimize the maximal routing timestep among all the n routing tasks.

For the droplet routing problem that considers electrode degradation, we define function \(d(e_{i, j})\) to describe the degradation status of an electrode, where \(0\le d(e_{i, j})\le 1\). \(d(e_{i, j})\) is 1 when the electrode \(e_{i, j}\) is completely healthy. The modeling of the degradation status function will be explained in Section 5.3. Note that the value of the electrode status function cannot be obtained by the users during the execution since the degradation status is not measurable. As an electrode degrades, the success rate of droplet transition decreases. A failing transition causes the droplet to stay in the same position. For an electrode with the degradation status \(d(e_{i, j})\), we assume that the success rate of transition is \(d(e_{i, j})\) and the expected steps for a success transition is \(1/d(e_{i, j})\). Therefore, the objective of a droplet routing problem with electrode degradation can be formulated as following: find a path \(\lbrace e_{x_1, y_1}, e_{x_2, y_2}, \ldots , e_{x_T, y_T}\rbrace\) that minimizes \(\sum _{i=1}^{T} 1/d(e_{x_i, y_i})\),where \(e_{x_1, y_1}\) the is the source and \(e_{x_T, y_T}\) is the destination.

An agent in an RL formulation is placed within an environment. The agent’s goal is to accomplish a using task with the best performance given a set of small actions. At each step, the agent takes one of these actions, and the agent receives an observation and reward from the environment [75].

RL problems can be formally stated using Markov decision processes. A Markov decision process contains two sets (S and A), a probability function f, a reward model R, and a variable \(\gamma\). The observations made by the agent are included in a set S, and an observation is also referred to as a state. An element \(s_t\in S\) is an observation made by the agent at time t. We use A to denote a set of actions taken by the agent. An action \(a_t \in A\) denotes the action made by the agent at time t. Note that \(P(s_{t+1}|a_t, s_t)\) refers to the transition model; it describes what the next state \(s_{t+1}\) will be after the agent takes action \(a_t\) while in state \(s_t\). The reward model is denoted by \(R(s_t)\); it describes the agent’s reward when it enters the state \(s_t\). The parameter \(\gamma\) is a discount factor, where \(0\le \gamma \le 1\) and \(\gamma \in \mathbb {R}\). It represents the relative importance of immediate and future rewards. The agent’s goal is to select the best policy \(\pi\) that maximizes the total reward received from the environment from the start state to an end state. The expected cumulative discounted reward is expressed as \(U(t) = \mathbb {E}[\sum _t \gamma ^t \cdot R(s_t)]\).

We briefly describe three deep RL algorithms that we use to evaluate our RL framework; these algorithms are Temporal-Difference (TD), on-policy gradient descent, and off-policy actor-critic approaches.

We consider three widely used training schemes for our MARL framework: centralized, concurrent, and parameter sharing [20]. We briefly describe how each approach can be used with MARL.

Centralized. The centralized learning approach assumes a joint model that receives all the observations and generates the joint actions for all the agents. A drawback of this approach is that it leads to exponential growth in the observation and actions spaces with the number of agents.

Concurrent. In concurrent learning, each agent learns its own individual policy. Each independent policy maps an agent’s private observation to an action. In the policy gradient approach, this means optimizing multiple policies simultaneously from the joint reward signal.

Parameter Sharing. Similar to concurrent learning, each agent is assigned with a neural network policy. However, in the parameter sharing approach, all the agents share the parameters of a single policy. This allows the policy to be trained with the experiences of all agents simultaneously. However, each agent is still able to act differently based on the observation it receives.

We formulate droplet routing as a sequence of decision-making problems within the RL framework. We utilize a droplet routing agent that can make real-time observations of the DMFB, it can move a droplet to an adjacent electrode at a timestep, and the agent’s goal is to transport the droplet from a given start electrode to a given destination electrode. The agent is rewarded or punished based on the state transition result after it takes an action.

Actions. At any timestep, a droplet can be transported to one of the four directions: north, south, east, and west. Therefore, we define the action set as \(A=\lbrace a_n, a_s, a_e, a_w\rbrace\); each element denotes a direction along which the droplet can be moved.

States. A state \(s_t\) consists of the location of the transported droplet, the droplet destination, and electrodes that are concurrently utilized by other fluidic operations. During a bioassay, multiple operations may be carried out concurrently to achieve high throughput. If a droplet is moved while a mixing operation is also being carried out, the set of electrodes used for the mixing operation cannot be used for droplet transportation in order to prevent undesirable contamination.

At any given timestep, observation made on the DMFB is processed as an RGB image. Control software is used to determine the locations of on-chip droplets [45]. The resolution of the RGB image is given by the number of electrodes on the DMFB. An electrode with a droplet on it is interpreted as a blue pixel. The destination electrode is interpreted as a green pixel. The electrodes occupied by all the other concurrent operations are interpreted as red pixels (see Figure 6(b)).

Rewards. The agent is rewarded if the droplet is transported to its destination. Let \(e_{i, j}\) be the \(i^{\text{th}}\) row and the \(j^{\text{th}}\) column electrode of the DMFB. Suppose that in state \(s_t\), a droplet is present at \(e_{i, j}\), and its destination is \(e_{k, m}\). We define \(D(s_t)\) as the Manhattan distance of the droplet from the destination at state \(s_t\); \(D(s_t) = |i - k| + |j - m|\). After an action \(a_t\) is taken, if \(D(s_{t+1}) = 0\), the agent receives a positive reward of \(+1.0\). Otherwise, the reward is computed as follows:

In the first case, the action leads to a state in which the droplet is closer to the destination. Therefore, the reward is positive. Any positive value can facilitate agent convergence because the total reward is maximized. In the second case, the agent is punished because the action does not result in a better state. In the third case, the agent is punished with a negative value of larger magnitude because it leads to a worse state.

We formulate parallel droplet in the MARL framework where agents are fully cooperative. The action space and state for each agent are similar to that of the single-agent formulation.

Rewards. We consider the cooperative setting for the MARL framework [12, 19, 39, 85] because the agents should not compete with each other to transport droplets. We first compute an assessment value \(r^i\) of an agent i after state transition. Similar to prior definition, let \(D^i(t)\) be defined as the Manhattan distance of the droplet \(d_i\) from the destination at timestep t. After an action \(a^i_t\) is taken, if \(D^i(t+1) = 0\), the assessment value \(r^i\) is assigned a positive value of \(+1.0\) because the droplet has reached the destination. Otherwise, the assessment value is computed as follows:

In the first case, the action leads to a state in which the droplet is closer to the destination. In the second case, the action results in the same state or even a worse state. Therefore, we use a smaller value as the assessment value. In this reward setting, to gain the maximum value in a game, the agent is encouraged to take as few steps as possible to reach the destination.

As all the agents take a combination of actions, a possible resultant state is that droplets may get too close to each other, which can lead to unintended merging and sample/reagent contamination. To prevent this scenario, we also adjust the assessment values for droplets that are too close to each other. Assume that, after a joint set of actions is taken, the resultant locations of two droplets \(d^i\) and \(d^j\) are \(e_{a, b}^{d^i}\) and \(e_{c, d}^{d^j}\), respectively. The distance of the two droplets is computed as \(D(d^i, d^j)=|a-c| + |b-d|\). If \(D(d^i, d^j) \le 2\), the assessment values are adjusted as \(r^i = r^i - 0.8\) and \(r^j = r^j - 0.8\). In decentralized learning, each agent i is rewarded by its own assessment value \(r^i\); in centralized learning, we give each agent a team-average reward \(R_{avg}=\frac{\sum _{i=1}^N r^i}{N}\).

We next describe an online droplet router, incorporated as an RL agent, that can execute all the droplet routing tasks. To train the agent, we developed an OpenAI-Gym environment named DMFB-Env. The DMFB matrix consists of \(N\times M\) electrodes, where N and M are inputs to DMFB-Env.

Transition Model. DMFB-Env operates in two modes: healthy and degrading. Recall that \(e_{i, j}\) denotes an electrode at the \(i^{\text{th}}\) row and the \(j^{\text{th}}\) column of the DMFB. The transition function is defined aswhere \(1\lt i\lt N\) and \(1\lt j\lt M\). If the droplet is present at the boundary of the electrode array and the action is toward the outside of the biochip, the droplet will remain at the same location. For example, if the droplet is present at \(e_{0,0}\) and the action is either \(a_N\) or \(a_W\), the droplet remains at \(e_{0, 0}\). Similarly, if the next location of the droplet is in the electrodes that are used for the other concurrent fluidic operations, the droplet stays at the same electrode.

For the degrading mode, we introduce a function \(d(e_{i, j})\) that describes the degradation status of an electrode, where \(0\le d(e_{i, j})\le 1\). If the electrode \(e_{i, j}\) is healthy, \(d(e_{i, j}) = 1\); otherwise, \(d(e_{i, j}) = 0\). The study in the work of Dong et al. [13] showed that an electrode can only be actuated up to 200 times before it is completely degraded. Therefore, we define a degradation factor \(\tau\), where \(0.5\le \tau \le 0.7\), and the degradation function \(d(e_{i, j})\) is defined aswhere n is the number of actuations. Each electrode is randomly assigned a different value of \(\tau\) to simulate the geometric variance of the electrode array.

A Bernoulli random variable \(X_{i, j}\) is defined as the transition outcome when the droplet is present at \(e_{i, j}\): when \(X_{i, j}=1\), the transition is successful as \(T(e_{i, j}, a_t)\); when \(X_{i, j}=0\), the transition fails, and the droplet remains at the same electrode. The probability mass function of \(X_{i, j}\) is defined as

RL Agent. The RL agent is a deep neural network (see Figure 6). It observes images and chooses an action \(a_t \in A\). It receives a reward value based on the outcome of the previous action.

Over the past few years, many neural network architectures have been proposed [27, 36, 68]. Because DMFBs commercially available today typically include a few hundred electrodes [86], we evaluate the effectiveness of RL-based adaptation using DMFBs of size \(N\times M\), where \(25\le N\times M\le 1,225\). While fully connected neural networks are adequate for small DMFB instances (less than 100 electrodes), we found that they do not converge for large DMFBs. Our evaluation showed that Convolutional Neural Networks (CNNs) are effective for the preceding DMFB instances. However, because the network needs to be loaded on a DMFB, the computational resources on the associated controller may be limited compared to a server. For example, in the work of Willsey et al. [81], the DMFB includes only a quad-core 1.2-GHz ARMv7 processor with 1 GB of RAM, and it does not contain a GPU; therefore, large CNNs are not feasible in this application scenario. We tested several options for the number of hidden layers and the number of neurons per layer. Our results show that a simple CNN, as described in Table 1, can solve the droplet- routing problem for large DMFBs with more than 1,000 electrodes.

We consider fabricated DMFBs as test cases and evaluate the effectiveness of RL-based adaptation using arrays of size \(N\times N\). N is set as \(10\le N\le 35\) since the number of total electrodes for recent commercial microfluidics biochips is around 500 on a chip [32]. For each training game of DMFB-Env, a random routing task is generated. In addition, DMFB-Env generates some random concurrent modules to simulate high-throughput bioassay execution during droplet routing. We evaluated three RL algorithms (i.e., double DQN, PPO, and ACER) described in Section 4 in DMFB-Env in the healthy mode. We used default parameter settings in the work of Hill et al. [23] for the three algorithms. The training was executed on a Linux platform integrated with an 11-GB-memory GPU (NVIDIA GeForce RTX 2080 Ti). The training processes using PPO take nearly 2 hours to converge, which is the fastest among the other algorithms. Although it takes several hours to train a model to perform as well as the offline method, training needs to be carried out only once, and the trained model can subsequently be used for all fabricated DMFBs. We compare the RL approaches with an offline optimization method [86].

The training processes for different sizes of DMFBs are shown in Figure 7. For each RL algorithm, we ran 18 simulations with random seeds; the average performance of each algorithm is plotted as a solid line, and the similar color region shows the interval between its best performance and its worst performance. A training epoch contains 20,000 timesteps. We observe that double DQN does not converge in all training settings. In some cases, double DQN learned sub-optimal policies first, and then the policy learned lower-reward experiences, which results in converging to more passive policies. The results are similar to RL training in other environments [34, 43]. We observe that PPO performs well in all training settings, but it sometimes takes more training epochs to converge. This is because PPO is sensitive to initialization [28, 35, 79]. In addition, the update rule of PPO encourages the policy to exploit rewards that it has already found over the training course. Therefore, if an initial network policy is far from global optima, the policy can be easily trapped in local minima. We also observe that ACER does not perform well in some training settings. As the action space and observation space grow exponentially, the experiences stored in the limited replay buffer become important for ACER training.

Our training results show that in all training settings, PPO can outperform the other two RL algorithms. To fine-tune the RL approach using PPO, we tested two significant parameters in PPO to find the best performance of our RL agent for different sizes of DMFBs, the number of concurrent environments, and the number of steps for each update.

Figure 9 shows the training rewards for agents with varying the number of concurrent environments and the number of steps for each update. Here, we show the training rewards for a \(10\times 10\) DMFB, a \(20\times 20\) DMFB, and a \(30\times 30\) DMFB. The training is not stable when there are only a few concurrent environments. For example, when there are four environments, we found out that the performance of the training model (updated every 16 steps) drops significantly after a few training epochs. We also observed that for eight environments, irrespective of the update step interval, the model’s performance is consistently better. Similar trends are observed in training for other sizes of DMFBs. Therefore, we chose eight concurrent environments as the PPO setting for model training.

We produced a video recording of droplet routing for a \(5\times 8\) DMFB during training (see [41]). From the video, we see that, at first, the agent moved the droplet randomly without knowing the policy needed to reach the destination. After 200K timesteps, the agent started to “learn” from past experience; early on, after 400K timesteps, it could transport the droplet to the destination using the shortest path for only a few of the routing tasks. However, after 800K timesteps, the agent was able to complete all the routing tasks using the shortest paths.

To train the agents, we developed a PettingZoo-Gym environment to simulate the parallel droplet scenarios. For each training game, \(n_{rt}\) random routing tasks are generated, where \(n_{rt} = \lbrace 2, 3\rbrace\). Each routing task is performed concurrently by one of the agents. The size of the DMFB is \(N\times N\), where \(10 \le N\le 30\). We first performed the agent training using three RL algorithms (PPO, double DQN, and ACER). We also used three different MARL training schemes, including centralized, concurrent, and parameter sharing.

Figure 10 shows the training processes for two and three concurrent routing tasks in the healthy mode. A training epoch contains 20,000 timesteps. The performance of different algorithms is compared with the offline optimization method (Baseline) and the RL agents that are trained under single routing task environments (Single). The results show that the concurrent scheme is the most effective and efficient scheme to train the MARL routing models for DMFBs. We observed that PPO and ACER have similar performance, whereas DQN fails to converge in all the training settings. In some of the settings, such as the concurrent training with DMFBs of size \(20 \times 20\), the ACER algorithm converges faster than the PPO algorithm.

However, the figure shows that single agents can achieve comparable performance as PPO and ACER when the size of DMFB is small. However, as the size of DMFB grows and the number of concurrent routing task increases, the performance of single-agent models rapidly decrease since the single-agent models did not learn the coordination between droplets. The results illustrate the importance of MARL models for concurrent routing scenarios.

We compared the performance of the agent with the work of Zhao and Chakrabarty [86]. We set \(50\%\) of the degrading electrodes for DMFBs, and the results are shown in Figure 8. Here, we show the number of actuation cycles required in a game as the performance. The fewer actuation cycles required in a game, the better the performance is. We observe that the agent performs similar to the static (offline) method when the DMFBs start to degrade. This is because the RL agent has been trained to perform as well as the baseline in the healthy mode of DMFB-Env. After a small number of training games, the RL agent sometimes performs slightly worse because the agent may explore other alternative routes to avoid the degraded electrodes, and the alternative solutions may be worse than the original route. However, as DMFBs degrade further, the agent can outperform the baseline. We also observe that the proposed solution is more effective for smaller DMFBs. This is because, in our experimental setting, the DMFB with 25 electrodes is the most dynamic environment. The performance of the baseline method decreases if electrode degradation occurs in a DMFB. We see that the performance of the baseline method significantly decreases in the \(5\times 5\) DMFB. The experimental results show that the agent can adapt to all sizes of DMFBs, including the most dynamic environment (i.e., the \(5\times 5\) DMFB).

We recorded a video of droplet transportation in a simulated degraded environment; the video, called Simulation.mp4, can be found in the work of Liang et al. [41]. As some electrodes started to degrade, the agent can still use them to transport the droplet. In the simulated environments, sets of faults with different sizes have been injected. However, the agent is able to learn the changing health conditions of these electrodes. For subsequent tasks, the agent transports the droplet without using these degraded electrodes.

In the degrading mode of MARL, we set 10% of the degrading electrodes, and the degrading level of these electrodes increases as these electrodes are used over time. We compared the performance of the MARL models with the baseline method. The results are shown in Figure 11. We used the concurrent method to train the MARL models since concurrent is the most effective training method as discussed in Section 5. For DMFBs with the size of \(10 \times 10\) and \(20 \times 20\), we used PPO as the training algorithm since PPO and ACER achieve similar performance while the training processes of PPO are faster. For DMFBs with the size of \(30 \times 30\), we used ACER as the training algorithm since ACER achieves the best performance among the three algorithms.

The degradation processes are shown in Figure 11, where the performance is evaluated using the number of cycles needed to transport all the droplets to the destinations. Figure 11 shows that as the electrodes start to degrade, the MARL agent performs slightly worse than the baseline method since the agents are learning to avoid degraded electrodes and exploring alternative routes, which are longer than the routes taken by the baseline method. After several training epochs, the MARL agents outperform the baseline method as the DMFBs degrade further and the MARL agents have learned from the previous training games. As shown in Figure 11, for DMFBs with sizes of \(10 \times 10, 20 \times 20\), and \(30 \times 30\), the number of training epochs that the model needs to adapt to the degrading environments are around 5 to 10, 10 to 15, and 20, respectively. The results show that the MARL agents can adapt to dynamically degraded environments under different sizes of DMFBs and provide more reliable routing strategies than the baseline method.

As the RL router learns to adapt to a degrading biochip, the RL agent needs to be repeatedly trained and referenced on the micro-computer of the DMFB system during the biochip execution. We profiled the runtime of the PPO training and referencing for each timestep on a micro-computer (Raspberry Pi 4) for various sizes of DMFBs (Table 2). Although the micro-computer includes a modest 1.5-GHz quad-core processor and only 4 GB of memory, one training timestep takes only about 0.04 seconds, and one referencing timestep takes only about 0.06 seconds. In our DMFB design, the actuation time required to move one droplet from an electrode to an adjacent electrode is 1 second. Therefore, the training step can be carried out concurrently while the fluidic operation occurs. The additional referencing time for the RL agent to determine the next fluidic operation is 0.06 seconds. The timing overhead of using the RL framework is therefore \(6\%\) when compared with the original DMFB system, which is negligible in practice.

In this section, we show the feasibility of deploying our RL model on a fabricated chip. The model deployment is general regardless of the sizes of biochips. In addition, the proposed RL framework can be used for any bioassay. As a specific case study, We designed and executed an epigenetic bioassay on a fabricated DMFB because benchtop epigenetic bioassays require large sample volumes and long execution time, and are labor intensive. Previous work has shown the effectiveness of epigenetic bioassays on DMFBs [30]. This epigenetic bioassay includes 19 routing tasks. We used the trained RL droplet router to transport droplets.