Smart Multi-Level Tool for Remote Patient Monitoring Based on a Wireless Sensor Network and Mobile Augmented Reality

3.1.1. Design of Sensor Prototypes

Each patient node monitors the activity of two sensors, the BT sensor and the HR sensor, and informs other patient nodes about any events. The design of the sensor prototypes is explained in the following.

(a)

Body Temperature Sensor Prototype

The BT is a critical measure of the health status of the patient. Typically, the BT can be classified into three categories: hypothermia, which occurs when the patient temperature is below 36.0 °C; normothermia, which is the acceptable range of 37.5 °C to 36.5 °C for the human body; and hyperthermia, which occurs when the patient temperature exceeds 37.5 °C. Hypothermia is considered a potentially fatal emergency condition, and it is possible that lethargy, cardiac arrest, or even coma could occur if it is not treated quickly. Hyperthermia is an elevated body temperature due to failed thermoregulation, a situation that may cause organ damage or death [16].

There are many places where the BT may be measured, including the armpit, mouth, rectum, and ears. In general, the armpit is the easiest location to measure the BT, and nurses place a thermometer under the patient's armpit to acquire the BT. For this reason, an elastic band was developed to locate the sensor in the armpit. The prototype for BT measurement was developed for the purpose of testing the SMTRPM; consequently, the medical security protocols that must be followed when testing medical devices in PHIs were not incorporated in the design. The final BT prototype is shown in Figure 3. The prototype is an assembly of an NTC-CL80 thermistor and an elastic band. When the device is placed on the patient, the sensor must be located in contact with the patient's armpit.

Conventional digital thermometers use different types of thermistors to acquire the body temperature. The BT prototype was characterized using Equation (1), based on the NTC-CL80 thermistor curve [17].

$$K=(\beta )\left(\frac{1}{\mathit{\text{In}}\left(\frac{R}{R_0}\right)+\left(\frac{\beta }{T_0}\right)}\right)$$

(1)

where β is the constant of the thermistor material, R₀ and T₀ are reference points on the NTC thermistor characterization curve, R is the resistance value of the body temperature, In is the natural logarithm, and K is the BT of the patient in degrees Kelvin.

The BT prototype was tested and compared with a commercial thermometer using different β values. Note that β is temperature dependent and is specified between two temperature points on the NTC-CL80 curve. Different β values were compared in order to improve the accuracy of the BT sensor. The body temperature of the test patient was 36.5 °C when measured with a digital thermometer. The closest match between the instruments was obtained for β = 3260.5, with a relative error value of 0.001. This β value was used in the program implementing the BT algorithm, which is described in Section 3.1.2. Table 2 shows the experimental results from the 30 comparative runs between the BT prototype and the commercial thermometer, which were performed for three values of β.

(b)

Heart Rate Sensor Prototype

The HR is one of the most critical measures of patient health. In fact, evidence of high HR variability can support diagnoses such as infection, high levels of triglycerides or cholesterol, and lethal arrhythmias [18,19]. The SMTRPM is able to measure the patient's HR through a non-invasive photoplethysmographic sensor shown in Figure 4a. The system detects the heart beats per minute (BPM) and normal-to-normal heartbeat intervals (N-N intervals) from a continuous ECG wave record. Conventionally, the time at which the photoplethysmographic pulse waveform reaches 50% of its maximum value is used to indicate the beginning of a pulse [20]. The N-N interval is the elapsed time between pulses. Figure 4b shows the R wave and the corresponding N-N interval.

The SMTRPM system uses the BPM information to determine the category of the heart rate: normal (60–100 BPM), tachycardia (ϯ100 BPM) [21], or bradycardia (<60 BPM) [22]. In some cases, tachycardia can significantly disrupt normal heart function, increasing the risk of stroke or causing sudden cardiac arrest or death. Bradycardia can be a serious problem if the heart does not pump enough oxygenated blood to the body. N-N intervals are essential to measuring some characteristics of tachycardia and bradycardia. For example, one of the important parameters in determining if a heart attack has occurred is the SDANN (standard deviation of the averaged N-N intervals) [21].

To validate the accuracy of the HR prototype, ten measurements were obtained from ten patients by a nurse using a digital watch. The same action was then performed using the HR prototype. As shown in Table 3, the results obtained were very similar, verifying that the prototype performed correctly.

3.1.2. Smart Monitoring

The SMTRPM performs smart monitoring of both the BT and HR variables through algorithms programmed into the Arduino Uno platform. The algorithms determine the patient's condition in real time based on the two variables. The smart monitoring system thus implements basic medical knowledge in the form of algorithms to diagnose conditions related to the BT and HR variables [16,20].

Figure 5 shows the flow diagram for the BT algorithm that runs locally on each patient node. It can be observed that the monitoring process can be initiated in two ways. The first is based on the BT monitoring interval established by the nurse. The second occurs when another patient node in the network detects a BT anomaly. When a WiFi communication from another node initiates the acquisition of the BT from the patient, ten BT samples are collected to verify whether the data consistently indicate the patient's condition to be hyperthermia, hypothermia, or normothermia. Finally, the patient node sends the BT and its classification to the last patient node. The last node sends the packet of information to the nurse server and the smartphone application.

The HR smart monitoring process uses the algorithm developed by Jadav et al. [20] to determine the patient's condition. Figure 6 shows the flow diagram for the algorithm used to identify the occurrence of a heartbeat. The algorithm identifies the time at which the pulse reaches 50% of the difference between the maximum and minimum of the R wave. Data are obtained from the photoplethysmographic sensor every 2 ms using an interrupt routine. The N-N interval must be greater than 250 ms to be considered a true N-N event. When the system identifies a beat, it stores the elapsed time to establish a time reference for the next beat, and the N-N interval is estimated using the elapsed time between two beats. Every 60 s the system computes the average of the N-N intervals and the BPM. After obtaining the HR, the system identifies a tachycardic or bradycardic condition using a comparison process similar to the BT algorithm.

3.2.1. Ad Hoc Mode

The patient nodes and the nurse server are connected via a local ad hoc remote wireless network. The dynamic nature of this type of network requires intelligence in each of the network nodes for the delivery of information packets and for routing analysis [23]. Although the patient nodes have little mobility, they must find routes to deliver information. The patient nodes must determine if a specific or general anomaly has occurred in the room by sharing individual diagnostic information with the last patient node. When a patient presents an anomaly, the node sends the information to the last patient node and sends an alarm to the neighbor. This alarm causes the neighbor patient node to begin its own monitor process, and both patient nodes send the information to the last patient node, which is the only sink that obtains information from all the patient nodes. A reactive AODV (ad hoc on-demand vector) protocol is used to determine routing and anomaly conditions. The AODV algorithm determines a destination sequence number to evaluate a destination route [24]. The SMTRPM was tested with five patient nodes per room. Figures 7 and 8 show examples of specific and general anomaly detection, respectively.

In Figure 7, only two patient nodes have detected a hyperthermia anomaly. This information is sent to the last patient node via AODV routing. In the example, the last patient node determines that a specific hyperthermia anomaly has occurred because the SMTRP is programmed to identify a general anomaly only when three or more patients present the same anomaly, as in Figure 8.

The system adopts a hierarchical structure whereby there is a last patient node in each room. The last patient node is also a patient node with a program routine to create HR and BT package. Then, the last patient node sends the package to the nurse server. Table 4 shows an example of the structure of the HR package with specific anomalies at patients D and E.

3.2.2. Internet Mode

Once the communication with the central nurse server is completed, the last patient node can connect to the Internet to send a diagnostic packet to a mobile device using an Android notification application. The mobile device application provides an alternative alert about the patient's health to the nurse even if the nurse is not at the central nursing station. The SMTRPM is implemented on a smartphone running Android OS 4.3.2. The mobile device and the last patient node are connected to the Internet. It is important that the network service company provide a static IP address to the PHI to avoid changes in the IP configurations of the last patient node and the mobile device from their original addresses.

3.3.1. Nurse Server Interface (NSI)

The NSI is a prototype that was created using LabView to support the analysis that the nursing staff performs to update patient health information. Typically, doctors give care instructions to the nurses when a patient has undergone treatment. For example, if a patient with an infection develops hyperthermia, the nurse dispenses medicine to reduce the patient's BT. Consequently, it is necessary to measure the BT frequently to determine the patient's rate of reaction to the treatment. Using the NSI, the nurse has the ability to remotely monitor the BT and HR at any time, reducing the time needed to monitor the patient and avoiding the need for continuous movement between rooms, especially when a nurse is in charge of many patients. Figure 9 shows the monitor screen of the NSI displaying a BT measurement.

The NSI runs on a PC at the nursing station. The application displays a waveform chart of the BT and HR variables on the monitor screen. The waveform chart can be used to support the nurse's analysis of the evolution of a patient's health or the reaction of a patient who has undergone treatment. In some cases, a patient may develop an allergy or rejection related to treatment, and then it becomes necessary to stop treatment. In those cases, the NSI can identify the patient's condition to the nurse faster than can remote alarms. Figure 10 shows the main screen of the NSI; the beta version is designed for two rooms with four patients in each room. When a patient develops an anomaly, the nurse can identify in which room the event occurred, the type of anomaly, and the patient with the anomaly. Then, the nurse can select the room and the patient on the monitor screen to observe the behavior of the specific variable associated with the anomaly because this information is associated with the IP address of the patient node.

3.3.2. Mobile Nurse Monitor (Data Interoperability)

In recent years, advances in communications technology have enabled data sharing between two or more systems. Data interoperability allows the user to be informed of, and updated on, what is occurring in databases, files, or devices inside a network in specific situations [25]. The SMTRPM uses data interoperability to share information about hospitalized patients on a managed ad hoc WSN. The mobile nurse monitor (MNM) was developed in an Eclipse environment using the Android SDK (specifically, the ADT bundle for Windows) to provide continuous remote monitoring. The MNM provides the ability to test communication with individual clients via the TCP/IP protocol; however, the main objective was to create a mobile node that behaves in a server mode awaiting an alarm from the last patient node. Figure 11 shows the visual interface of the MNM application.

When an anomaly occurs at a patient node, the last patient node immediately sends a real-time alarm to the MNM application through the Internet mode. The application has the ability to inform the nurse about the alarm condition, identifying which patient node presented the anomaly and which type of anomaly was diagnosed. Because the nurse is notified anywhere inside the hospital, the MNM application can save valuable response time. The MNM application is particularly useful when the nurse is not at the central nursing station.

3.3.3. Virtual Patient File (Augmented reality)

Frequently, nurses and doctors use handwritten patient files to view and update a patient's health status. The virtual patient file (VPF) application was developed to offer an alternative for the digital management of this information. Several technologies were evaluated for the design and implementation of the VPF. The options reviewed included near field communication (NFC), quick response (QR) codes and augmented reality (AR).

NFC and AR are considered to be among the most promising emerging mobile technologies. NFC can provide a good means of data tracking to prevent the possibility of human error when a doctor or nurse visits a patient, such as forgetting to note in a log that the patient was visited. However, given that proximity to the device is required for NFC, NFC cannot always serve as a substitute for AR or QR codes, which can both operate over a considerable distance. Today, many companies refuse to use NFC. Another major risk of NFC is computer hacking or phone hacking. Finally, NFC is relatively new, and not every mobile phone is compatible with this technology [26,27]. Because of these concerns, NFC was not further considered for the VPF.

QR codes are used to transfer information from a transitory medium to a cell phone. Once a QR code has been transferred to a cell phone, it may, for example, provide details about the business that provided the code, show a URL that can be clicked to watch the trailer for a movie, or issue a coupon that can be used at a local outlet. QR codes are more useful than standard barcodes because they can store much more data, including URL links, geographic coordinates, and text. However, one of their main disadvantages is people's lack of familiarity with QR codes. Another major disadvantage is the heterogeneity of mobile devices, which must be equipped with a camera and the correct reader software [28,29]. Unfortunately, many users whose mobile phones have cameras are unable to obtain QR reading software for their phones. For these reasons, QR codes were not further considered for the VPF.

AR is changing the way that people access information in their environment. The main goal of AR is to “augment everything everywhere for everyone”. The use of AR in conjunction with mixed reality (MR) breaks the barriers between virtual media, the physical world and our imagination by enriching our ability to interact with all three. AR could literally change the way we see the world, instantly providing information on any topic we choose to study [30]. The layering of information over a 3D space produces a new experience of the world. AR is fueling the broader migration of computing from desktops to mobile devices, bringing with it new expectations regarding access to information and new opportunities for learning [31]. Unlike QR codes, a key characteristic of AR is its ability to respond to user input. Additionally, AR is transforming the medicine and healthcare sectors significantly. In the domains of medicine and healthcare, AR not only can help save lives but also can help healthcare organizations make their existing processes more precise and efficient. Consequently, AR was selected for the implementation of the VPF.

The VPF was developed to offer an alternative for the management of digital information. The VPF is an MAR environment developed under the DroidAR framework [32]. DroidAR allows, among other AR features, marker-based tracking and the superposition of 2D and 3D models.

The VPF enables nurses to interact with digital information embedded within the physical environment and can be used to display information that otherwise would not be perceived by the regular human senses [33,34]. There are two forms of AR currently available: marker-based AR and markerless AR. For this research, marker-based AR was selected.

The VPF application uses two markers to display the BT and HR variables in real time. Each variable is associated with a marker ID. Both markers are pasted on the patient's headboard. The MAR application requires the following stages. Initially, a camera calibration process is performed in which the fundamental parameters of the camera are determined. These parameters determine where a 3D point is projected onto a 2D point (i.e., a pixel).

In an ideal camera, a 3D point (X, Y, Z) in space will be projected onto a pixel according to Equation (2).

After the camera calibration process has been performed, the system is ready to use. For example, when the nurse approaches a bed, he or she executes the application. Instantly, live video from the smartphone camera is acquired, and the nurse points to the markers. Then, the application performs the following computer vision algorithms. The images from the video stream are converted to binary images by a simple thresholding process described by Equation (3) [36].

If there is a marker in the scene, it is identified. Next, the positions and orientations of the markers relative to the camera coordinates are calculated. The symbol inside the marker is matched with templates, for which the only two possibilities are the marker for BT and the marker for HR. If a different figure is observed, it is discarded and the streaming continues. Finally, the position and orientation of the marker are computed. This is performed to align the virtual object with the marker. If the VPF detects the BT marker, a 2D thermometer image is superimposed onto the real scene in the position and orientation defined by the marker. In the case of the HR marker, a 2D heart image is superimposed.

Additionally, there are two buttons (one at the top left and the other at the bottom left) in the application display used to request the status of a variable at any time. A touch event was programmed into the 2D images. Using this event, the nurse obtains the variable status in real time by touching its 2D image. Figure 12 shows the VPF application running.

All the data obtained are stored in a database to maintain information about each patient. Because the VPF application was developed with a commercial license for DroidAR, there are some limits of the AR environment. For example, only five markers can be detected in the same scene, and the VPF application cannot handle occlusion.