Nothing Special   »   [go: up one dir, main page]

This paper is in the following e-collection/theme issue:

Diabetes Reviews and Scoping Studies (32) Machine Learning (1476) Glucose Tracking and Self-Monitoring of Blood Glucose (163)

Published on in Vol 6, No 1 (2021): Jan-Mar

Preprints (earlier versions) of this paper are available at https://preprints.jmir.org/preprint/22458, first published .
Ability of Current Machine Learning Algorithms to Predict and Detect Hypoglycemia in Patients With Diabetes Mellitus: Meta-analysis

Ability of Current Machine Learning Algorithms to Predict and Detect Hypoglycemia in Patients With Diabetes Mellitus: Meta-analysis

Ability of Current Machine Learning Algorithms to Predict and Detect Hypoglycemia in Patients With Diabetes Mellitus: Meta-analysis

Authors of this article:

Satoru Kodama1 Author Orcid Image ;   Kazuya Fujihara2 Author Orcid Image ;   Haruka Shiozaki2 Author Orcid Image ;   Chika Horikawa3 Author Orcid Image ;   Mayuko Harada Yamada2 Author Orcid Image ;   Takaaki Sato2 Author Orcid Image ;   Yuta Yaguchi2 Author Orcid Image ;   Masahiko Yamamoto2 Author Orcid Image ;   Masaru Kitazawa2 Author Orcid Image ;   Midori Iwanaga2 Author Orcid Image ;   Yasuhiro Matsubayashi2 Author Orcid Image ;   Hirohito Sone2 Author Orcid Image
Article Authors Cited by (32) Tweetations (4) Metrics

    Original Paper

    1Department of Prevention of Noncommunicable Diseases and Promotion of Health Checkup, Niigata University Graduate School of Medical and Dental Sciences, Niigata, Japan

    2Department of Hematology, Endocrinology and Metabolism, Niigata University Graduate School of Medical and Dental Sciences, Niigata, Japan

    3Department of Health and Nutrition, Faculty of Human Life Studies, University of Niigata Prefecture, Niigata, Japan

    Corresponding Author:

    Satoru Kodama, MD, PhD

    Department of Prevention of Noncommunicable Diseases and Promotion of Health Checkup

    Niigata University Graduate School of Medical and Dental Sciences

    1-757, Asahimachi-dori, Cyuoh-ku

    Niigata, 951-8510

    Japan

    Phone: 81 25 227 2117

    Email: ybbkodama@gmail.com


    Background: Machine learning (ML) algorithms have been widely introduced to diabetes research including those for the identification of hypoglycemia.

    Objective: The objective of this meta-analysis is to assess the current ability of ML algorithms to detect hypoglycemia (ie, alert to hypoglycemia coinciding with its symptoms) or predict hypoglycemia (ie, alert to hypoglycemia before its symptoms have occurred).

    Methods: Electronic literature searches (from January 1, 1950, to September 14, 2020) were conducted using the Dialog platform that covers 96 databases of peer-reviewed literature. Included studies had to train the ML algorithm in order to build a model to detect or predict hypoglycemia and test its performance. The set of 2 × 2 data (ie, number of true positives, false positives, true negatives, and false negatives) was pooled with a hierarchical summary receiver operating characteristic model.

    Results: A total of 33 studies (14 studies for detecting hypoglycemia and 19 studies for predicting hypoglycemia) were eligible. For detection of hypoglycemia, pooled estimates (95% CI) of sensitivity, specificity, positive likelihood ratio (PLR), and negative likelihood ratio (NLR) were 0.79 (0.75-0.83), 0.80 (0.64-0.91), 8.05 (4.79-13.51), and 0.18 (0.12-0.27), respectively. For prediction of hypoglycemia, pooled estimates (95% CI) were 0.80 (0.72-0.86) for sensitivity, 0.92 (0.87-0.96) for specificity, 10.42 (5.82-18.65) for PLR, and 0.22 (0.15-0.31) for NLR.

    Conclusions: Current ML algorithms have insufficient ability to detect ongoing hypoglycemia and considerate ability to predict impeding hypoglycemia in patients with diabetes mellitus using hypoglycemic drugs with regard to diagnostic tests in accordance with the Users’ Guide to Medical Literature (PLR should be ≥5 and NLR should be ≤0.2 for moderate reliability). However, it should be emphasized that the clinical applicability of these ML algorithms should be evaluated according to patients’ risk profiles such as for hypoglycemia and its associated complications (eg, arrhythmia, neuroglycopenia) as well as the average ability of the ML algorithms. Continued research is required to develop more accurate ML algorithms than those that currently exist and to enhance the feasibility of applying ML in clinical settings.

    Trial Registration: PROSPERO International Prospective Register of Systematic Reviews CRD42020163682; http://www.crd.york.ac.uk/PROSPERO/display_record.php?ID=CRD42020163682

    JMIR Diabetes 2021;6(1):e22458

    doi:10.2196/22458

    Keywords

    machine learning; hypoglycemia; meta-analysis 


    Hypoglycemia is a major barrier to achieving the tight glycemic control in patients with diabetes mellitus (DM) that is required to delay the progression of late DM-related complications. Although many patients exhibit symptoms of hypoglycemia such as anxiety, heart palpitations, and confusion, a significant number have diminished ability to recognize these hypoglycemic symptoms [1,2], which is defined as “impaired awareness of hypoglycemia” [3]. This impaired awareness can lead to severe hypoglycemia, which is associated with seizures, coma, and death. Real-time glucose monitoring can help patients maintain optimal glycemic control while avoiding symptomatic or asymptomatic hypoglycemia [4]. However, the traditional monitoring method, intermittent glucose monitoring by finger stick, provides only a limited number of readings and is unlikely to detect hypoglycemia of a short duration. Continuous glucose monitoring (CGM) typically produces a reading every 5 minutes and can alert the patient to not only the occurrence of hypoglycemia but also impending hypoglycemia [5]. Accuracy of CGM has progressively improved, with overall measurement errors reduced by twofold than in the first commercially available CGM devices introduced in 2000 [5].

    However, even if CGM advancements enabled patients to continuously track their subcutaneous glucose levels, the statistical disadvantage of the CGM data stream would remain as a major limitation. The autocorrelation of the CGM reading vanishes after 30 minutes, meaning that the projection of blood glucose levels more than 30 minutes ahead would be inaccurate [6]. This finding suggests that the algorithm for identifying hypoglycemia should consider a patient’s contextual information such as diet, physical activity, and medications (including insulin) as well as various features of the CGM trend arrow [7].

    Machine learning (ML) algorithms have been widely introduced to diabetes research including those for identification of hypoglycemia. The growing use of mobile health (mHealth) apps, sensors, wearables, and other point-of-care devices, including CGM sensors for self-monitoring and management of DM, have made possible the generation of automated and continuous diabetes-related data and created the opportunity for applying ML to automated decision support systems [8]. Combining ML-based decision support systems with the abundance of generated data has the potential to identify hypoglycemia with greater accuracy.

    Conventionally, ML has been applied to detect abnormalities in blood glucose levels using physiological parameters that are highly correlated with hypoglycemia (eg, changes in brain or cardiac electrical activities) [7]. Recently, in addition to the detection of hypoglycemia, ML-based decision support systems have been proposed for predicting hypoglycemia by using various historical data (eg, series of blood glucose data, other laboratory and demographic data, verbal data in medical records, or secure messages suggesting occurrence of hypoglycemic events) [8]. Despite many reports of ML algorithms for detecting or preventing hypoglycemia, their abilities have not been comprehensively or quantitatively assessed. This meta-analysis aims to assess the current ability of ML algorithms to detect or predict hypoglycemia in patients with DM.


    Protocol Registration

    The study protocol has been registered in the international prospective register of systematic reviews (PROSPERO; Registration ID: CRD42020163682).

    Literature Searches

    We used Dialog to perform the electronic literature searches. The platform allows users to access and search 96 databases of peer-reviewed literature. Publication dates ranged from January 1, 1950, to September 14, 2020. Search terms consisted of 2 elements: (1) thesaurus and text words related to ML and (2) text terms related to hypoglycemia and thesaurus terms related to glucose monitoring or blood glucose. The use of the thesaurus term was limited to 2 databases: EMBASE (EMTREE terms) and MEDLINE (MeSH terms). The above 2 elements were combined using the BOOLEAN operator “AND” (Multimedia Appendix 1). Manual searches were added to review reference lists in relevant studies. If eligible studies were obtained from the reference lists, the reference lists in those studies were also examined. Manual searches were continued until no eligible study was found in the references lists.

    Study inclusion criteria were (1) all participants had DM; (2) study endpoint was hypoglycemia; (3) researchers clarified that they originally trained the ML algorithm using training data to build a model for detecting or predicting hypoglycemia or the same researchers trained the ML algorithm in a previous study; (4) the model’s performance was tested using the test data; and (5) sensitivity and specificity for detection or prediction of hypoglycemia were presented or could be calculated.

    Exclusion criteria were (1) an event-based study (ie, specificity could not be estimated because nonhypoglycemia data were not included in the test data), (2) a case study (ie, training and test data were derived from only 1 patient), and (3) a 2 × 2 contingency table consisting of the number of true positives, false positives, false negatives, and false positives could not be reproduced. If studies met all of the inclusion criteria but did not allow the reproduction of a 2 × 2 contingency table, we asked the corresponding author of these studies for the total number of test data sets (N-total) and events (N-hypo) so that we could reproduce the 2 × 2 table. If the same test data were shared by 2 or more eligible studies, we chose the most updated study in which the ML algorithm was considered to show the best performance.

    The outcome of meta-analyses of diagnostic or prognostic tests is the extent of consistency between an index test and a reference standard. The index test is defined as a new test that is proposed when the method for perfectly diagnosing a target condition in all individuals does not exist or cannot be used. In this meta-analysis, it corresponded to an ML algorithm that classified the input data as either hypoglycemia or nonhypoglycemia. The reference standard is defined by a procedure that is considered the best available method for categorizing participants into having or not having a target condition. In this meta-analysis, it corresponded to methods for diagnosing hypoglycemia in clinical practice, which included measurement of glucose levels, the International Classification of Diseases (ICD) code for hypoglycemia, or experts’ subjective judgment. Evaluation item was the ability of ML algorithms to detect hypoglycemia (ie, alert to hypoglycemia coinciding with its symptoms) or the ability to predict hypoglycemia (ie, alert to hypoglycemia before its symptoms have occurred). In studies that assessed the ability for detection, data used for the index test (ie, the ML algorithm) and data used for a reference standard (ie, diagnosing hypoglycemia) had to be examined at the same time. In studies assessing predictive ability, the data input into the ML algorithm had to be examined before the diagnosis of hypoglycemia.

    Data Extraction

    Data were extracted by two authors (SK and KF) Disagreements were resolved by discussion with a third author (HiS). We fundamentally selected 1 datum if there were 2 or more extractable data for a set of test data in an individual study. If an individual study tested 2 or more ML classification methods or 2 or more models for 1 ML classifier, we extracted the datum related to the classifier or model that the study proposed as the best. If 2 or more different results were presented for the same model depending on the prediction window or horizon, we extracted data on the result in relation to the longest prediction window or horizon.

    The following study characteristics were extracted: first author, publication year, evaluated item (ie, detecting or predicting hypoglycemia), country, type of DM (ie, type 1 or type 2), number of study participants, N-total, N-hypo, mean or range of the patients’ age, time of day of hypoglycemic events, place of supposed hypoglycemic episode (ie, experimental, in-hospital, and out-of-hospital), ML algorithm used for classification into hypoglycemia and nonhypoglycemia, threshold of glucose level for hypoglycemia, method for diagnosing hypoglycemia, method for separating the database into training and test data, and profiling data that were input into ML algorithms for performance testing.

    Study Quality

    To evaluate study quality, we used a revised tool to assess diagnostic accuracy of studies (QUADAS-2). The QUADAS-2 consists of 4 domains: selection of participants, index test, reference standard, and flow and timing. All 4 domains were used for assessment of risk of bias and the first 3 domains were used to assess the consensus of applicability. Each domain has 1 query in relation to the risk of bias or applicability consisting of 7 questions (Multimedia Appendix 2) [9]. A “Yes” answer was assigned 1 point.

    Data Synthesis

    The ability of ML algorithms to detect hypoglycemia and predict hypoglycemia was independently assessed. For data that were used to test the model’s performance, the number of true positives, false positives, true negatives, and false negatives was calculated. The set of 4 data was pooled with a hierarchical summary receiver operating characteristic (HSROC) model [10]. Indicators for the model’s performance included sensitivity, specificity, positive likelihood ratio (PLR), which is calculated as (sensitivity/[1–specificity]), and negative likelihood ratio (NLR), which is calculated as ([1–sensitivity]/specificity). Study heterogeneity was assessed by calculating I2 values for PLR and NLR based on a multivariate random-effects meta-regression that considered within- and between-study correlations [11] and classifying them into quartiles (0% to <25%, low; 25% to <50%, low-to-moderate; 50% to <75%, moderate-to-high; >75%, high) [12]. Publication bias was statistically assessed as proposed by Deeks et al [13], wherein the logarithm of the diagnostic odds ratio is regressed against its corresponding inverse of the square root of the effective sample size.

    Sensitivity analyses were added, and the analysis was limited to studies that shared similar characteristics in terms of the type of DM, time of day when hypoglycemia occurred, place of supposed hypoglycemic events, and the profiling data input into the ML algorithm. It is of note that at least four data sets are necessary to perform these sensitivity analyses because the HSROC model has 4 parameters: sensitivity, specificity, accuracy, and threshold. A two-sided P-value <.05 was considered statistically significant. All statistical analyses were performed using Stata 16 (StataCorp).


    Literature Searches

    Multimedia Appendix 3 shows the flow chart of the procedure for selecting studies. Using prespecified search terms, 1226 articles were retrieved; 61 databases published at least one of the retrieved articles (Multimedia Appendix 4). Of these 1226 articles, 150 studies were selected for further review. Manual searches resulted in the addition of 32 studies for further review, making a total of 182 studies. Of these, 149 studies were subsequently excluded for various reasons. Specifically, 12 studies [14-25] presented insufficient data to allow reproduction of the 2 × 2 contingency table, although data on sensitivity and specificity were presented. We asked the authors of these studies to provide N-totals and N-hypos so that we could calculate the number of true positives, false positives, true negatives, and false negatives. However, only the author of 2 studies responded to our communication [15,25], and therefore the remaining 10 studies with insufficient data had to be excluded from the meta-analysis. Finally, 33 studies [15,20,25-55] were eligible.

    Data Extraction of Study Characteristics

    Table 1 shows the summary of study characteristics. Of the 33 studies, 19 studies (58%) [26-31,33,35,36,38-42,44-47,54] predicted hypoglycemia, and the remaining 14 studies (42%) detected hypoglycemia [15,20,25,32,34,37,43,48-53,55]. As much as 25 of the 33 included studies (76%) [15,20,25-27,29,30,32,35,36,38,39,41-44,46-53,55] specified type 1 as the type of DM. Type 2 DM was specified in only 3 of these studies (9%) [28,31,45] and the remaining 5 studies [33,34,37,40,54] did not specify the type of DM.

    Regarding the time of day when hypoglycemic events occurred, nocturnal hypoglycemia was the most frequently reported (14 studies of the 33 included studies; 42%) [15,20,26,30,32,35,36,41,44,49-53]). As to the place of the supposed hypoglycemic episode, 16 of the 19 studies that predicted hypoglycemia (84%) [26-30,35,36,38-42,44-47] supposed the event took place in an out-of-hospital setting. The remaining 3 studies (16%) [31,33,54] supposed hypoglycemia occurring in an in-hospital setting. Of the 14 studies that detected hypoglycemia, 11 studies (79%) [15,20,25,32,43,48-52,55] detected hypoglycemia in an experimental setting, where hypoglycemia was induced by a hypoglycemic clamp procedure. In 20 of the 33 included studies (61%) [15,20,25,27,29,31,32,35,36,38,41,43-45,49-52,54,55]), a hold-out method was used to separate the information in the database according to training and test data.

    Multimedia Appendix 5 shows the profiling data input into the ML algorithm for testing its performance in detecting or predicting hypoglycemia. In the majority of the 19 studies for predicting hypoglycemia (13 studies; 68%) [26-30,35,36,38,40-42,46,47], historical CGM data were input into the ML algorithm while the remaining 6 studies (32%) [31,33,39,44,45,54] did not use CGM. Of the 14 studies that detected hypoglycemia using ML, 7 studies (50%) [20,25,32,49,50,52,55] used information from electroencephalograms (EEGs) and 4 studies (29%) [15,43,51,53] used results of electrocardiography (ECG).

    Table 1. Study characteristics of the 33 included studies to assess the ability of machine learning to detect or predict hypoglycemia.
    Study sourceAssessmentaCountryType of DMPatients, nN-totalbN-hypocMean or range of age (years)TimedPlaceeMachine learningThreshold of Hypof (mmol/L)Method of Hypo detectiongMethod of separationh
    Bertachi et al [26]PrekSpainT1Dm101243932NocpOutsSVMv3.9CGMllnCVoo
    Dave et al [27]PreUSAT1D112637,73518,23313N/SOutRFw3.9CGMHOpp
    Elhadd et al [28]PreQatarT2Dn13391817251N/SOutXGBoostUnclearCGMnCV
    Marcus et al [29]PreIsraelT1D1143,533526418-39N/SOutKRRx3.9CGMHO
    Mosquera-Lopez et al [30], Test 1PreUSAT1D101171734NocOutSVM3.9CGMExV
    Mosquera-Lopez et al [30], Test 2PreUSAT1D20270625835NocOutSVM3.9CGMExV
    Mueller et al [31]PreUSAT2D453,48790,687258066N/SIntREFS3.9Blood/ICDHO
    Ngo et al [32]DeclAustraliaT1D81355312-18NocExpBNNy3.9BloodHO
    Ruan et al [33]PreUKN/So17,658327670366N/SInXGBoost3.9BloodnCV
    Rubega et al [25]DecItalyT1D342516125855N/SExpuNNz3.9BloodHO
    Chen et al [34]DecUSAN/SNo data30011No dataN/SInLRaaN/AkkExpertsmmnCV
    Guemes et al [35]PreUSAT1D655640-60NocOutSVM3.9CGMHO
    Jensen et al [36]PreDenmarkT1D4639217943NocOutLDAbb3BloodHO
    Jin et al [37]DecUSAN/SNo data4104132No dataN/SInSVMN/AICDnnnCV
    Oviedo et al [38]PreSpainT1D10144742041PosqOutSVM3.9CGMHO
    Reddy et al [39]PreUSAT1D55902933ExOutRF3.9BloodExV
    Seo et al [40]PreKoreaN/S104705241252PosOutRF3.9CGMnCV
    Arthur et al [41]PreUSAT1D651640-60NocOutANNcc3.9CGMHO
    Toffanin et al [42]PreItalyT1D2070963646N/SOutI-MPCdd3.9CGMExV
    Ling et al [43]DecAustraliaT1D162695515N/SExpFNNee3.3CGMHO
    Sampath et al [44], DIAiPreUkraineT1D341504018-65NocOutRA3.9BloodHO
    Sampath et al [44], ChildjPreUkraineT1D1794762223-16NocOutRAff3.9BloodExV
    Sudharsan et al [45]PreUSAT2DUnclear839428No dataN/SOutRF3.9BloodHO
    Eljil [46]PreUAET1D1066710025N/SOutBAGgg3.3CGMnCV
    Plis et al [47]PreUSAT1D25816152No dataN/SOutSVM3.9CGMExV
    Jensen et al [48]DecDenmarkT1D10126716044N/SExpSEPCORhh3.9BloodLOOqq
    Jensen et al [48]DecDenmarkT1D10126716044N/SExp+ SVM3.9BloodLOO
    Nguyen et al [49]DecAustraliaT1D51447612-18NocExpFNN3.3CGMHO
    Nguyen et al [50]DecAustraliaT1D5442012-18NocExpANN3.3CGMHO
    Nuryani et al [51]DecAustraliaT1D557513316NocExpPSOii + SVMUnclearCGMHO
    Chan et al [15]DecAustraliaT1D161005215NocExpFNN3.3CGMHO
    Ling et al [52]DecAustraliaT1D527816NocExpFuzzy SVM3.3CGMHO
    Nguyen and Jones [20]DecAustraliaT1D6792712-18NocExpBNN3.3BloodHO
    Skladnev et al [53]DecAustraliaT1D52521116NocInFNN3.9BloodExV
    Zhang et al [54]PreUSAN/S10041114556No dataN/SInDTjj3.3CGMHO
    Iaione and Marques [55]DecBrazilT1D8199099535MorrExpANN3.3BloodHO

    aAbility for which the machine learning algorithm was assessed.

    bN-total: total number of data included in test data.

    cN-hypo: total number of hypoglycemic episodes included in the test data.

    dTime of day when hypoglycemia occurred.

    ePlace of supposed hypoglycemic episode.

    fThreshold of glucose level that was used to diagnose hypoglycemia.

    gMethod for separating training and test data.

    hMethod used for diagnosing hypoglycemia.

    iDIA: DIAdvisor.

    jChild: ChildrenData.

    kPre: predicting hypoglycemia.

    lDec: detecting hypoglycemia.

    mT1D: type 1 diabetes mellitus.

    nT2D: type 2 diabetes mellitus.

    oN/S: not specified.

    pNOC: nocturnal hypoglycemia.

    qPos: postprandial.

    rMor: hypoglycemia during morning.

    sOut: out-of-hospital setting.

    tIn: in-hospital setting.

    uExp: experimental setting (ie, hypoglycemia is induced by injection of insulin. Exercise or drug intervention is included in out of hospital setting).

    vSVM: support vector machine.

    wRF: random forest.

    xKRR: Kernel Ridge Regression.

    yBNN: Bayesian neural network.

    zNN: neural network.

    aaLR: logistic regression.

    bbLDA: linear discriminant analysis.

    ccANN: artificial neural network.

    ddI-MPC: individual model-based predictive control.

    eeFNN: fuzzy neural network.

    ffRA: ranking aggregation algorithms.

    ggBAG: bagging (bootstrap aggregating).

    hhSEPCOR: separability and correlation analysis.

    iiPSO: particle swarm optimization.

    jjDT: decision tree.

    kkN/A: Not applicable.

    llCGM: continuous glucose monitoring.

    mmExperts’ subjective judgment.

    nnICD: International Classification of Diseases.

    oonCV: n-fold cross-validation.

    ppHO: hold-out method.

    qqLOO: leave-one-out cross-validation.

    Assessment of Study Quality

    Multimedia Appendix 6 shows the results of study quality assessments using QUADAS-2. Mean score (SD) was 5.6 (1.1), which corresponded to 80% of full marks (=7). The applicability of the reference test was evaluated to be low in 61% of the 33 included studies (20 studies) because hypoglycemia was not diagnosed by measuring blood glucose levels or ICD codes but by CGM (ie, glucose levels in blood are indirectly estimated from those in interstitial tissue) (19 studies) [15,26-30,35,38,40-43,46,47,49-52,54] or experts’ subjective judgement (1 study) [34]. The 2 factors were mainly responsible for lowering the study quality. We considered that the threshold of hypoglycemia in the index test was not specified in 7 studies, which used the cross-validation method [26,28,33,34,37,40,46], and 1 study, which used the leave-one-out method to separate test data from training data [48].

    Data Synthesis

    Ability for Detection of Hypoglycemia Using ML Algorithms

    Figure 1 shows the HSROC curve and pooled estimates of sensitivity and specificity based on the 14 studies that assessed the ability of the ML algorithm to detect hypoglycemia. The pooled estimates (95% CI) were 0.79 (0.75-0.83) for sensitivity and 0.80 (0.64-0.91) for specificity. The pooled estimates (95% CI) of PLR and NLR were 2.20 (1.46-3.32) and 0.37 (0.28-0.49), respectively. Between-study heterogeneity expressed as I2 was high both for PLR (98%; 95% CI 95%-99%) and NLR (80%; 95% CI 50%-90%). Statistically significant publication bias was detected (P=.15).

    ‎
    Figure 1. Hierarchical summary receiver-operating characteristic (HSROC) curve for detection of hypoglycemia using machine learning algorithms. Circles indicate study-specific sensitivity and specificity for each of the 14 included studies. The size of each circle is proportional to study sample size. The pooled point estimates of sensitivity and specificity are plotted in a filled square.
    View this figure

    We conducted several sensitivity analyses using a portion of the above 14 studies that had 1 study characteristic in common. It was not apparent that any of the sensitivity analyses showed results different from the overall analysis. Limiting the analyses to 12 studies [15,20,25,32,43,48-53,55] that specified type 1 as the DM type, pooled sensitivity, specificity, PLR, and NLR were 0.78 (95% CI 0.73-0.82), 0.71 (95% CI 0.60-0.79), 2.65 (95% CI 1.88-3.72), and 0.26 (95% CI 0.19-0.36), respectively. When analyses were limited to the 7 studies that detected nocturnal hypoglycemia using ML algorithms [15,20,49-53], the pooled estimates (95% CI) were 0.75 (0.70-0.80) for sensitivity, 0.65 (0.55-0.74) for specificity, 2.14 (1.67-2.76) for PLR, and 0.38 (0.30-0.48) for NLR. With analyses of the 11 studies that detected hypoglycemia in an experimental setting, pooled sensitivity, specificity, PLR, and NLR were 0.78 (95% CI 0.73-0.82), 0.71 (95% CI 0.60-0.80), 2.66 (95% CI 1.84-3.85), and 0.31 (0.24-0.41), respectively. The pooled estimate (95% CI) was 0.78 (0.71-0.84) for sensitivity, 0.67 (0.55-0.77) for specificity, 2.39 (1.63-3.50) for PLR, and 0.33 (0.22-0.48) for NLR when the analysis was limited to 7 studies that used EEG abnormalities for detecting hypoglycemia. These estimations were similar when limited to 4 studies that used ECG abnormalities for detection of hypoglycemia: pooled estimate (95% CI) was 0.76 (0.67-0.82) for sensitivity; 0.67 (0.54-0.78) for specificity; 2.31 (1.65-3.23) for PLR; and 0.36 (0.28-0.47) for NLR.

    Ability to Predict Hypoglycemia Using ML Algorithms

    Figure 2 shows the HSROC curve for predicting hypoglycemia based on the 19 studies that assessed the predictive ability for hypoglycemia. The point estimates (95% CI) were 0.80 (0.72-0.86) for sensitivity, 0.92 (0.87-0.96) for specificity, 10.42 (5.82-18.65) for PLR, and 0.22 (0.15-0.31) for NLR. Extremely high between-study heterogeneity was observed for both PLR (I2 [95% CI] 100% [100%-100%]) and NLR (I2 [95% CI] 99% [98%-100%]). Publication bias was not statistically significant (P=.68).

    ‎
    Figure 2. Hierarchical summary receiver-operating characteristic (HSROC) curve for prediction of hypoglycemia using machine learning algorithms. Circles indicate study-specific sensitivity and specificity for each of the 19 included studies. The size of each circle is proportional to study sample size. The pooled point estimates of sensitivity and specificity are plotted in a filled square.
    View this figure

    When the analyses were limited to 13 studies that specified type 1 as the DM type [26,27,29,30,35,36,38,39,41,42,44,46,47], the pooled estimates (95% CI) were 0.77 (0.67-0.85) for sensitivity, 0.92 (0.84-0.96) for specificity, 9.82 (4.58-21.04) for PLR, and 0.25 (0.16-0.38) for NLR. In the analyses of 7 studies that specified night as the time of hypoglycemic events [26,30,31,35,36,41,44], the predictive ability was low compared with that of the overall analysis—pooled estimate (95% CI): 0.74 (0.65-0.82) for sensitivity, 0.81 (0.72-0.88) for specificity, 3.98 (2.64-6.00) for PLR, and 0.31 (0.23-0.43) for NLR. Relatively high sensitivity and low NLR were observed in the 13 studies that used CGM historical data for predicting hypoglycemia—pooled estimate (95% CI): 0.82 (0.71-0.90) for sensitivity, 0.92 (0.83-0.97) for specificity, 10.41 (4.52-24.01) for PLR, and 0.19 (0.12-0.32) for NLR—compared with 6 studies that did not use CGM—pooled estimate (95% CI): 0.76 (0.66-0.84) for sensitivity, 0.92 (0.88-0.95) for specificity, 10.14 (6.13-16.77) for PLR, and 0.26 (0.17-0.38) for NLR). After excluding 3 studies [31,33,54] that showed that the supposed hypoglycemic events occurred in-hospital, the pooled estimates (95% CI) of the 16 studies with such events occurring in an out-of-hospital setting were 0.82 (0.74-0.88) for sensitivity, 0.92 (0.85-0.96) for specificity, 10.58 (5.44-20.55) for PLR, and 0.20 (0.13-0.39) for NLR.


    Principal Findings

    Overall, the PLR and NLR of ML algorithms for detecting hypoglycemia were 4.05 and 0.26, respectively. These estimates were almost unchanged throughout several sensitivity analyses that were limited to studies that shared 1 characteristic in common. According to the Users’ Guide to Medical Literature with regard to diagnostic tests [56], the PLR should be 5 or more to moderately increase the probability of persons having or developing a disease and the NLR should be 0.2 or less to moderately decrease the probability of having or developing a disease after taking the index test. In summary, the current ML algorithms had insufficient ability to detect the occurrence of hypoglycemia. However, that would not mean that ECG or EEG monitoring in combination with ML, which was the case with 79% (11/14) of the included studies, was useless in detecting hypoglycemia. For example, for patients with both DM and high cardiovascular risk, in particular, those who are vulnerable to cardiac arrhythmias, using ECGs for detecting hypoglycemia is useful considering that a hypoglycemia-induced arrhythmia could contribute to increased cardiovascular mortality [57]. Similarly, for patients with repeated episodes of hypoglycemia, the combination of ML and EEG was indicated to be beneficial to prevent hypoglycemia-induced neuroglycopenia resulting in cognitive impairment and ultimately death, because blood glucose levels alone do not appear to predict that condition [58]. Thus, the clinical applicability of these devices should be evaluated by the individual’s risk of hypoglycemia and its related arrhythmia and neuroglycopenia as well as the overall ability of algorithms for ML.

    The overall sensitivity, specificity, PLR, and NLR for predicting hypoglycemia were 0.80, 0.92. 10.42, and 0.22, respectively. Applying the above described guidelines for diagnostic tests to these results, it is worth considering the use of current ML algorithms as a tool for alerting patients to impending hypoglycemic events. In addition, it is considered that a test with a PLR over 10 has a particularly strong power to alter posttest probability of the targeted disease compared with pretest probability [56]. If a positive test result were to be received, patients with DM who are administered hypoglycemic treatments would be strongly recommended to pay more attention to the possibility of impeding hypoglycemic events than they would before receiving the predictive test for hypoglycemia. However, considering that the PLR and NLR values indicate relative risk (ie, risk of disease at posttest compared with that at pretest), the accuracy of predictive ability depends on patients’ risk of hypoglycemia in daily life. For example, even a less than 10% false-positive rate (8% in this meta-analysis) may be acceptable in patients at high risk of hypoglycemia but not in low-risk individuals due to too frequent false alarms. In such a case, there is fear that these patients will ignore the alarms and therefore miss the opportunity to take corrective action when the alarm is indeed true [59]. It is emphasized that the utility of ML algorithm depends on the extent of the patient’s risk of hypoglycemia. In addition, as indicated in the “Results” section, there was high between-study heterogeneity among studies. Specifically, when limiting analyses to the studies that predicted nocturnal hypoglycemia, the predictive ability was insufficient (pooled estimate: 3.98 for PLR; 0.31 for NLR). Considering that nocturnal hypoglycemia is the most common type of hypoglycemia among all hypoglycemic episodes [60], continued research is needed for further development of ML algorithms to predict hypoglycemia.

    Several limitations of this meta-analysis should be addressed. First, the principal major limitation is the pooling of studies among which there was much variability in the type of DM, profiling data for detecting or predicting hypoglycemia, time of day when hypoglycemic events occurred, setting of supposed hypoglycemic events, and ML classification methods. In particular, although the ability for predicting hypoglycemia depended largely on the ML classification methods [33], this meta-analysis did not consider the difference in the test performance among various ML methods. Instead, the meta-analysis focused on ML’s comprehensive ability across studies using data in relation to the best model in each study, if 2 or more models existed, rather than comparisons among 2 or more models within 1 study. Given that generalization of evidence is among the most important roles in all meta-analyses, the issue of the variation in ML methods, in particular, the difference between old and new ML techniques, might be beyond the scope of this meta-analysis. Nevertheless, it should be emphasized that successful application of ML lies in the correct understanding of the advantages and disadvantages of different ML methods. Second, only 3 studies exclusively targeted patients with type 2 DM. With the increasing use of insulin to treat type 2 DM in the elderly, the prevalence of hypoglycemia is likely to escalate. In addition, the response to hypoglycemia is different between type 1 and type 2 DM [61]. Future studies should aim to develop and validate ML algorithms for detecting or predicting hypoglycemia in type 2 DM. Third, in most of the included studies, the ML classification models were developed in an experimental setting or by using previously recorded data as training and testing data instead of live data. Future studies need to train and test the algorithm on data from DM patients in everyday clinical practice to determine feasibility.

    Conclusion

    Overall, current ML algorithms have insufficient ability to detect ongoing hypoglycemia and considerable ability to predict hypoglycemia in patients with DM receiving hypoglycemic treatments. However, the clinical applicability of these ML algorithms should be evaluated according to patients’ risk profiles such as for hypoglycemia and its associated complications (eg, arrhythmia, neuroglycopenia) as well as the average ability of the ML algorithm. Continued research is required to further develop ML algorithms to enhance their feasibility, considering the inaccuracy of CGM in the hypoglycemic range, the increased prevalence of hypoglycemia in the elderly, and increasing evidence for the effectiveness of tight glycemic control in preventing microvascular complications [62].

    Acknowledgments

    All authors thank Ms Haga and Ms Chino in Niigata University for their excellent secretarial work. SK was financially supported by a Grant-in-Aid for Scientific Research from the Japan Society for the Promotion of Science (JSPS) (ID: 19K12840). The sponsor had no influence over the design and conduct of the study; collection, management, analysis, and interpretation of the data; or preparation, review, or approval of the manuscript.

    Conflicts of Interest

    None declared.

    ‎
    Multimedia Appendix 1

    Search strategy in this meta-analysis.

    DOCX File , 13 KB
    ‎
    Multimedia Appendix 2

    Study quality assessment using the quality assessment of diagnostic accuracy studies (QUADUS-2).

    DOCX File , 13 KB
    ‎
    Multimedia Appendix 3

    Study flow in this meta-analysis.

    DOCX File , 34 KB
    ‎
    Multimedia Appendix 4

    Databases which published articles that were retrieved by the search terms (see Appendix 1).

    DOCX File , 16 KB
    ‎
    Multimedia Appendix 5

    Profiling data input into ML algorithm for testing its performance.

    DOCX File , 16 KB
    ‎
    Multimedia Appendix 6

    Results of assessing study quality using revised tool for the quality assessment of diagnostic accuracy studies (QUADUS-2). The criterion corresponding to each domain (D) and signaling question (SQ) is indicated in Appendix 2.

    DOCX File , 23 KB
    1. Schopman JE, Geddes J, Frier BM. Prevalence of impaired awareness of hypoglycaemia and frequency of hypoglycaemia in insulin-treated type 2 diabetes. Diabetes Res Clin Pract 2010 Jan;87(1):64-68. [CrossRef] [Medline]
    2. Geddes J, Schopman JE, Zammitt NN, Frier BM. Prevalence of impaired awareness of hypoglycaemia in adults with Type 1 diabetes. Diabet Med 2008 Apr;25(4):501-504. [CrossRef] [Medline]
    3. Graveling AJ, Frier BM. Impaired awareness of hypoglycaemia: a review. Diabetes Metab 2010 Oct;36 Suppl 3:S64-S74. [CrossRef] [Medline]
    4. Vigersky RA. The benefits, limitations, and cost-effectiveness of advanced technologies in the management of patients with diabetes mellitus. J Diabetes Sci Technol 2015 Mar 02;9(2):320-330 [FREE Full text] [CrossRef] [Medline]
    5. Rodbard D. Continuous Glucose Monitoring: A Review of Recent Studies Demonstrating Improved Glycemic Outcomes. Diabetes Technology & Therapeutics 2017 Jun;19(S3):S-25-S-37. [CrossRef]
    6. Kovatchev B, Clarke W. Peculiarities of the Continuous Glucose Monitoring Data Stream and Their Impact on Developing Closed-Loop Control Technology. J Diabetes Sci Technol 2008 Jan;2(1):158-163. [CrossRef]
    7. Woldaregay AZ, Årsand E, Botsis T, Albers D, Mamykina L, Hartvigsen G. Data-Driven Blood Glucose Pattern Classification and Anomalies Detection: Machine-Learning Applications in Type 1 Diabetes. J Med Internet Res 2019 May 01;21(5):e11030. [CrossRef]
    8. Tyler NS, Jacobs PG. Artificial Intelligence in Decision Support Systems for Type 1 Diabetes. Sensors 2020 Jun 05;20(11):3214. [CrossRef]
    9. Whiting PF, Rutjes AWS, Westwood ME, Mallett S, Deeks JJ, Reitsma JB, et al. QUADAS-2: a revised tool for the quality assessment of diagnostic accuracy studies. Ann Intern Med 2011 Oct 18;155(8):529-536. [CrossRef] [Medline]
    10. Harbord RM, Whiting P. Metandi: Meta-analysis of Diagnostic Accuracy Using Hierarchical Logistic Regression. The Stata Journal 2018 Nov 19;9(2):211-229. [CrossRef]
    11. White IR. Multivariate Random-effects Meta-regression: Updates to Mvmeta. The Stata Journal 2018 Nov 19;11(2):255-270. [CrossRef]
    12. Higgins JPT, Thompson SG, Deeks JJ, Altman DG. Measuring inconsistency in meta-analyses. BMJ 2003 Sep 6;327(7414):557-560 [FREE Full text] [CrossRef] [Medline]
    13. Deeks JJ, Macaskill P, Irwig L. The performance of tests of publication bias and other sample size effects in systematic reviews of diagnostic test accuracy was assessed. J Clin Epidemiol 2005 Sep;58(9):882-893. [CrossRef] [Medline]
    14. Cappon G, Facchinetti A, Sparacino G, Georgiou P, Herrero P. Classification of Postprandial Glycemic Status with Application to Insulin Dosing in Type 1 Diabetes-An In Silico Proof-of-Concept. Sensors (Basel) 2019 Jul 18;19(14) [FREE Full text] [CrossRef] [Medline]
    15. Chan K, Ling S, Dillon T, Nguyen H. Diagnosis of hypoglycemic episodes using a neural network based rule discovery system. Expert Systems with Applications 2011 Aug;38(8):9799-9808. [CrossRef]
    16. Cichosz SL, Frystyk J, Hejlesen OK, Tarnow L, Fleischer J. A novel algorithm for prediction and detection of hypoglycemia based on continuous glucose monitoring and heart rate variability in patients with type 1 diabetes. J Diabetes Sci Technol 2014 Jul;8(4):731-737 [FREE Full text] [CrossRef] [Medline]
    17. Lai J, Leung F, Ling S. Hypoglycaemia detection using fuzzy inference system with intelligent optimiser. Applied Soft Computing 2014 Jul;20:54-65. [CrossRef]
    18. Ling S, Nuryani N, Nguyen H. Evolved fuzzy reasoning model for hypoglycaemic detection. Conf Proc IEEE Eng Med Biol Soc 2010;2010:4662-4665. [CrossRef] [Medline]
    19. Mathews S. Fuzzy Inference System And Multiple Regression For Detection Of Hypoglycemia. IJCSA 2012 Apr 30;2(2):37-50. [CrossRef]
    20. Nguyen HT, Jones TW. Detection of nocturnal hypoglycemic episodes using EEG signals. In: Conf Proc IEEE Eng Med Biol Soc. 2010 Presented at: In Annual International Conference of the IEEE Engineering in Medicine and Biology.Vol . IEEE; 2010; Buenos Aires, Argentina p. 4930-4933. [CrossRef]
    21. Nguyen H, Ghevondian N, Jones T. Neural-network detection of hypoglycemic episodes in children with type 1 diabetes using physiological parameters. Conf Proc IEEE Eng Med Biol Soc 2006;1:6053-6056. [CrossRef] [Medline]
    22. Nguyen L, Ling S, Jones T, Nguyen H. Identification of hypoglycemic states for patients with T1DM using various parameters derived from EEG signals. Conf Proc IEEE Eng Med Biol Soc 2011;2011:2760-2763. [CrossRef] [Medline]
    23. Vehí J, Contreras I, Oviedo S, Biagi L, Bertachi A. Prediction and prevention of hypoglycaemic events in type-1 diabetic patients using machine learning. Health Informatics J 2019 Jun 13;26(1):703-718. [CrossRef]
    24. Daskalaki E, Prountzou A, Diem P, Mougiakakou SG. Real-Time Adaptive Models for the Personalized Prediction of Glycemic Profile in Type 1 Diabetes Patients. Diabetes Technology & Therapeutics 2012 Feb;14(2):168-174. [CrossRef] [Medline]
    25. Rubega M, Scarpa F, Teodori D, Sejling A, Frandsen CS, Sparacino G. Detection of Hypoglycemia Using Measures of EEG Complexity in Type 1 Diabetes Patients. Entropy 2020 Jan 09;22(1):81. [CrossRef]
    26. Bertachi A, Viñals C, Biagi L, Contreras I, Vehí J, Conget I, et al. Prediction of Nocturnal Hypoglycemia in Adults with Type 1 Diabetes under Multiple Daily Injections Using Continuous Glucose Monitoring and Physical Activity Monitor. Sensors 2020 Mar 19;20(6):1705. [CrossRef]
    27. Dave D, DeSalvo DJ, Haridas B, McKay S, Shenoy A, Koh CJ, et al. Feature-Based Machine Learning Model for Real-Time Hypoglycemia Prediction. J Diabetes Sci Technol 2020 Jun 01:1932296820922622. [CrossRef] [Medline]
    28. Elhadd T, Mall R, Bashir M, Palotti J, Fernandez-Luque L, Farooq F, for PROFAST-Ramadan Study Group. Artificial Intelligence (AI) based machine learning models predict glucose variability and hypoglycaemia risk in patients with type 2 diabetes on a multiple drug regimen who fast during ramadan (The PROFAST - IT Ramadan study). Diabetes Res Clin Pract 2020 Nov;169:108388 [FREE Full text] [CrossRef] [Medline]
    29. Marcus Y, Eldor R, Yaron M, Shaklai S, Ish‐Shalom M, Shefer G, et al. Improving blood glucose level predictability using machine learning. Diabetes Metab Res Rev 2020 Jun 14;36(8). [CrossRef]
    30. Mosquera-Lopez C, Dodier R, Tyler NS, Wilson LM, El Youssef J, Castle JR, et al. Predicting and Preventing Nocturnal Hypoglycemia in Type 1 Diabetes Using Big Data Analytics and Decision Theoretic Analysis. Diabetes Technology & Therapeutics 2020 Nov 01;22(11):801-811. [CrossRef]
    31. Mueller L, Berhanu P, Bouchard J, Alas V, Elder K, Thai N, et al. Application of Machine Learning Models to Evaluate Hypoglycemia Risk in Type 2 Diabetes. Diabetes Ther 2020 Feb 3;11(3):681-699. [CrossRef]
    32. Ngo CQ, Chai R, Nguyen TV, Jones TW, Nguyen HT. Electroencephalogram Spectral Moments for the Detection of Nocturnal Hypoglycemia. IEEE J. Biomed. Health Inform 2020 May;24(5):1237-1245. [CrossRef]
    33. Ruan Y, Bellot A, Moysova Z, Tan G, Lumb A, Davies J. Predicting the Risk of Inpatient Hypoglycemia With Machine Learning Using Electronic Health Records. Diabetes care 2020;PMID:32350021. [CrossRef]
    34. Chen J, Lalor J, Liu W, Druhl E, Granillo E, Vimalananda VG, et al. Detecting Hypoglycemia Incidents Reported in Patients’ Secure Messages: Using Cost-Sensitive Learning and Oversampling to Reduce Data Imbalance. J Med Internet Res 2019 Mar 11;21(3):e11990. [CrossRef]
    35. Guemes A, Cappon G, Hernandez B, Reddy M, Oliver N, Georgiou P, et al. Predicting Quality of Overnight Glycaemic Control in Type 1 Diabetes Using Binary Classifiers. IEEE J Biomed Health Inform 2020 May;24(5):1439-1446. [CrossRef] [Medline]
    36. Jensen MH, Dethlefsen C, Vestergaard P, Hejlesen O. Prediction of Nocturnal Hypoglycemia From Continuous Glucose Monitoring Data in People With Type 1 Diabetes: A Proof-of-Concept Study. J Diabetes Sci Technol 2020 Mar;14(2):250-256. [CrossRef] [Medline]
    37. Jin Y, Li F, Vimalananda VG, Yu H. Automatic Detection of Hypoglycemic Events From the Electronic Health Record Notes of Diabetes Patients: Empirical Study. JMIR Med Inform 2019 Nov 08;7(4):e14340 [FREE Full text] [CrossRef] [Medline]
    38. Oviedo S, Contreras I, Quirós C, Giménez M, Conget I, Vehi J. Risk-based postprandial hypoglycemia forecasting using supervised learning. Int J Med Inform 2019 Jun;126:1-8. [CrossRef] [Medline]
    39. Reddy R, Resalat N, Wilson LM, Castle JR, El Youssef J, Jacobs PG. Prediction of Hypoglycemia During Aerobic Exercise in Adults With Type 1 Diabetes. J Diabetes Sci Technol 2019 Sep;13(5):919-927 [FREE Full text] [CrossRef] [Medline]
    40. Seo W, Lee Y, Lee S, Jin S, Park S. A machine-learning approach to predict postprandial hypoglycemia. BMC Med Inform Decis Mak 2019 Nov 06;19(1). [CrossRef]
    41. Arthur B, Lyvia B, Iván C, Ningsu L, Josep V. editors. 2018 Jul 13 Presented at: Prediction of Blood Glucose Levels And Nocturnal Hypoglycemia Using Physiological Models and Artificial Neural Networks. The 3rd International Workshop on Knowledge Discovery in Healthcare Data; July.13 .; Stockholm, Sweden; 2018; Stockholm, Sweden.
    42. Toffanin C, Del Favero S, Aiello E, Messori M, Cobelli C, Magni L. Glucose-insulin model identified in free-living conditions for hypoglycaemia prevention. Journal of Process Control 2018 Apr;64:27-36. [CrossRef]
    43. Ling SH, San PP, Nguyen HT. Non-invasive hypoglycemia monitoring system using extreme learning machine for Type 1 diabetes. ISA Trans 2016 Sep;64:440-446. [CrossRef] [Medline]
    44. Sampath S, Tkachenko P, Renard E, Pereverzev SV. Glycemic Control Indices and Their Aggregation in the Prediction of Nocturnal Hypoglycemia From Intermittent Blood Glucose Measurements. J Diabetes Sci Technol 2016 Nov;10(6):1245-1250 [FREE Full text] [CrossRef] [Medline]
    45. Sudharsan B, Peeples M, Shomali M. Hypoglycemia prediction using machine learning models for patients with type 2 diabetes. J Diabetes Sci Technol 2015 Jan;9(1):86-90 [FREE Full text] [CrossRef] [Medline]
    46. Eljil K. Predicting Hypoglycemia in Diabetic Patients using Machine Learning Techniques. United Arab Emirates: American University of Sharjah; 2014:A.
    47. Plis K, Bunescu R, Marling C, Shubrook J, Schwartz F. editors. A Machine Learning Approach to Predicting Blood Glucose Levels for Diabetes Management. AAAI Workshop: Modern Artificial Intelligence for Health Analytics; 2014.
    48. Jensen MH, Christensen TF, Tarnow L, Seto E, Dencker JM, Hejlesen OK. Real-time hypoglycemia detection from continuous glucose monitoring data of subjects with type 1 diabetes. Diabetes Technol Ther 2013 Jul;15(7):538-543. [CrossRef] [Medline]
    49. Nguyen L, Nguyen A, Ling S, Nguyen H. Combining genetic algorithm and Levenberg-Marquardt algorithm in training neural network for hypoglycemia detection using EEG signals. Conf Proc IEEE Eng Med Biol Soc 2013;2013:5386-5389. [CrossRef] [Medline]
    50. Nguyen L, Nguyen A, Ling S, Nguyen H. An adaptive strategy of classification for detecting hypoglycemia using only two EEG channels. Conf Proc IEEE Eng Med Biol Soc 2012;2012:3515-3518. [CrossRef] [Medline]
    51. Nuryani N, Ling SSH, Nguyen HT. Electrocardiographic signals and swarm-based support vector machine for hypoglycemia detection. Ann Biomed Eng 2012 Apr;40(4):934-945. [CrossRef] [Medline]
    52. Ling S, Nuryani N, Nguyen H. editors. 2010 Jul 18 Presented at: Hypoglycaemia detection for type 1 diabetic patients based on ECG parameters using Fuzzy Support Vector Machine. the International Joint Conference on Neural Networks (IJCNN); ; Barcelona, Spain; 2010; Barcelona, Spain. [CrossRef]
    53. Skladnev VN, Ghevondian N, Tarnavskii S, Paramalingam N, Jones TW. Clinical evaluation of a noninvasive alarm system for nocturnal hypoglycemia. J Diabetes Sci Technol 2010 Jan 01;4(1):67-74 [FREE Full text] [CrossRef] [Medline]
    54. Zhang Y. Predicting Occurrences of Acute Hypoglycemia during Insulin Therapy in the Intensive Care Unit. Conference proceedings: Annual International Conference of the IEEE Engineering in Medicine and Biology Society IEEE Engineering in Medicine and Biology Society Conference; 2008 Presented at: 30th Annual International Conference of the IEEE Engineering in Medicine and Biology Society; 20-25 August 2008; Vancouver, BC, Canada. [CrossRef]
    55. Iaione F, Marques JLB. Methodology for hypoglycaemia detection based on the processing, analysis and classification of the electroencephalogram. Med Biol Eng Comput 2005 Jul;43(4):501-507. [CrossRef] [Medline]
    56. Jaeschke R, Guyatt GH, Sackett DL. Users' guides to the medical literature. III. How to use an article about a diagnostic test. B. What are the results and will they help me in caring for my patients? The Evidence-Based Medicine Working Group. JAMA 1994 Mar 02;271(9):703-707. [CrossRef] [Medline]
    57. Chow E, Bernjak A, Williams S, Fawdry RA, Hibbert S, Freeman J, et al. Risk of Cardiac Arrhythmias During Hypoglycemia in Patients With Type 2 Diabetes and Cardiovascular Risk. Diabetes 2014 Apr 22;63(5):1738-1747. [CrossRef]
    58. Blaabjerg L, Juhl CB. Hypoglycemia-Induced Changes in the Electroencephalogram. J Diabetes Sci Technol 2016 Jul 28;10(6):1259-1267. [CrossRef]
    59. Palerm CC, Bequette BW. Hypoglycemia Detection and Prediction Using Continuous Glucose Monitoring—A Study on Hypoglycemic Clamp Data. J Diabetes Sci Technol 2016 Jun 24;1(5):624-629. [CrossRef]
    60. Brunton S. Nocturnal hypoglycemia: answering the challenge with long-acting insulin analogs. MedGenMed 2007;9(2):a.
    61. Zammitt NN, Frier BM. Hypoglycemia in type 2 diabetes: pathophysiology, frequency, and effects of different treatment modalities. Diabetes Care 2005 Dec;28(12):2948-2961. [Medline]
    62. Zoungas S, Arima H, Gerstein HC, Holman RR, Woodward M, Reaven P, Collaborators on Trials of Lowering Glucose (CONTROL) group. Effects of intensive glucose control on microvascular outcomes in patients with type 2 diabetes: a meta-analysis of individual participant data from randomised controlled trials. Lancet Diabetes Endocrinol 2017 Jun;5(6):431-437. [CrossRef] [Medline]

    ‎
    CGM: continuous glucose monitoring
    DM: diabetes mellitus
    HSROC: hierarchical summary receiver operating characteristic
    ICD: International Classification of Diseases
    ML: machine learning
    N-hypo: total number of events
    NLR: negative likelihood ratio
    N-total: total number of test data sets
    PLR: positive likelihood ratio

    Edited by K Mizokami-Stout; submitted 13.07.20; peer-reviewed by R Reddy, YK Lin, Y Ruan; comments to author 14.08.20; revised version received 09.11.20; accepted 07.12.20; published 29.01.21

    Copyright

    ©Satoru Kodama, Kazuya Fujihara, Haruka Shiozaki, Chika Horikawa, Mayuko Harada Yamada, Takaaki Sato, Yuta Yaguchi, Masahiko Yamamoto, Masaru Kitazawa, Midori Iwanaga, Yasuhiro Matsubayashi, Hirohito Sone. Originally published in JMIR Diabetes (http://diabetes.jmir.org), 29.01.2021.

    This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Diabetes, is properly cited. The complete bibliographic information, a link to the original publication on http://diabetes.jmir.org/, as well as this copyright and license information must be included.