GWSkyNet-Multi: A Machine-learning Multiclass Classifier for LIGO–Virgo Public Alerts

The LIGO Scientific and Virgo Collaborations (LVC) made a monumental discovery in their second observing run (O2) with the detection of the first gravitational wave (GW) originating from a binary neutron star merger. This event, known as GW170817, is the only GW source which has been associated with electromagnetic (EM) emission (Abbott et al. 2017a), marking a breakthrough for GW+EM multi-messenger astronomy. Multi-messenger astronomy has particularly exciting implications in fundamental physics; notably, joint GW and EM detections can place independent constraints on the Hubble constant (Abbott et al. 2017b), constrain the neutron star equation of state (Radice et al. 2018), and characterize the production of r-process elements in compact binary coalescences (Mészáros et al. 2019). However, EM follow-up of GW events is hindered by high operating costs, competitive telescope time allocations, false-positive GW alerts, and the short time during which the associated kilonova is detectable.

To facilitate EM follow-up, the LVC introduced the Open Public Alert (OPA) system for their third observing run (O3). OPAs are publicly available within minutes of a detection, and include a skymap (see Figure 1) indicating the localization region where a given event most likely originates (from Magee et al. 2019). An organized list of all the OPAs can be found on the Gravitational-wave Candidate Event Database (GraceDB; Stachie et al. 2021a). Although OPAs do not provide the raw GW data for a given event, they are useful in determining whether EM follow-up is warranted. To supplement the predictions available on GraceDB, Cabero et al. (2020) introduced a "real-versus-noise" binary classifier, known as GWSkyNet, which leverages OPAs to inform potential EM follow-up seconds after the OPA is published. GWSkyNet achieves a test set accuracy of 93.5% and correctly predicts 37 of the 40 O3a events published in the second Gravitational-Wave Transient Catalog (GWTC-2; Abbott et al. 2021a), before the publication of this catalog. This binary classifier demonstrates the great potential for applications of machine learning to classification of GW candidates in real-time from the information provided in OPAs. These positive results encourage the implementation of a new extension to now have a multi-class classifier.

Zoom In Zoom Out Reset image size

Here we present GWSkyNet-Multi, our new multi-class classifier which is an extension of the original GWSkyNet, and assess its performance. In contrast with the previous binary classification, our novel multi-class classifier categorizes GW candidates into their possible progenitors: (1) binary black hole (BBH) mergers, (2) mergers involving a neutron star (NS) (either NS+NS or NS+BH), or (3) non-astrophysical glitches. GWSkyNet-Multi empowers EM astronomers to make decisions on whether to follow-up an event informed by predictions of the nature of the source that supplement the information available in OPAs (Chatterjee et al. 2020; LIGO/Virgo Public Alerts User Guide ⁴ ). GWSkyNet-Multi is a sequence of three convolutional neural network (CNN) one-versus-all classifiers. As a CNN, GWSkyNet-Multi distinguishes itself from other prediction methods such as Stachie et al. (2021b) and Essick (2017) (skymap_statistics ⁵ ) the latter of which employed mutual information distance of 2D GW skymaps to distinguish between signal and noise during O2. By running all of our models on a given event, we obtain three scores quantifying the possibility that a source belongs to each class. Furthermore, using a hierarchical scheme, a refined prediction is obtained. We present this hierarchical scheme in Figure 3.

All GWSkyNet-Multi and GWSkyNet models, and scripts required to make predictions for a given event, are made publicly available. ⁶

The outline of this paper is as follows. In Section 2, we describe the new data set used for training and testing our new classifier and describe GWSkyNet-Multi. In Section 3, we asses the performance of each individual model and the overarching hierarchical classifier. We further make predictions for sources from the LVC O3 observing run. Section 4 describes the performance and interpretability of our models. In Section 5 we summarize our findings.

We begin by describing the data used to train and test GWSkyNet-Multi. We then discuss the model's architecture and the creation of the multi-class classifier.

2.1. Data Set

2.2. Developing a Multi-class Classifier

In order to make a prediction from an event, GWSkyNet-Multiuses the following inputs derived from the OPA FITS file generated by BAYESTAR(Singer et al. 2016):

• 1.  
  a skymap image;
• 2.  
  three volume-projected images;
• 3.  
  a list of detectors observing at the time of detection;
• 4.  
  estimated mean and maximum distance to the source of the GW;
• 5.  
  four normalization factors (one for the skymap image, and one for each of the three volume-projected images);
• 6.  
  the logarithm of the Bayes factors for the signal versus noise hypothesis (LogBSN) and the coherence versus incoherence hypothesis (LogBCI).

To generate a multi-class classifier, we use a sequence of three one-versus-all classifiers. All one-versus-all models have the same architecture, shown in Figure 2. Each model is used to classify an event as belonging to a given class versus all remaining classes, e.g., BBH-versus-all classifies events as either a BBH merger or anything which is not a BBH merger. We thus obtain three models: BBH-versus-all, NS-versus-all, and glitch-versus-all. We tune these models by varying the "learning rate," a scalar used to vary the magnitude of the step size in a gradient descent algorithm, as well as the "batch size," the number of examples used in each step during training. To find the hyperparameters leading to the best models for each class, we use Hparams ⁷ to explore hyperparameter space with learning rates ranging from 10⁻⁵ to 10⁻¹, and batch sizes ranging from 5 to 110. Once trained, all three models are applied to an event, and so we obtain a score quantifying the possibility that a candidate belongs to each class. Finally, to return a single prediction for our analysis, we use the hierarchical scheme shown in Figure 3.

Zoom In Zoom Out Reset image size

Zoom In Zoom Out Reset image size

For a given input, if a one-versus-all classifier returns a score above the user-set prediction threshold, the candidate is classified as a member of that particular class. If the score is below the threshold, we proceed to the next classifier in the hierarchical scheme. If none of the scores are above the models' thresholds, we classify the event based on which score was closest to its threshold.

We run models in order of best-performing to worst, based on both test set accuracies and F₁ scores:

where TP is the number of true-positive predictions, TN is the number of true negatives, FP is the number of false positives, and FN is the number of false negatives. We primarily focus on the accuracy of each model; however, we also restrict ourselves to using models with high F₁ scores (≥0.95) as they predict a sufficient amount of TPs.

We employ a hierarchical scheme to ensure that the prediction of our best-performing one-versus-all model is given priority over the predictions of the less trustworthy models. As seen in the next section, our best-performing classifier is glitch-versus-all, followed by NS-versus-all, and finally BBH-versus-all. However, note that GWSkyNet-Multi provides the user with both the hierarchical prediction and the score for each individual classifier, to maximally inform EM follow-up.

Next, we present the performance of each one-versus-all classifier, as well as the predictions of the multi-class classifier, for candidates in LIGO–Virgo's O3 run.

Figure 4 shows the false-negative rate (FNR) and false-positive rate (FPR) of the predictions on the test set made by each of the one-versus-all classifiers as a function of the threshold. We find that the best performance for each one-versus-all model occurs when the prediction threshold is set to the intersection between the FNR and the FPR. However, the user may change this threshold in GWSkyNet-Multi to suit their preferences for the FPR and FNR of the classifiers. The accuracy and F₁ score for each one-versus-all classifier, applied to the 427 examples in the test set using the intersection threshold, are presented in Table 1.

Zoom In Zoom Out Reset image size

The glitch-versus-all, NS-versus-all, and BBH-versus-all classifiers achieve accuracies of 95.1%, 94.4%, and 93.7%, and F₁ scores of 0.96, 0.95, and 0.96, respectively. To calculate the accuracy for the overall multi-class classifier, the three one-versus-all models are run sequentially on the test set as described in Section 2.2. We obtain an overall test set accuracy of 93.4% with this hierarchical scheme. The test set confusion matrix for each one-vs.-all model is presented in the Appendix. We also obtain multi-class predictions for O3a events with this hierarchical method. The hierarchical classifications and individual one-versus-all scores are reported in Table 2. Finally, although the refined classifications for O3b events have not yet been published, we present the predictions of GWSkyNet-Multi in Table 3.

To evaluate our model, we compare our O3 predictions with GWTC-2 (Abbott et al. 2021a), GraceDB, and GWSkyNet. We then characterize the interpretability of our model using Gradient-weighted Class Activation Mapping (Grad-CAM).

4.1. O3 Predictions Discussion

4.2. Ambiguous Events

4.3. Interpretability

Intuitively, we expect that a given classifier would perform best when it focuses its attention on the high-probability regions of the skymap. We tested this intuition using Grad-CAM. ⁸ Grad-CAM makes use of the fact that convolutional layers preserve spatial information to create an "activation map". Each map is produced using gradients of the convolutional layers calculated with respect to the feature map activations. From the activation map, regions of a given skymap which receive the most attention in making a given classification are identified (Selvaraju et al. 2019). With Grad-CAM, we see that this intuition is generally correct. However, we caution that we observe a general trend in the glitch-versus-all classifier, where there is more attention on the low-probability regions than is present for the other classifiers (see Figure 5). Despite this fact, the glitch-versus-all classifier performs best of all the classifiers.

Zoom In Zoom Out Reset image size

We also caution that there are notable exceptions to these expectations. For certain sources, the event is confidently and correctly classified, despite the model focusing on the exterior of the high-probability region. We show the example of S190512at in the top panels of Figure 6. In contrast, some sources are misclassified despite the relevant models placing all of their focus on the high-probability region of the given skymap. We show the example of S190923y in the bottom panels of Figure 6.

Zoom In Zoom Out Reset image size

In sum, the activation map alone is not a good indication of the validity of a classification. We leave the exploration of other interpretability techniques for future work.

In this paper, we introduce GWSkyNet-Multi, a multi-class machine learning classifier that enhances the original GWSkyNet by further categorizing GW candidates as BBH mergers, mergers containing one or more NSs, or experimental glitches. GWSkyNet-Multi is created using three one-versus-all classifiers with high accuracy and F₁ scores. The glitch-versus-all, NS-versus-all, and BBH-versus-all classifiers have test set accuracies of 95.1%, 94.4%, and 93.7%, respectively, and all F₁ scores are above 0.95. The classifier also achieves comparable accuracies on the O3a events to GWSKyNet and higher accuracies than GraceDB. These results, in addition to the high speed at which the classifiers predict, are very encouraging toward the goal of increasing the success rate of EM follow-up of GW events, especially in the upcoming O4 observing run. Furthermore, with the addition of the Kamioka Gravitational Wave Detector, and other GW detectors, OPA skymaps will offer better localizations, which will also greatly increase the likelihood of successful EM follow-up.

The authors acknowledge funding from the New Frontiers in Research Fund Exploration program. N.V. and D.H. acknowledge funding from the Bob Wares Science Innovation Prospectors Fund. D.H. acknowledges support from the Canada Research Chairs (CRC) program and the NSERC Discovery Grant program. This research has made use of data, software and/or web tools obtained from the Gravitational Wave Open Science Center (https://www.gw-openscience.org/), a service of LIGO Laboratory, the LIGO Scientific Collaboration and the Virgo Collaboration. LIGO Laboratory and Advanced LIGO are funded by the United States National Science Foundation (NSF) as well as the Science and Technology Facilities Council (STFC) of the United Kingdom, the Max Planck Society (MPS), and the State of Niedersachsen/Germany for support of the construction of Advanced LIGO and construction and operation of the GEO600 detector. Additional support for Advanced LIGO was provided by the Australian Research Council. Virgo is funded, through the European Gravitational Observatory (EGO), by the French Centre National de Recherche Scientifique (CNRS), the Italian Istituto Nazionale di Fisica Nucleare (INFN) and the Dutch Nikhef, with contributions by institutions from Belgium, Germany, Greece, Hungary, Ireland, Japan, Monaco, Poland, Portugal, Spain.

Software: astropy ⁹ : Astropy Collaboration et al. (2018); BAYESTAR ¹⁰ : Singer & Price (2016); ligo.skymap ¹¹ ; scikit-learn ¹² ; TensorFlow. ¹³

In Figure 7, we present the confusion matrices of each one-vs.-all classifier. Each confusion matrix shows the number (and percentage) of TP, FP, FN, and FP predictions on the test set.

Zoom In Zoom Out Reset image size