Pump Up Password Security! Evaluating and Enhancing Risk-Based Authentication on a Real-World Large-Scale Online Service

Despite decades of efforts to replace passwords with more secure alternatives [9], they are still the main authentication method for the majority of online services [47]. The recent rise of data breaches further increased the threats to password-based authentication [29, 35, 63]. Credential stuffing and password spraying attacks automatically enter leaked login credentials (username and password) on other websites in the hope that users have reused them [16, 28]. Such attacks can be scaled with little effort and promise financial returns, which is why they are very popular today [4, 43]. Targeted password guessing methods using machine learning (ML) are less scalable but also very effective [44, 68]. This increases the need for responsible online services to protect their users. Two-factor authentication (2FA) [46] is a commonly offered measure to increase account security, but users tend to refuse it outside of use cases involving sensitive and financial data, such as online banking [20, 37, 41, 48, 49, 65, 70]. The 2FA adoption rates are low on popular online services, for example, 2.5% on Twitter [65] and \(\approx\) 4% on Facebook in 2021 [41]. Google also has a low 2FA adoption rate [37, 46], which is why they tried to force 2FA on 150M of their users, that is, less than 10% [25], by the end of 2021 [26]. The outcome of this experiment regarding user acceptance, however, is unclear to date. To still protect users without enabled 2FA and to increase the cost for attackers, risk-based authentication (RBA) is a solution that is preferred by users over equivalent 2FA in many use case scenarios [70]. It can be considered a scalable interim solution to strengthen password-based authentication until more secure authentication methods are in place. Therefore, government agencies such as the National Institute of Standards and Technology (NIST, United States), the National Cyber Security Centre (NCSC, United Kingdom), and the Australian Cyber Security Centre (ACSC, Australia) recommend RBA to protect users against attacks involving stolen passwords [7, 27, 40]. US federal agencies even have to establish RBA across their enterprise by presidential executive order [8].

Different RBA models exist to derive a risk score for a login attempt. The first approach was published by Freeman et al. [24] in 2016 and is based on statistical relations. Other models, some more related to the area of implicit authentication, are based on more complex ML schemes such as Siamese neural networks [2, 17, 51] and one-class support vector machines (OCSVM) [38]. Other relevant ML schemes for outlier detection include isolation forest, local outlier factor, and robust covariance [54]. They have not been seen in RBA literature yet, but we found them relevant for our model selection. We used the RBA model of Freeman et al. since a preliminary evaluation showed it to be the most suitable for our dataset. We outline the model selection in the following.

We aimed to focus on scalable and practical RBA solutions that can be used with large-scale online services. Like other popular online services observed in practice [71], the online service collected categorical feature data, for example, IP address and user agent. Two types of RBA models exist in literature that focus on categorical data [69]. One type, the most published one (SIMPLE model), is very simple and checks only exact matches in the user’s login history [18, 30, 42, 61]. The other type, published by Freeman et al. [24] and probably used by popular online services [71], compares statistical feature relations between all users. This model outperformed the other models in a previous evaluation [69]. To provide a baseline comparison, we also repeated this evaluation on the large-scale online service (see Appendix E). Both results confirm that the SIMPLE model cannot block the same high amount of attackers as the model by Freeman et al. The latter also allowed more fine-grained security configurations than the SIMPLE model. For instance, varying a tiny fraction of the access threshold greatly lowered VPN attacker protection from 96% to 53% with the SIMPLE model on the large-scale online service. In contrast, attacker protection lowered for only a tiny fraction with the model by Freeman et al. On a geographically close user base, it also blocked significantly more targeted attackers while making significantly fewer requests to re-authenticate [69].

We also evaluated ML schemes for outlier detection on a subset of our collected data to assess their capabilities in terms of RBA risk estimation. However, they did not qualify for our use case for the following reasons: (i) Our feature data are categorical with a nominal scale, that is, they have no ranking. We did not find a rational reason to rank the categorical feature values in descending order, with each having an individual rank (e.g., is Firefox Mobile a better feature than Firefox Desktop?). Most of these ML schemes, however, currently showed good performance only with numerical data having an interval scale, for example, time or sensor measurements [2, 17, 51]. We acknowledge that nominal categorical data without a ranking can also be transformed into binary numerical data with one-hot encoding [53]. Due to their properties, however, unknown values—such as a new user agent—can be transformed into only one value, that is, the “unknown” category [10]. As this one-hot encoded value has a low scalar distance to the known values, the classification produced inaccurate results in our tests. (ii) The outlier detection of these mechanisms proved to be suitable only when including the user’s own login history in the training [38]. Otherwise, as the global feature values in our data were largely distributed, the mechanisms would not detect any outliers in most cases. The data per average user, however, were very sparse at the online service, that is, four categorical data points per feature (see Section 4). Based on our tests, this was not enough for sensible outlier classification using most of these models, that is, almost every login risk is high unless all feature values are the same. (iii) In contrast to the selected model, these ML- and non-ML–based mechanisms did not allow for a relation between local and global feature value distributions. This led to inaccurate results with respect to newly established legitimate feature values, for example, new browser versions across multiple users. (iv) In addition, the ML mechanisms need to be trained with new feature values for each login to optimize the outlier detection for each user. Otherwise, legitimate users would potentially be prompted for re-authentication more often, as new legitimate feature values would be lacking in the trained model. This, in combination with the classification, was time-consuming compared with the Freeman et al. model (e.g., 2.5 s for isolation forest and 270 ms for OCSVM, compared with 0.003 ms for the Freeman et al. model). Therefore, we did not consider them practical for large-scale online services in their current form.

As a result, to focus on a practical and scalable solution for large-scale online services, we chose the RBA model of Freeman et al. [24] for further evaluation. Nevertheless, as its risk score has a numerical interval scale, we still found ML mechanisms useful to optimize the selected RBA model parameters (see Section 10). Parameter optimization can be used as a tool for administrators to set up RBA as effectively as possible in their individual environments.

The selected RBA model is comparable to a multi-features model derived from observations on the behavior of popular online services [71]. According to the observations, the online services Google, Amazon, and LinkedIn used this model and probably still use it in some form [71]. The model calculates the risk score \(S\) for user \(u\) and a given feature set \((FV^1, \ldots , FV^d)\) with \(d\) features asThe calculation uses the probabilities \(p(FV^k)\) of a feature value in the global login history, and \(p(FV^k | u, legit)\) of a feature value in the legitimate user’s login history. The user login probability is based on the proportion of the user logging in, that is, \(p(u | legit) = \frac{Number\ of\ user\ logins}{Number\ of\ all\ logins}\) . Attack data can optionally be included to protect users at risk. As in related work [69], we considered all users equally likely being attacked when not having attack data. Thus, we set \(p(u | attack) = \frac{1}{|U|}\) with \(U\) being the set of users with \(u \in U\) . We did not use attack data in most of our studies, unless otherwise noted, to be able to compare our results to a related study [69]. We also assumed that use cases without attack data are more common in practice, especially for medium and small websites that have limited storage and computing capacity.

When considering attack data, we estimated \(p(u | attack)\) based on the number of failed login attempts per user . We expect that users with a high number of failed login attempts are likely being targeted in credential stuffing or password spraying attacks. If the user is not present in the attack data, we set the probability to the minimum value: \(p(u | attack) = \frac{1}{Total\ number\ of \ logins}\) . We did this as it is still possible that other users are being attacked.

Furthermore, the equation can be extended to include an attack probability for the used feature values:In our use case, \(p(attack | FV^k)\) is the number of feature value occurrences in the failed login attempts . Similarly, wxg361 e returned the minimum probability when the feature value was not present in the attack data, that is, \(p(attack | FV^k) = \frac{1}{Total\ number\ of\ attacks}\) . Returning a minimum probability is necessary, as we assume that there is always a residual risk of other feature values being used in an attack .

We collected login feature data of 3.3M users between February 2020 and February 2021 at an SSO service belonging to the Norwegian telecommunications company Telenor Their customers used the SSO service to access other services provided by the company, for example, cloud storage or billing information services. The users could authenticate with their username in two ways: either by entering a password or by letting the online service’s mobile network verify a mobile phone number (see Section 12). In some cases, users also had to verify their mobile phone number or email address by entering a one-time code (OTP) that was sent to the corresponding number or address.

We evaluated the RBA model with four attacker models, with three of them based on known models in literature [24, 69, 72]. Due to verified account takeover data and the relevance in real-world scenarios [12, 63], we introduce the additional very targeted attacker. We describe the attacker models in this section. All attackers possess the login credentials of the victim (see Figure 2).

The naïve attacker signs in from a random IP address from somewhere in the world. We simulated this attacker with IP addresses of failed login attempts that match a dataset identifying real-world attacks [23]. We expect that a large number of failed login attempts for a user likely indicates that the user was targeted in an attack . However, the majority of identified attacks came from a particular ASN and a country that was far from the online service’s main country. Therefore, detecting those would be trivial. To better diversify the sample, and to simulate global attackers, we included only the most occurring IP address per ASN. Then, we extracted two types of naïve attackers to simulate a wide range of them. Half of the attacks resembled a very naïve attacker type comparable to one that randomly tries to enter a login credential. These attacks involved random IP addresses from all 180 countries identified in the dataset.⁴ The user agents were also randomly selected. The other half of attacks resembled naïve attackers that were able to buy additional resources, for example, servers or botnets. These attacks involved the top IP addresses and user agents having most occurrences in the dataset.

The VPN attacker knows the country of the victim. Therefore, the attacker spoofs the IP geolocation using a VPN connection and popular user agent strings. The IP address does not necessarily have to belong to a VPN service provider. We simulated this attacker with identified attack IP addresses located in the users’ main country [23]. As this attacker does not target specific users, we randomly sampled these attack IP addresses from the failed login attempts. The attacker used the most popular user agents from legitimate login attempts .

The targeted attacker knows the location, device, and browser of the victim. Therefore, the attacker uses IP addresses and user agent strings that the victim would likely choose. We simulated this attacker with identified attack IP addresses located in the users’ main country [23]. As the attacker targets specific user groups, we sampled the IP addresses and user agents of the failed login attempts with most occurrences.

The very targeted attacker is the strongest of our attacker models, which attacks a victim only once, but with all information gathered about it. This could, but not necessarily, include accessing the device or the network of the victim by malware. We simulated this attacker by replaying the 87 identified successful account takeovers. We did this to check how many of these very targeted attacks would be detected by the RBA system in practice.

We did the calculations for our analysis on a high-performance computing (HPC) cluster with more than 2,400 CPU cores and 7,500 GB RAM. This was crucial, as calculating the risk scores was computational intensive, especially with the large dataset used. Computing the results on an eight-core desktop PC would have taken more than a year. The HPC cluster reduced this computation time to approximately 1.5 weeks.

Feature Set. Unless otherwise noted, our RBA model used IP address and user agent string as features. The subfeatures were ASN and country for the IP address, and browser, OS, and device type for the user agent string. We did this to allow a fair comparison to the related study and since this feature set can be considered the state of RBA practice [69, 71].

Optimizations. The huge dataset size largely affected computation time and limited computing capacity . Thus, we optimized the RBA algorithm to speed up calculation (see Section 11.1). The optimizations allowed us to calculate the risk scores for all legitimate logins. However, we still had to limit the attacker calculations to a data subset to be able to obtain the results after a reasonable period of time.

Stratification. Due to the increased storage and computational load caused by the attacker simulation, we calculated the attackers’ risk scores for naïve, VPN, and targeted attackers on a stratified dataset containing 10% of users. We stratified the user set based on the number of logins to retain the dataset properties regarding login histories as efficiently as possible. Stratifying datasets is common practice in ML solutions [55]. To focus our attack detection on real-world attacks, the stratified set included those users who had the highest number of failed login attempts, that is, likely targeted users.

To make sure that the stratification did not influence the general validity of our results, we compared the results for RQ1 using all legitimate logins to those using a stratified dataset. The results were identical for the very large majority of login history sizes lower than 100. Thus, we considered the stratified sample appropriate for our analysis.

Significance Testing. For statistical tests, we used Kruskal-Wallis (K-W) for the omnibus cases and Dunn’s multiple comparison test with Bonferroni correction (Dunn-Bonferroni) for post-hoc analysis. We considered p values lower than 0.05 as significant.

To estimate RBA’s performance on a large online service, we repeated a related study [69] with our dataset. The related study estimated RBA characteristics on a small online service. In addition, since the dataset was large enough to do this, we further analyzed the number of re-authentication requests depending on login frequency. We used the same methodology, as it has already been used for RBA analysis. This allowed us to compare the results.

The RBA model offered two possibilities to include attack data (see Section 3) that had not been investigated to date. Thus, we evaluate the attack data variations in this section. We simulated the login behavior as in the previous study (see Section 7.1). However, we added the two different attack data variations in our risk score calculation. We outline the procedures together with the results in the following.

User Attack Data (RQ2a). In the first variation, we used the user attack probability \(p(u | attack)\) calculated from the number of failed login attempts in the attack data. Figure 6 shows the results. When adding the user attack probability, the re-authentication rate significantly increased for naïve, VPN, and targeted attackers (p \(\ll\) 0.0001). For very targeted attackers, however, the re-authentication rate significantly decreased for TPRs lower than 0.94 (p \(\ll\) 0.0001). Thus, very targeted attackers could be better distinguished from legitimate users than before, whereas the opposite was true for the other attackers. User and Feature Value Attack Data (RQ2b). In the second variation, we further added probabilities for feature values being used in an attack. Therefore, we used Equation (2) (see Section 3) with both the user attack \(p(u | attack)\) and the feature attack probability \(p(attack | FV^k)\) calculated from the failed login attempt data.

When also including the feature attack probability, the re-authentication rates significantly increased for all attacker models (p \(\ll\) 0.0001, see Figure 6). Thus, attackers could not be distinguished from legitimate users as well as before.

Discussion. Using the user attack data in the RBA model reduced re-authentication requests when blocking very targeted attackers. As the targeted users were likely attacked before, the risk scores increased for them. However, the risk scores decreased for attacks on those users who were not attacked before. Thus, the usability performed worse for the other attacker models that targeted a wide range of users. Therefore, the decision to include attack data should be made with great caution. Related work suggests that other metrics, such as social media followers [24] or frequently guessed passwords [64], could help to identify accounts that should receive higher protection. This, however, requires further research.

Adding feature attack data made it harder for the RBA system to distinguish between legitimate users and attackers. Attackers identified in the dataset likely chose popular feature values, for example, the user agent. Thus, legitimate users having these feature values received a high risk score. Attackers trying unpopular feature values, however, likely received risk scores in the range of legitimate users using popular feature values. As a result, legitimate users could not be distinguished from attackers as well as before. With this is mind, we do not recommend using the tested feature attack data variant in the RBA model to calculate the risk score.

Over the last few years, an increasing number of data protection regulations aimed to protect users from massive data collection by online services. These regulations, such as the CCPA [60] and GDPR [22], suggest limiting data storage as much as possible. In terms of RBA, this means that the feature data have to be disposed of as soon as they no longer serve the purpose of RBA [73]. One measure to minimize data is to delete global login history entries after a fixed amount of months [73]. Such a data-limiting procedure is also useful to maintain an acceptable authentication speed [69]. To estimate the potential of the minimization approach, we evaluated it in the coming sections.

In order to optimize the usability and security properties of RBA, administrators need to analyze the risk scores of legitimate users and attackers to set a suitable access threshold. This is a time-consuming and complex process, however. For this reason, we analyzed whether the RBA configuration process can be automated and improved with ML mechanisms to achieve a good RBA setup in a short time.

Using RBA with large global login history sizes can greatly increase risk score calculation time [69]. To be able to compute the results in reasonable time, we optimized the RBA algorithm. We describe our optimizations, including an analysis, in this section. The optimizations are also applicable to productive environments.

Wiefling et al. [69] proposed a server-originated RTT feature based on the WebSocket technology [36] to estimate device location. The server requests a data packet (ping frame) from the client and measures the time until the response (pong frame). Only devices near the server location can achieve low RTTs, which is why this is hard to spoof for attackers without high effort — they need access to a device physically located in the target area. Thus, this feature can help to determine whether the client’s device location is really in the indicated region or spoofed by VPNs or proxies [1, 12]. Content Delivery Networks (CDNs) can even improve the reliability of RTT, as edge nodes close to the client’s device are also considered for the measurement [69]. Concurrent and independent work confirmed this potential [33]. For most re-identification attacks with leaked data, the RTT is useless because it depends on the server location, which is widely distributed in practice. Therefore, the RTT also has potential to be a privacy-enhancing alternative to the sensitive IP address feature [73]. To investigate its potential as a location verifier and as a drop-in replacement for the IP address feature, we evaluated the RTT feature’s abilities.

The online service did not collect RTTs for all users during login. However, the online service offered mobile users the option to verify their mobile phone number when connected to the service’s network provider. In this case, the online service sent a request to the mobile phone and the phone responded. The service measured the time (in ms) for the response, which can be considered an RTT variant. The RTT was recorded in 5.1M login attempts (2.5M successful, 2.6M failed), which were 17.6% of all login attempts.

The study results presented in this article are representative of a certain country (Norway) and online service type (SSO of telecommunication services). They do not represent all online services worldwide; rather, they show an example of a typical large-scale online service with sensitive data involved.

Since the dataset corresponds to the global browser market share (see Section 4.1), our results can help to apply RBA globally. Nevertheless, cultural differences in some countries may lead to different login patterns. For example, people in rural Myanmar rely on mobile shop staff to set up online accounts for them [21]. In this case, online services can notice that many different user accounts were created over the same Internet connection. From a Western perspective, this behavior may seem suspicious, but it is normal in these areas. Therefore, it makes sense for service owners to tailor their RBA systems to cultural conditions in the target locations to achieve a high user acceptance.

The RTT was measured only once per login attempt. We assume that the RSRs for the RTT would improve with multiple RTT measurements per login attempt, as shown in Wiefling et al. [69] with five RTT measurements during the login process.

With our contributions, we extend the related work in various aspects, which we discuss in this section.

Freeman et al. [24] tested their RBA model in a case study using 300K legitimate logins collected during 6 months on the popular large-scale online service LinkedIn. However, they did not provide an analysis of RBA characteristics in practice. Also, for such a large-scale online service, the sample was rather small. To provide a more realistic estimate, we analyzed RBA characteristics using 12.5M legitimate logins collected in more than one year. Wiefling et al. [69] tested the model by Freeman et al. on a small online service with 9,555 legitimate logins collected over 1.8 years. Their and our results confirmed that IP address, user agent string, and RTT are useful RBA features. Our work also extended the analysis for different login frequencies. Wiefling et al. [73] proposed and tested privacy enhancements for RBA. Due to their small dataset, they were not able to test their proposed login history minimization approach. We filled this research gap and tested this approach on a large-scale online service.

Alaca and van Oorschot [5] evaluated a list of features regarding their ability to identify users. Regarding distinguishing information, they rated the IP address feature higher than the RTT. They noted, however, that the latter required further study. There were different RTT implementations for RBA proposed in concurrent and independent work. Rivera et al. [51] proposed a client-originated version. Wiefling et al. [69] proposed a server-initiated solution that they considered harder to spoof for attackers. The studies in both works showed the RTT’s potential to verify countries [51] and users [69]. Our work further showed RTT’s potential to verify regions and to replace the IP address as a feature. Campobasso and Allodi [12] studied a criminal infrastructure that used malware-infected devices to bypass RBA. The RTT feature can detect such attempts, as the infrastructure used SOCKS5 proxies.

Doerfler et al. [19] evaluated RBA re-authentication challenges on a sample of 1.2M legitimate Google users. They stated that their RBA implementation was able to block 99.99% of automated account takeover attempts and 92% of phishing attacks. Our study results with 3.3M legitimate users reflect their findings.

Besides RBA, there are other related methods that aim to protect user accounts against automated attacks. CAPTCHAs [66] are a widely used measure to block bot traffic by presenting a challenge that, in contrast to bots, is solvable by humans with a lower error rate. Some RBA systems also present CAPTCHAs on some occasions to increase friction only for bot-like traffic [71]. However, solving CAPTCHAs is still error prone for humans [13, 50]; thus, achieving usable CAPTCHAs can be considered a challenge. Bots can also bypass them by proxying the CAPTCHA challenge to real humans [71] or machine learning algorithms [62]. These bypass methods are scalable in practice. RBA, in contrast, depends on the specific user. Thus, proxying cannot scale when RBA uses appropriate features, for example, the RTT.

Another method is attribute-based authentication using attribute-based credentials (ABCs) [11]. In this case, users authenticate to an online service by providing a verified attribute, for example, their age, to prove that they are over 18 years old. The verification is done by a trusted authority and the information presented to the online service can be done using pseudonyms. Therefore, online services receive only the information that is required for their service, for example, that the user is over 18, without getting personal details. ABCs can increase user privacy but are independent from RBA, as the latter do not require verified personal details. However, ABCs can be considered to be a re-authentication challenge in high-risk environments in which users expect increased security, for example, online shopping or online banking [52, 70].

Online services are a popular attack target [4]. Stolen passwords used, for example, in credential stuffing attacks pose new security challenges to service developers, operators, and owners [19]. As these attacks have increased over the years [3, 4], RBA has become an important measure for online services to protect their users. NIST, NCSC, and ACSC recommend RBA [7, 27, 40]; some have recommended it for more than 3 years. Still, current literature does not provide any insights on implementing, configuring, and operating RBA on these types of online services. These insights are important, though, to achieve RBA’s security gain while balancing usability and privacy. Only a fraction of work exists that focuses on RBA algorithms or studies, all of which is on small online services. We studied RBA characteristics on a large-scale online service to close this research gap.

Our results, based on our dataset of a large online service, confirm previous results, based on a small online service, that RBA rarely requests re-authentication in practice, even when blocking more than 99% of targeted attackers. As new results, our evaluation furthermore shows that RBA can even detect a high number of very targeted account takeover attempts. The RBA behavior also significantly depends on the users’ login frequencies. For daily-use websites, RBA can achieve very high security with few re-authentication requests. For monthly-use websites, the percentage of blocked attackers needs to be lowered down to around 2% to 3% to achieve a similar usability for targeted attackers. Still, a high percentage of attackers would be blocked for all user types. Therefore, we consider RBA an effective measure to protect users from the majority of attacks involving stolen passwords. In addition, RTT proved to be a promising RBA feature to verify regions and to identify users while increasing user privacy.

RBA needs to be carefully evaluated for and adapted to each individual use case scenario. Based on our findings, RBA system designers should consider the following to find the best configuration for their environment: (i) The amount of minimizing the login history needs to be carefully adjusted to users’ login frequencies. Online services with a daily user base should keep at least 6 months of login history for a stable setup that blocks targeted attackers. When blocking naïve attackers only, which represented 97% of all identified incidents at the online service, reducing to 3 months is possible. (ii) Attack data should be used with caution or not at all, as these can greatly decrease usability and security properties. (iii) To achieve an optimal basic configuration in a short period of time, administrators can use the introduced ML-based RBA parameter optimization. For further adjustment, they can increase or reduce the resulting access thresholds to decrease or increase the security and average re-authentication count, respectively. (iv) Non-optimized RBA is potentially vulnerable to denial of service when multiple users log in at similar times. To mitigate this, and to considerably shorten authentication time, implementation of the RBA algorithm needs to be carefully designed using efficient data structures such as hash tables.

Our results and the synthesized dataset support online services to gain a widespread understanding of RBA. In future work, we will use these insights to create an open-source RBA solution that significantly improves usability, security, and privacy compared with the only open solution currently available, OpenAM [42]. The dataset will be used as testing data to evaluate our solution.

We would like to thank Rudolf Berrendorf and Javed Razzaq for providing us a huge amount of computational power for our big data analysis. Also, thanks to Florian Dehling and Jan Tolsdorf for providing feedback on the paper.

Our synthesized dataset can be downloaded at https://github.com/das-group/rba-dataset. As this article introduces our open dataset, we ask researchers to cite this article whenever they publish their own results obtained from adopting our dataset. For broad compatibility, we provide the data as a CSV file. Table 6 shows the available features in the synthesized dataset and the data types. We kept the feature values in readable notation to facilitate understanding. The values may need to be converted to more efficient data types to speed up risk score calculation. For instance, the IP address could be converted to its integer value. Other string-based features could be hashed and converted to integer values as well.

Common synthetic data generation methods rely on learning the real values of a dataset, which, however, does not provide a strong privacy guarantee [57]. Therefore, we used our own generation method that was tailored to the use in RBA models employed by the majority of online services in the wild [24, 42, 69]. We created the synthetic dataset as follows. As common RBA models rely only on the number of occurrences of categorical data, we did not have to reuse any of the original feature values. We transformed all unique dataset values to a random pseudonymous identifier and added randomness to the timely order to keep the statistical relations between different feature values. Then, we randomly assigned feature values from a Location-to-ASN-to-IP dataset in backwards hierarchy (i.e., Country \(\rightarrow\) Region \(\rightarrow\) City \(\rightarrow\) ASN \(\rightarrow\) IP address). If the original IP address was used in an attack, we made sure that the randomly assigned IP address was an attack IP address as well. We repeated the same procedure for the user agent strings. Depending on the synthesized geolocation data and the login success status, we randomly drew the RTTs for the corresponding entries to ensure that the RTT results were realistic. Finally, we generated the login timestamps with random noise following the distributions and timely order (see Figure 10 in Section 11).

We verified the synthesized dataset by recalculating the study results from this article. As our generation process does not reuse feature values from the original dataset, the synthesized dataset should not provide any personal identifiable information. Nevertheless, we evaluated the privacy attributes of the synthesized dataset using Stadler and Oprisanu [56] and other comparable frameworks.

To provide a baseline evaluation, we repeated the study of RQ1a and RQ1c (see Section 7) with the SIMPLE model used in the open-source SSO solution OpenAM [42] (see Section 2). The model checks the feature values for an exact match in the user’s login history. In contrast to the Freeman et al. model [24], the SIMPLE model does not allow taking attack data into account. As in the other studies, the model used the IP address and user agent string as features. The results show that the achievable TPRs were very coarse grained, for example, the TPR for targeted attackers dropped from 0.9552 to 0.5295 when the access threshold was lowered by a tiny fraction to the next possible level (Figure 18). This confirms previous findings [69]. Also, the high end of TPRs varied largely across the different attackers, for example, 0.9816 for naïve attackers and 0.9494 for VPN attackers. This uncertainty makes the SIMPLE model difficult to configure for large-scale online services.