7.2 Results
We now present the results in order of the research questions, followed by a discussion.
Number of Re-authentication Requests (RQ1a). The users logged in 3.8 times on mean average. Therefore, we considered a user’s login history size of 4 to estimate the re-authentication count for the average user. The median re-authentication count decreased with an increasing login history size (Figure
3).
When configuring RBA to block 99.5% of naïve attackers, the average legitimate user was asked for re-authentication every second login. When blocking a lower percentage, users were prompted every fourth login (TPR 0.99–0.95) or not at all (TPR <0.95). When blocking VPN or targeted attackers, legitimate users were prompted every second (TPR 0.995–0.99) or fourth time (0.98–0.90). When blocking at least 82.76% of very targeted attackers, legitimate users were asked every login attempt.
For a fair comparison to the related study [
69], we also considered a login history size of 12 to estimate the median login count until re-authentication. In this case, users were asked every fourth login or even less often for TPRs lower than 0.995 (Table
1). This is similar to the related study. Summarizing our findings, RBA rarely asks for re-authentication in practice, even when blocking a high number of targeted attackers.
Re-authentication Requests per Login Frequency (RQ1b). We classified the users of our dataset into five groups having different login frequencies: daily, several days per week, once a week (weekly), once in several weeks, and once in more than 30 days (less than monthly). For each user, we calculated the time differences between successive logins. We then classified the user to the group to which the majority of time differences belonged (e.g., the user used the online service mainly on a daily basis). To ensure a sufficient data basis for the classification, we then dropped all users who logged in on less than two different months, two different weeks, and four different days, depending on the class. We based the limit on the median login count and our own observations on the different user groups in the dataset. Based on the labels, we calculated the re-authentication counts per login frequency .
The median number of re-authentication requests varied significantly between the different login frequency groups (Table
2). Daily users were asked significantly less for re-authentication than those who logged in once in several days and less than several weeks. For instance, when aiming to block 99% of VPN and targeted attackers, the average daily user had to re-authenticate every fourth login, whereas weekly or less frequently logging in users had to do it almost every time (Figure
4). Less-than-monthly users were also prompted for re-authentication significantly more than those who logged in daily, weekly, or once in several weeks. Concluding the results, daily users were mostly less asked for re-authentication than those who logged in less frequently.
Required Login History Size (RQ1c). In order to achieve RBA’s usability gain, users need to notice a difference to 2FA [
70]. A baseline for this can be to request re-authentication less than every second time [
69]. Thus, we considered the required login history size as the point after which the median re-authentication rate remained below 0.5.
Following this definition, most TPRs lower than 0.99 required one or no entry in the login history for a stable setup (see Figure
3). The other TPRs required 4 (naïve attacker, TPR 0.995; VPN attacker, TPR 0.99), six (targeted attacker, TPR 0.99), 8 (VPN and targeted attacker, TPR 0.995), and 18 entries (very targeted attacker, TPR 0.8276). This would achieve a baseline setup that, on median, requests re-authentication less than every second time.
However, the required login history size differed based on the login frequency of the users. Even when blocking 99.5% of all targeted attackers, the median re-authentication rate for daily users remained below the baseline after a login history size of 4 (see Figure
4). Thus, in this case, 4 entries were required for a stable setup targeted at daily users. This was different from the other user types, which required at least 8 entries for targeted attackers in this case. The results for the very targeted attacker did not pass the baseline for TPRs greater than 0.89 until a login history size of at least 40.
In general, the login history of each user does not have to contain a large amount of entries for a stable setup. Based on the results, we conclude that storing 8 entries is sufficient for a stable RBA setup that blocks 99.5% of targeted attackers at this online service. When blocking naïve or VPN attackers, 4 to 6 entries are required in that case.
Comparison with Small Online Service (RQ1d). Wiefling et al. [
69] evaluated RBA on a small online service having 780 users and 9,555 legitimate logins. Due to the same setup, we can compare both studies. Our results mainly reflect their findings (Figure
5). Users were less requested for re-authentication with an increasing login history size. Although our re-authentication rates mostly declined slower, the main tendency was the same. Therefore, the RBA characteristics did not differ greatly between a large and small online service.
7.3 Discussion
The vast majority of identified attacks on the online service came from a different country than the victim (2.2M incidents, 97%), representing naïve attackers. VPN, targeted, and very targeted attacks using the same country as the victim were less common (58K incidents, 3%). Therefore, blocking naïve attackers already helps to protect against most attacks in practice. Freeman et al. [
24] reported similar findings regarding botnet attacks, that is, only 1% involved the same country.
The data for very targeted attackers were very sparse, which is why the TPRs were very coarse grained. Still, the results show that RBA was able to detect the majority of successful account takeovers (78.16%) with low re-authentication rates.
The results confirm that RBA can achieve good usability and security after only a few login attempts. However, this varies depending on the user type. Less-than-monthly users had to re-authenticate significantly more than daily users. This is due to the fact that feature values are more likely to differ after a longer period of time because, for example, the device or browser was updated or a new IP address was set [
6].
Daily to weekly users received less frequent re-authentication requests, even when RBA was configured for high security. Therefore, RBA can achieve a high acceptance among these users in practice [
70]. Less frequent users, however, would be prompted for re-authentication almost every time in this case. The burden of re-authentication is limited here, as users are prompted once per month. Thus, we expect that they will tend to accept it [
15,
32,
70]. However, when being surprised by a re-authentication request after some months, there is a risk of users not being able to solve the re-authentication challenge [
70]. This could result in users being annoyed by RBA [
15,
32,
70]. To mitigate this effect, we recommend to set the targeted TPR based on the login frequencies of the general user base. The TPR can be set high for a mostly daily user base, while it has to be lowered for less frequent users. For instance, to achieve a user experience for less-than-monthly users that is comparable to daily users, the TPR needs to be lowered from 0.995 to 0.97 for targeted attackers at this online service (see Figure
4).