Comparison and benchmark of name-to-gender inference services

Related work

Bibliometric studies based on systematic assignment of gender with the purpose of analyzing specific aspects of the academic landscape have been conducted for at least a decade. Mozaffarian & Jamali (2008) studied the productivity of Iranian women in science by analyzing over 2,500 publications by Iranian authors from 2003, taken from WoS. Gender inference was done manually, resorting to an internal list of Iranian academics and to Internet searches. While not scalable, this approach showed that a high degree of familiarity with names from a particular country greatly aids the gender inference task. The article reported lower productivity of Iranian female authors with respect to their male counterparts. With a much broader scope, Frietsch et al. (2009) considered patent applications from 14 European countries between 1993 and 2001 and publications in eight scientific areas from 1996 to 2006, extracted from Scopus. Comprehensive and country-specific lists of names collected, post-processed, and tested by Naldi et al. (2005) were applied to assign a gender to inventors and authors. The analyzed data set comprised almost 2,500,000 inventors and 500,000 authors from over 150,000 publications, after rejecting over 60% of names that could not be assigned a definite gender. Findings included stark national differences in female contributions, with central European countries being the most prone to exhibit a wide gender gap.

Recently, various studies have focused on large-scale, scalable bibliometric analyses of academic publications in relation to gender. West et al. (2013) analyzed over 1,500,000 publications from JSTOR, a digital library corpus in science and humanities. They could assign a gender to 73% of authors with full first names by using data from the US Social Security Administration records. Their analysis showed that authorships by women are not evenly distributed over the dimensions time, field, coauthorship, and authorship position. A similar approach was followed by Larivière et al. (2013a), who resorted to both universal and country-specific sources such as the US and Quebec censuses, WikiName portal, Wikipedia, and various Internet sites to assign a gender to all articles between 2008 and 2012 indexed in WoS. This resulted in more than 5,000,000 articles and over 27,000,000 authorship instances, 86% of which could be assigned a gender, provided a full first name was available. They reported significant gender disparities, such as fewer citations for articles with women in dominant author positions, as well as underrepresentation of women as first authors. Analogous findings, but restricted to mathematics, were reported by Mihaljević-Brandt, Santamaría & Tullney (2016), who analyzed over 2,000,000 publications from bibliographic service zbMATH between 1970 and 2014. Using the names list from Michael (2007), they could assign a gender to 61% of all authorship instances. Most recently, Holman, Stuart-Fox & Hauser (2018) estimated the gender gap in STEMM disciplines with help of the PubMed and arXiv databases. Thirty-six million authors with publications over the last 15 years were assigned a gender using genderize.io, which is one of the services that we evaluate. Their results confirm previously reported high variations of the gender gap across countries. Furthermore, according to their data model, gender parity won’t be reached during this century in various fields, including mathematics, physics, and computer science.

It is worth mentioning that all four above-mentioned studies performed some kind of validation of their gender inference methods, yet there is room for improvement regarding assessment of manual gender labels, data size, or reproducibility. For instance, Holman, Stuart-Fox & Hauser (2018) estimate their gender misclassification rate to be 0.3% based on a collection of 372 manually labeled author names via Web searches. Considering the expected name variance, an extrapolation of the error estimate to the entire data set does not seem reliable.

Despite its importance for estimating the error rate on gender assignments, only a few studies are devoted to comparing different gender inference tools. Vanetta (2014) tested and compared four gender assignment methods (beauvoir, Sexmachine, genderPredictor, and genderize.io) on a control group provided by a government office and consisting of over 400 first names. No claims were made as to which one of the services was better, arguing that different purposes may pose different requirements to the services.

Karimi et al. (2016) used the test set from Larivière et al. (2013a) to evaluate precision, recall and accuracy of several methods, including data from various censuses, genderize.io, face recognition algorithm Face++, and two novel approaches consisting of mixing the predictions of the latter two. They reported improved accuracy of the mixed methods, noting that the quality of the assignments depended on country and was worse for non-Western regions. The brevity of the paper prevents an extended discussion regarding the definition of the quality metrics, particularly in the handling of the predicted unknowns. At any rate, face recognition techniques clearly hold potential, albeit they must be used with caution in view of their likely intrinsic bias, e.g., towards darker-skinned females (Buolamwini, 2017). Similarly, equating country of residence with regional origin of a name does not seem to be a well suited assumption, given that academics often move internationally. Holman, Stuart-Fox & Hauser (2018) also query genderize.io with the country of affiliation, potentially incurring the same bias.

A more extensive comparison is that of Wais (2016), who revisited the methods and results of Larivière et al. (2013a) and West et al. (2013), with the additional introduction of the R package genderizeR based on the genderize.io service. To compare the three approaches, a common test data set was evaluated that contained 2,641 authorships of 2,000 articles manually labeled using Internet searches. The results were compared in terms of the metrics described in ‘Performance metrics’. The method based on genderize.io outperformed the others, at least with respect to metrics that focus on retrieving a gender for as many names as possible. The percentage of non-classifications was consequently the lowest. While genderize.io and genderizeR seem to offer the best performance for gender prediction, the author points out the bias towards English names.

Recently, Bonham & Stefan (2017b) have published an analysis of female underrepresentation in the field of computational biology, for which they performed a preliminary comparison of three gender assignment tools (Bonham & Stefan, 2017a). The methods tested were the Python package GenderDetector and web APIs genderize.io and Gender API. After inferring the gender of 1,000 names they ultimately chose Gender API for its superior coverage. They did not compute further metrics to validate their election.

Lastly, a crucial discussion of the benefits as well as ethical concerns when using gender inference methods was posed in Matias (2014), which presented a multitude of examples of meaningful projects that have applied such tools to various data sources. Links to evaluations and comparisons of name-based gender inference services, including the Open Gender Tracker, which the author co-developed, were provided as well.

Overview of surveyed services

We compare five different services for inferring gender from name strings that are among the methods most frequently employed to perform gender assignments in the field of bibliometric studies. Several of them are broadly used by organizations and companies in the private sector as well. Table 1 showcases key characteristics; below we briefly describe each of them.

Assemblage of test data

We have gathered, revised, and combined human-annotated author-gender data sets used in various bibliometric studies to date, which we describe below.

Origin of the names

The assembled data set described above does not include any information regarding the origin or geographic provenience of the persons’ names. It is well known that the cultural context is an important aspect affecting the reliability of gender inference methods. We thus seek to evaluate the geographical and cultural diversity of our data set as well as to measure the impact of the names’ origins on the performance of the surveyed services. As described above, NamSor’s origin API is able to produce an anthroponomical classification of a personal name by allocating an onomastic class, i.e., a country, subregion, and region, to it. The inferred origin is reported in conjunction with a score that calibrates its reliability. According to NamSor’s internal evaluations, for names from Europe, Asia, and Africa the classifications with score > 0 are trustworthy. Note that the USA is not considered an onomastic class on its own but rather a melting pot of other ‘cultural origins’ such as Ireland or Germany (Carsenat, 2014)². Similarly, names from other parts of the Americas are considered to be of (Southern) European descent. Table 2 shows a few examples of the anthroponomical classifications produced by NamSor’s origin API.

We have applied NamSor’s origin API to our collection of 7,076 names and are able to assign a provenience to 97% (6,866) of them that return a score above 0, and that we keep for further analysis. The service was queried in May 2018. NamSor estimates 4,228 names (61%) to be of European origin, 2,304 (34%) of Asian, and 334 (5%) of African provenience. We have split the analysis by the different data sources; results are displayed in Fig. 1. The wos collection contains approximately 45% Asian and 50% European names; given the fact that wos is larger than any of the other data subsets, this ensures a satisfactory representation of Asian names in the whole test data set. For the other data sources, names of European origin clearly predominate, especially in the genderize_r subsets. In short, our data set shows a majority of European and Asian names; understandably for a sample coming from scientific publishing, the proportion of African names is small.

A more fine-grained geographical analysis focusing on countries, rather than regions, reveals that the names in our test data originate from 113 countries; however, just 16 of them already cover 75% of the whole data set. The most frequent country of origin is the UK, followed by Germany, China, and Ireland. Splitting the analysis among the five data sources confirms that both genderize_r datasets are very Eurocentric: they have almost no Asian countries among the top 10 most frequent ones. They also show the lowest diversity, with the top three countries (UK, Germany, and Ireland) covering 50% of all names and the top 10 covering 75%. On the contrary, the highest variability appears in the smallest data set zbmath, where the top three countries (Germany, Japan, and the UK) contain 26% of all names and the top 10 cover 60%. The larger wos data set exhibits the best representation of Asian origins: China, Japan, and Korea appear in positions 1st, 3rd, and 6th in terms of frequency. Full figures and statistics can be found in our dedicated Jupyter notebook in Mihaljević & Santamaría (2018).

The analysis of cultural and geographical provenience of the personal names contained in our data set brings us to the conclusion that our collection is reasonably diverse and shows an acceptable variability with respect to origin, with the aforementioned caveats about African names.

Retrieval of gender assignments

We have performed systematic queries to the services under study to obtain an inferred gender for all records in our test data set. All queries were performed in mid December 2017. Depending on their peculiarities, we sent different requests to the various APIs. Concretely, genderize.io and gender-guesser do not handle last names, therefore we queried using first and middle names only. When both a first and a middle name were present we tried their combinations (e.g., ‘Jae Il’, alternatively ‘Jae-Il’). If no response was obtained, only the first name was used (e.g., ‘Jae’). The free NamSor API requires the name to be structured as split forename(s) and last name. Gender API offers two distinct end points, one for forename(s) only and another capable of splitting unstructured full name strings. We evaluated both methods in our benchmark and found that the former performs notably better, thus we report results for that variant. NameAPI accepts unstructured strings per se. We tested it using the full name string and the forename(s) only, for better comparison with the name splitting mechanism of Gender API. However, in this case querying with the full name string achieves significantly better results, hence we report the performance of NameAPI for this variant.

Performance metrics

Name-based gender inference can be considered as a classification problem, and as such there exist plenty of metrics to measure the performance of a classifier. The choice of performance indicators needs to be suitable for the problem at hand and is usually a non-trivial task in the development of data-based algorithms. For further background readings related to classification and learning algorithms in general, such as training and testing, cross-validation, typical error estimates and statistical significance testing, see e.g., Hastie, Tibshirani & Friedman (2009) and Japkowicz & Shah (2014).

The names in our evaluation data set have been manually labeled as ‘female’, ‘male’, or ‘unknown’. Recall that those labeled as ‘unknown’ refer to individuals for whom it was not possible to find sufficient gender-related information online. Therefore the class ‘unknown’ is rather a heterogeneous label applied to people with either very common names or those that avoid providing much personal information online. In particular, it is not a ‘gender class’ in any suitable sense, and cannot be included appropriately in quantitative evaluations using performance metrics. In what follows, we will not make use of the items with manual label ‘unknown’ for any of our calculations and results, working instead with the 5,779 names in our data set which possess a defined gender label. (For a discussion of ethical considerations of name-based gender inference methods and the methodological shortcomings of such approaches, see the ‘Discussion’.) On the other hand, the services evaluated here do return, along with the responses ‘female’ and ‘male’, at least a label ‘unknown’ for the unidentified cases. Hence, in terms of the true labels we are dealing with a binary classification problem, while the predictions contain one or more extra output classes.

This makes it difficult to pose name-based inference as a standard classification problem and to utilize commonly used metrics such as precision and recall. In our case it makes sense to work instead with metrics derived from the confusion matrix defined as follows:

Let us introduce the following nomenclature for the components of the confusion matrix, which in general we will refer to as assignments: elements in the diagonal (m_m and f_f) are the correct classifications, while elements outside it (m_f and f_m) are thus misclassifications. The sum of both can be simply referred to as classifications. Consequently, elements m_u and f_u represent non-classifications, since the algorithm fails at predicting one of the classes ‘male’ or ‘female’. All mistakes, both misclassifications and non-classifications, are included under the term inaccuracies. Based on the confusion matrix, Wais (2016) introduced four performance metrics:

${\displaystyle \begin{array}{ccc}\hfill \text{errorCoded}& =\hfill & \frac{{\text{f}}_{\text{m}}+{\text{m}}_{\text{f}}+{\text{m}}_{\text{u}}+{\text{f}}_{\text{u}}}{{\text{m}}_{\text{m}}+{\text{f}}_{\text{m}}+{\text{m}}_{\text{f}}+{\text{f}}_{\text{f}}+{\text{m}}_{\text{u}}+{\text{f}}_{\text{u}}},\hfill \\ \hfill \text{errorCodedWithoutNA}& =\hfill & \frac{{\text{f}}_{\text{m}}+{\text{m}}_{\text{f}}}{{\text{m}}_{\text{m}}+{\text{f}}_{\text{m}}+{\text{m}}_{\text{f}}+{\text{f}}_{\text{f}}},\hfill \\ \hfill \text{naCoded}& =\hfill & \frac{{\text{m}}_{\text{u}}+{\text{f}}_{\text{u}}}{{\text{m}}_{\text{m}}+{\text{f}}_{\text{m}}+{\text{m}}_{\text{f}}+{\text{f}}_{\text{f}}+{\text{m}}_{\text{u}}+{\text{f}}_{\text{u}}},\hfill \\ \hfill \text{errorGenderBias}& =\hfill & \frac{{\text{m}}_{\text{f}}-{\text{f}}_{\text{m}}}{{\text{m}}_{\text{m}}+{\text{f}}_{\text{m}}+{\text{m}}_{\text{f}}+{\text{f}}_{\text{f}}}.\hfill \end{array}}$

The errors above are to be interpreted as follows: errorCoded treats a non-classification as a regular error and penalizes it in the same way as a misclassification, therefore it encodes the fraction of inaccuracies over the total number of assignments; errorCodedWithoutNA measures the share of misclassifications over the total number of classifications while ignoring non-classifications; naCoded computes the proportion of non-classifications over the total number of assignments; errorGenderBias estimates the direction of the bias in gender prediction, indicating whether there are more females misclassified as male, or vice versa. If positive, then the estimated number of women is higher than in the real data.

Depending on the concrete usage of an algorithm, these metrics can be suitable or not. For instance, if not being able to assign a gender to a large number of names is acceptable while high prediction accuracy is essential, errorCodedWithoutNA should be minimized. For most purposes, however, it is desirable to infer the gender for as many names as possible without treating non-classifications as a regular error. For this purpose we have defined two extensions of the metrics above.

Let w ∈ [0, 1]. We define the weightedError as ${\displaystyle {\text{weightedError}}_w=\frac{{\text{f}}_{\text{m}}+{\text{m}}_{\text{f}}+w\ast \left({\text{m}}_{\text{u}}+{\text{f}}_{\text{u}}\right)}{{\text{m}}_{\text{m}}+{\text{f}}_{\text{m}}+{\text{m}}_{\text{f}}+{\text{f}}_{\text{f}}+w\ast \left({\text{m}}_{\text{u}}+{\text{f}}_{\text{u}}\right)}.}$

For w = 0 the weightedError equals errorCodedWithoutNA and for w = 1 we recover errorCoded exactly. For 0 < w < 1 we define a metric which penalizes misclassifications more than non-classifications. To clarify this further, consider the following examples of confusion matrices:

For both confusion matrices the fraction of inaccuracies over all assignments is the same (and equals 0.419), while C₁ exhibits a smaller proportion of misclassifications, given that the number of non-classifications is larger than in C₂:

${\displaystyle {\text{errorCoded}}_{0.2}\left(C_1\right)=\frac{1+2+0.2\ast \left(6+4\right)}{11+1+2+7+0.2\ast \left(6+4\right)}=0.217,}$ ${\displaystyle {\text{errorCoded}}_{0.2}\left(C_2\right)=\frac{4+6+0.2\ast \left(2+1\right)}{11+4+6+7+0.2\ast \left(2+1\right)}=0.371.}$

Another even simpler possibility to penalize non-classifications without giving them the same importance as to misclassifications is to minimize a metric such as errorCodedWithoutNA, which ignores the class ‘unknown’, while enforcing a constraint on naCoded, i.e., the rate of non-classifications. Indeed any two metrics can be combined in this way. However, a disadvantage of this approach is that, for certain constraint values, there is no solution in the parameter space of a given gender inference model. Thus, it makes sense to consider the distribution of error values depending on the tuning parameters before setting a definite constraint value. In our benchmarks, if the constraint cannot be satisfied on a training set, we have fixed the test set error to 1, which is the maximum achievable value.

Parameter tuning

All methods under study with exception of gender-guesser return, in addition to the gender, one or two numerical parameters to estimate the quality of the inference. Figure 2 shows the distribution of these free parameters when using NameAPI, NamSor, Gender API, and genderize.io to assign genders to all names in our test data set. For gender-guesser (not displayed) we have created one such parameter by setting it to 0.75 for responses ‘mostly_female’ or ‘mostly_male’ and 1 for ‘female’ or ‘male’. Figure 2A suggests that NameAPI’s parameter confidence exhibits a bimodal distribution that peaks at around 0.90, with a secondary maximum at 1 (absolute certainty on the gender assignment). All names reach a confidence of at least 0.70. Figure 2B indicates that NamSor assigns a gender with absolute certainty to most of the names in the data set, although some outliers are spread out at smaller values of the scale parameter. Figures 2C and 2D show the two-dimensional parameter spaces of Gender API and genderize.io: one parameter encodes the number of appearances of a name (denominated samples and count, respectively), and the other shows the confidence on its gender assignment (named accuracy and probability). Most names fall in the bottom right region of high confidence and low number of counts.

In general, there is no mathematical model that can be trained in the classical sense of machine learning. Instead, for every service, we have ‘trained’ (or ‘tuned’) the algorithms by trying out randomly sampled parameter values. Assuming that a gender classification might not be reliable under some given threshold for the confidence indicators, we have searched for those parameter values that minimize a certain error. A particular instantiation of a random grid of sampled parameters used for each service (except gender-guesser) is displayed as black dots in Fig. 2 as well.

Benchmark setting

We define a series of benchmarks to compare the methods under study. To begin with, we compute performance metrics using the default gender responses for all services, i.e., with no error minimization through parameter tuning (Benchmark 1). Next, we introduce further Benchmarks 2, 3, and 4, each of which concentrates on a particular scenario defined by conditions on the metrics to be optimized.

All performed benchmarks are evaluated two-fold: (a) on the entire data set, and (b) differentiating between the five data sources as described in ‘Assemblage of test data’. The latter case is particularly relevant e.g., for researchers in scientometrics looking for the most suitable gender inference algorithm for a particular data source such as PubMed, whereas the former is most appropriate when analyzing different data collections, as e.g., in Holman, Stuart-Fox & Hauser (2018). To better illustrate the role of the names’ origin, Benchmark 1b is additionally broken down by geographical regions first and then by Asian subregions, which turn out to be the most challenging.

For Benchmarks 2a, 3a, and 4a we run 10 trials of 10-fold cross-validation. In each, we randomly select (at most) 40 parameter values per service for training, and record the average error on the test sets with those parameters that minimized the respective error on the training sets. The tuning parameters as well as the training/test set splits are randomly selected with fixed seed numbers, thus ensuring reproducibility.

In Benchmarks 2b, 3b, and 4b we perform one trial of 10-fold cross-validation per service and data set, and record the average error on the test sets. Since the individual data sets differ significantly in size, we have used at most 20 parameters for the smaller data sets zbmath, genderize_r_authors and genderize_r_titles. For the larger data sets pubmed and wos, we have allowed at most 30 and 35 tuning parameters, respectively.

Finally, we apply several tests to assess the statistical significance of the observed differences. In the b-versions of our benchmarks, we first apply the non-parametric Friedman test and then suitable post-hoc tests. We refrain from using ANOVA, which is the usually recommended parametric alternative to evaluate multiple classifiers on multiple data sets, since the homogeneous covariance assumption is not satisfied across all services and data sets. In each benchmark, we compare the two best performers using suitable parametric or non-parametric tests for two classifiers. For more details on statistical significance tests suited for classification algorithms, see Demšar (2006) and Japkowicz & Shah (2014)[Chapter 6].

The code for the benchmark evaluations can be found in Mihaljević & Santamaría (2018).

Benchmark 1: Default responses

First, we use the gender inference services employing their default responses, i.e., considering all ‘female’ and ‘male’ gender attributions regardless of the value of the confidence parameters.

Benchmark 1b: analysis by name origin and data source

Next, we investigate the impact of the names’ origin on the performance of the services under evaluation. As described above, all services return a confidence parameter indicating the trustworthiness of the classification; recall that for Python package gender-guesser we have created one such parameter by setting it to 0.75 for responses ‘mostly_female’ or ‘mostly_male’ and 1 for ‘female’ or ‘male’. We investigate the confidence parameters for different geographical origins of the test names. The boxplots in Fig. 3 show statistics from the quartiles of the parameters’ distributions, split by the top regions predicted by NamSor’s origin API.

Note that all services produce responses that are discernibly dependent on the names’ origin: the most confident gender predictions are for names of European origin, while Asian names generally lead to a lower median and a higher deviation. The service NameAPI displayed in Fig. 3D stands out insofar as the medians of confidence values are lower than those of the other services, indicating a different kind of normalization. This is in accordance with the bimodal parameter distribution peaking at around 0.90 which is depicted in Fig. 2A. There is also little difference between the median values for all three geographical regions, suggesting that NameAPI’s confidence parameter is not as useful to discriminate among easy versus complex cases. It is also worth noting in Fig. 3C that service genderize.io assigns gender to Asian names with significantly higher confidence values than the other services: the median value for Asian names is surprisingly almost as high as for European names.

The lower confidence in gender assignments for Asian names reported by almost all services suggests focusing on that case. As shown in Fig. 4, which displays results broken down by Asian subregions, names from Eastern and Southeastern Asia yield significantly smaller values than other Asian regions. This is to be expected, in particular due to Chinese names for which the difficulty of inferring a gender of Latin transcriptions is well known. Nonetheless, while NamSor and gender-guesser have almost no confidence in the gender inference of East Asian names, Gender API shows a notably high median value. NameAPI and genderize.io again exhibit similar medians for all subregions, confirming that the values of their confidence parameters are decoupled from the names’ origins, and thus from the complexity of the assignment. This fact makes us doubt that NameAPI’s confidence and genderize.io’s probability parameters are sufficiently significant.

Table 5 quantifies errors incurred by the different services depending on the names’ origin. Gender API achieves the best results for errorCoded, however its performance is strongly affected by the names’ origin, being one order of magnitude worse for Asian (18% inaccuracies) than for European names (3%). NamSor performs similarly for European (3%) and African names (7%), but is considerably worse for Asian ones (35%). Regarding the fraction of misclassifications (errorCodedWithoutNA) we note that gender-guesser, with its small but highly accurate database, achieves a mere 1.5% error in classifying European names, while for Asian names the figure increases to 6%. Generally speaking, we conclude that all services clearly show the challenging nature of inferring gender for Asian names.

Table 5:

Benchmark 1b, name origin: performance of all services with their default gender assignments in terms of the metrics errorCoded and errorCodedWithoutNA, broken down by name origin.

Values are rounded to four decimal figures.

	errorCoded			errorCodedWithoutNA
	Africa	Asia	Europe	Africa	Asia	Europe
Gender API	0.0538	0.1759	0.0281	0.0469	0.112	0.0213
gender-guesser	0.2437	0.5171	0.0752	0.0365	0.0641	0.0147
genderize.io	0.1039	0.3282	0.0507	0.053	0.1206	0.0218
NameAPI	0.1505	0.3772	0.0807	0.0405	0.0897	0.0136
NamSor	0.0645	0.3459	0.0273	0.044	0.0903	0.0211

DOI: 10.7717/peerjcs.156/table-5

Next let us consider errors in gender inference depending on the source of the names as displayed in Table 6. For both errors all services perform much worse on the wos data subset than on the others. This is in accordance with the fact that the wos collection incorporates a higher percentage of Asian names, as evidenced by Fig. 1. Regardless, the results follow the general trend: Gender API achieves the smallest fraction of inaccuracies for all data sources, whereas gender-guesser often beats other services in terms of misclassifications. Overall we conclude that the breakdown of errors by data source is consistent with the analysis split by names’ origin. Data sets composed of Western names have a much larger chance of being correctly attributed a gender than Asian ones.

Table 6:

Benchmark 1b, data source: Performance of all services with their default gender assignments in terms of the metrics errorCoded and errorCodedWithoutNA, broken down by data source.

Values are rounded to four decimal figures.

	errorCoded					errorCodedWithoutNA
	zbmath	genderize_r_authors	genderize_r_titles	pubmed	wos	zbmath	genderize_r_authors	genderize_r_titles	pubmed	wos
Gender API	0.0086	0.0289	0.034	0.04	0.1327	0.0029	0.0123	0.0173	0.0294	0.0853
gender-guesser	0.0659	0.0795	0.0787	0.1154	0.3699	0.0031	0.0052	0.0137	0.0105	0.0544
genderize.io	0.0659	0.0675	0.0809	0.0697	0.2296	0.0091	0.0203	0.0336	0.0235	0.0872
NameAPI	0.063	0.0988	0.0723	0.103	0.283	0.018	0.0158	0.0113	0.021	0.0572
NamSor	0.043	0.0289	0.0404	0.0697	0.214	0.006	0.0123	0.0217	0.024	0.0741

DOI: 10.7717/peerjcs.156/table-6

Benchmark 2: Minimization of inaccuracies constrained to a 5% misclassification error

We measure the performance of all services with respect to the fraction of inaccuracies (errorCoded) under the constraint that at most 5% of all successfully classified names are misclassified as female or male, i.e., they fulfill errorCodedWithoutNA < 0.05. This is a realistic constellation for applications requiring the rate of misclassifications not to exceed a given threshold.

Benchmark 3: Minimization of misclassifications constrained to a 25% non-classification rate

Next we evaluate the effectiveness for achieving correct classifications, i.e., we minimize the rate of misclassifications (errorCodedWithoutNA) constrained to the amount of names that cannot be classified being lower than 25% of all assignments (naCoded < 0.25). This represents a rather frequent situation of wanting to incur as few misclassifications as possible, while at the same time being able to assign a gender to at least a pre-defined fraction of the evaluated names.

Benchmark 4: Minimization of the weighted error with w = 0.2

Finally we analyze the case of minimizing the weighted error with w = 0.2, namely the metric that treats all inaccuracies (misclassifications and non-classifications) as errors, but puts five times more weight into the former. This corresponds to an intermediate situation between Benchmarks 1 and 2, and the approach has the flexibility of allowing a continuous range of values for the weight w, depending on the particular needs of each analysis.