The Impact of Judgment Variability on the Consistency of Offline Effectiveness Measures

As noted in Section 1, conduct of an offline experiment requires three resources: a collection of documents, a set of queries with answers among those documents, and a set of relevance judgments, often referred to as qrels. A common protocol for developing the relevance judgments associated with a given topic is to take a set of representative retrieval systems, generate a run from each for that topic, and then pool the runs to some selected depth d to form the union of the top-d sets. This can be done to a uniform pre-chosen depth d across each of the topics; or, if there is a judgment budget that must be complied with, then to a depth that is variable on a per-topic basis, balancing the overall judgment budget equally across the topics. A range of other judgment strategies have also been proposed [16, 31, 32, 38] that in various ways seek to shape the pool of documents to achieve the greatest benefit, where benefit is often (but usually on an informal basis) equated with “finding the greatest number of relevant documents.”

It is effectively impossible to create comprehensive judgments in which every document is assessed against every query—in all but inconsequential collections the cross-product of topics and queries is just too huge for this to be viable. Any given pooling strategy thus develops a tiny minority of the available judgments, and investigations have been carried out to determine the extent to which the collected judgments are comprehensive [47, 64]. In response, techniques for quantifying the effect of missing judgments uncertainty have been proposed [37], and recommendations provided in regard to experimental methodologies that measure and disclose the degree to which the available qrels are a fit for the experiment being reported on [41]. Other researchers have sought to develop techniques for scoring incomplete rankings that bypass, or infer values for, any unjudged documents [3, 7, 44]. In addition, comparisons of absolute and relative system orderings have been carried out, considering the extent to which incomplete judgments affect experimental outcomes [12, 59, 63].

Regardless of the issues associated with judgment coverage, there are also other variables that potentially affect the quality of any judgments that are collected, including document characteristics, judgment conditions, information requirement statement, and assessor behavior [29].

Most of the studies mentioned so far have employed multiple assessor groups, for example, expert and non-expert, to calculate the inter-assessor consistency; in general, they have argued that, even with non-negligible fractions of disagreement among assessors, the quality of the test collection and the relative ranking of runs is only moderately unaffected. For example, Voorhees [59] conducted a study involving several TREC collections that assessed the impact of NIST assessor judgment disagreements on relative system ordering when measured via AP, finding that the rank correlation of the runs remained high even as the groups of assessors were altered. Whether the ordering of the strongest systems remained consistent was not examined, nor was preservation or loss of significance, but the overall outcome was of similar results from different judgments.

This result had been anticipated by Lesk and Salton [29] in a much smaller experiment making use of a collection of 1,268 documents, who attributed the stability of relative system orderings to three contributing factors: effectiveness measures report the average performance over topics; assessor disagreements tend to arise more with low-ranking documents than with high-ranking ones; and effectiveness is measured on the relative positions of the relevant and irrelevant documents. Voorhees [59] disagreed with those three suggested explanations for system stability and reported that the larger size of TREC collections and the diversity of topics were the reasons. Voorhees also noted that, for queries with a small number of relevant documents, effectiveness metrics such as AP were unstable. Using a different collection, a similar experiment was conducted by Burgin [10], comparing four groups of assessors, again finding that the resultant system orderings were relatively consistent.

In another early study, Cleverdon [15] introduced random changes to the set of relevant documents, namely, for a subset of topics, a random number of irrelevant documents replaced the known relevant documents. Cleverdon compared the relative rankings of different indexing methods using four independent sets of relevance judgments and found that system rank order was largely unaffected. Lee and Kantor [28] also carried out an early study of judgment consistency; and Saracevic [49] surveys much of that early work.

More recently, Bailey et al. [4] compared three groups of judges, with a group of expert judges asked to judge a subset of the topics, while two other groups with decreased levels of expertise judged all of the topics. Bailey et al. found that on a per-topic basis system scores varied as the judging groups were switched; and that while per-topic score variability was common, average system score variations were smaller.

Scholer et al. [50] developed further experiments to explore intra-assessor consistency, asking for each judge how stable their judgments are over time. Scholer et al. looked at duplicates and assessor consistency, as well as analyzed the impact on system orderings arising from factors such as the time between assessments, the previous decision made by the assessor, and the ordering of the document sequence. They found that order of judgment was the most significant influence on the consistency of each person’s judgments.

Overall, these various investigations suggest that experimental outcomes—at least to the extent of system orderings as measured via an unweighted correlation coefficient—tend to be insensitive to errors in the relevance judgments. Our detailed experiments in Section 5 using a top-weighted correlation coefficient suggest that such a view is only partially reliable.

Webber et al. [63] experimented with a value they called document meta-rank, using it to build a model of assessor behavior. Document meta-ranks are computed using the popularity of the document, its rank in the returned results for the topic, and its relevance score. They then compared meta-ranks and assessor disagreement and found that, while there is only limited disagreement on the top-meta-ranking relevant documents for a topic, the assessor might overturn the judgment with a probability close to 50% for low meta-ranking documents.

Soboroff et al. [55] investigated the impact that random relevance assessments have on the system orderings. They generated pseudo-qrels by using a normal distribution centered around the ratio of relevant to irrelevant documents, sampling randomly on a per-topic basis. Although overall system orderings remained stable, the best-performing runs were the most affected by this sampling. Soboroff et al. also explored shallow pooling (top 10 retrieved) to increase the chance of including a rare relevant document in the pseudo-qrels. The use of a shallow pool improved the rank correlation of the runs for several of the ad hoc tracks that were explored.

Carterette and Soboroff [12] enumerated several patterns of assessor behavior that could affect the quality of relevance judgments, namely, assessor optimism, fatigue, patience, enthusiasm, and conditional dependence of an assessor’s judgments on their previous decisions. Carterette and Soboroff implemented their model on the Million Query Track and generated new query relevance judgments per topic, where they reported that assessor models that underestimate the number of relevant documents generate more accurate system orderings.

Li and Smucker [30] suggest that assessors be modeled using two orthogonal variables, discrimination and bias. Bias models the assessor’s liberal, neutral, or conservative judging behavior, while discrimination accounts for their ability to discriminate relevant documents from irrelevant. Li and Smucker simulated query relevance judgments using a range of values for their assessor behavior model, finding that AP is relatively resilient to change when measured by rank correlation, and that while deep measures such as AP and NDCG benefit from conservative relevance assessments, shallow measures such as P@10 require less conservative assessing behavior.

In more recent work, Ferrante et al. [17] use an unsupervised approach, where the ratio of relevant to irrelevant documents is not known to assessors, and a uniform distribution is used to select a document as being relevant or not. Ferrante et al. consider random, underestimating, and overestimating assessor behavior models. Their framework merges different performance measures based on the estimated accuracy of crowd workers and results in more stable system orderings as well as a more accurate prediction of system scores. Ferrante et al. [18] go on to consider the question of relevance itself, exploring the consequences of modeling relevance via continuous-valued binomial variables, as a further broadening of the traditional binary relevance assumption.

We make use of both the Li and Smucker [30] and Webber et al. [63] models of assessor behavior, adopting them because of their probability-based formalisms in terms of how to achieve simulated disruptions that then translate directly into an implementation and because of their plausible underlying models of assessor behavior.

The ability to make relevance assessments was at one time regarded as requiring reasonably good language and analytical skills [59]. More recently, the advent of crowdsourcing platforms has allowed the role to be broadened and for much wider pools of people to be considered, each of them doing a smaller volume of work and with individual variance (or errors) smoothed by amalgamating the responses of multiple people for each query–document pair. That ease of access has, in turn, led to further studies in terms of judgment consistency and of assessor agreement, and of the consequential effect they have on experimental outcomes; see, for example, Büttcher et al. [11], Vuurens and de Vries [61], Kazai et al. [24], Kazai et al. [25], Scholer et al. [51], Turpin et al. [56], Maddalena et al. [35], Kutlu et al. [27], and Han et al. [22].

Another area in which crowdsourcing has been useful is as a method for eliciting queries. Traditional offline methodologies make use of a single query associated with each topic; if bootstrapping is applied, then samples with replacement are drawn from that fixed set of “canonical” queries. However, different users are likely to generate different queries even for the same underlying information need [6, 9, 39, 40, 66]. Crowd workers can be used to create such query variations as a response to a supplied backstory, a process that has been used to add query variations to existing test collections [5] and also as an integral part of the collection formation process [33].

Another critical aspect of offline evaluation is the use of statistical confidence testing. With multiple topic scores typically being averaged to obtain a system score, it is expected (and reasonably so) that a p value will be computed, as evidence of the likely strength of any claimed relativity. Sakai [45] has surveyed testing prevalence; and other work has considered the complex question of which test to make use of [13, 20, 54, 57, 58]. Following that advice, in our experiments here, we make extensive use of the Student t-test, which has been found to be robust for small samples and reliable with regard to Type-I errors [57].

Another key statistical technique is that of bootstrapping. If a single set of n observations drawn from an unknown distribution is all that is available, then an arbitrary number of further statistically indistinguishable n-samples can be formed by sampling with replacement from that known set, thereby allowing certain attributes of the overall distribution to be inferred from the accumulation of corresponding individual attributes [43]. For example, a confidence interval for the mean of the underlying distribution can be established by considering the properties of hundreds or thousands of such “with-replacement” n-samples. When n is large, the distribution of replication factors for each individual item can be approximated by \({{Poisson}}(k;1) = [(1/e)/k!\mid k\in \lbrace 0,1,2,3,\ldots \rbrace ] =[ 0.367879, 0.367879, 0.183940, 0.061313, 0.015328, 0.003066, \ldots ]\), that is, with each original item appearing zero times in any given replicate with a probability of \(0.367879\), appearing one time with an equal probability of \(0.367879\), appearing twice with a probability of \(0.183940\), and so on. Bootstrapping across a set of n topics in this way allows other sets of topics to be inferred and statistical confidence to be established experimentally.

Other experimental work has made use of the fact that random assignment of documents across a set of partitions also generates replicate collections with identical statistical properties [60]; of the fact that collections can be split based on source, domain, and content type, to generate distinct sub-collections [48]; and of the observation that all of topic, system, and sub-collection can be combined into a model that allows multiple system comparisons [19]. Related work has created sub-collections (also known as “shards”) and studied the impact of topic-shard interaction on system performance [21].

Zobel and Rashidi [65] combined these ideas, introducing the notion of collection bootstrapping, complementing previous evaluations that had made use of randomized collection splitting. Each generated bootstrapped collection is “like” the seed collection, but not identical to it, with each document replicated a number of times given by \({ {Poisson}}(k;1)\). Statistics gathered over a large number of bootstrapped collection replicates can then be used to infer statistics in regard to the original collection. In follow-up work, Rashidi et al. [42] introduced a controllable systematic bias to the random collection sampling process and explored the extent to which slightly different collections resulted in different experimental outcomes.

The first strategy we describe was originally explored by Li and Smucker [30]; we refer to it as random qrel flips. The intuition behind this method is that crowd workers such as Amazon Mechanical Turkers are unlikely to be subject experts in the tasks that they are asked to assess and are also likely to be working at speed because they get paid by the item, not by the hour. Both factors can be argued as likely to lead to systematic random assessment errors.

In this strategy, assessor behavior is modeled in terms of true positive and false negative rates, TPR and FPR. Li and Smucker [30] use signal detection theory as a model for describing assessors’ discrimination and bias, denoted as disc and bias, respectively, and defined as:where \(z(\cdot)\) denotes the inverse of the normal distribution function, disc denotes the assessor’s ability to discriminate between a relevant and irrelevant document, and bias denotes the extent to which the user’s judgments are liberal or conservative. As disc increases, the reliability of the assessor is increased. In tension with that relationship, as bias increases, the assessor becomes more reluctant to deem a document relevant, that is, more conservative in their willingness to make a positive judgment.

It is thus possible to express TPR and FPR in terms of the discrimination and the bias:where \({{ {CDF}}}(\cdot)\) is the cumulative density function of the standard normal distribution \(\mathcal {N}(0, 1)\). That is, by varying disc and bias it is possible to simulate a range of assessor behaviors.

In the random qrel flips perturbation approach, we assume that TPR and FPR are constant across all query–document pairs and attributes of the assessment environment as a whole. The position in the run of each document is not considered, and the rate of flip is the same for all queries. Figure 1(a) shows the corresponding TPR and FPR values for assessors across a range of discrimination factors, \(0.5\le disc\le 3\); and a range of biases, \(-2\le bias\le 2\). Note that when \(bias=0\) the expected number of relevant documents remains the same after perturbation.

The second judgment perturbation strategy we consider is rank-biased qrel flips. This mechanism is designed to mimic the errors made by expert assessors and proceeds from the assumption that rare relevant documents and popular irrelevant documents are the ones most likely to be misclassified. To model that propensity, Webber et al. [63] introduce the notion of the meta-rank of a document, which is an aggregation of its rank position in the set of runs generated by the available systems. As was the case with the work of Webber et al., the meta-rank method we use here is due to Aslam et al. [2], who suggest computation of meta-AP as a rank fusion technique calculated by averaging the contribution of each document in each run based on its rank position k in the run. The document’s overall meta-AP score (which is only loosely connected to the metric AP) then reflects its normalized contribution across the set of runs.

The meta-AP of a document d is calculated as the average across the set of runs of that document’s contribution scores, where the contribution score for a document at rank k in a run of length N is given by \(1+H_{N}-H_{k}\) if \(k\le N\) and is zero otherwise; and where \(H_k=\sum _{i=1}^{k}(1/i)\) is the k th harmonic number. For large N, we have \(H_N\approx \gamma + \ln N\) where \(\gamma\) is a constant, meaning that \(1+H_{N}-H_{k} \approx \ln (eN/k)\), and hence that the average of the document’s contributions is proportional to the natural log of the geometric mean of that document’s fractional positions across the set of runs, with fractional positions expressed as ratios relative to N. In the experiments described in Section 5, we typically employ \(N=1,\!000\), which is the maximum run length in the ad hoc and robust tracks of TREC. As indicative values, the meta-AP of a document that is retrieved at rank 1 by all systems is 7.5 when \(N=1,\!000\); if the same document is retrieved at rank 2 by all systems, then it gets a meta-AP of 7.0; and if a document is retrieved at rank 10 by half of the systems and not retrieved at all by the other half, then it gets a meta-AP score of 2.8. Alternative run aggregation techniques could also be used to compute meta-ranks, and in other work have been used in rank fusion over query variations as well as over systems [6].

In analysis based on TREC datasets, Webber et al. [63] found that there is only limited disagreement between judges for relevant documents with a high meta-AP score, but that disagreements for relevant documents with low meta-AP scores approached 50%. Conversely, for irrelevant documents, low meta-rank documents tended to yield agreements between different assessors, while high meta-rank documents were often subject to disagreements. These observations suggest that, given a sequence of document ranks derived from a set of pooled runs, it is reasonable to use meta-rank as the basis of a model of the probability with which each document’s relevance judgment might be flipped during the judgment elicitation process. More precisely, for expert assessors TPR and TNR are not distributed uniformly across the documents. Instead, TPR possesses a positive correlation with the aggregate rank of a document, and TNR has a negative correlation, the latter relationship then meaning that FPR also has a positive correlation.

Webber et al. [63] propose logistic regression models that estimate the probabilities of assessor agreement and disagreement for relevant and non-relevant documents, in each case as a function of meta-rank. They suppose that an assessor A has deemed a document d relevant or irrelevant and estimate the probability p of a second assessor B also judging that document as relevant. Using the two corresponding pairs of parameters \((\beta _0,\beta _1)\) shown in Table 1, the two required probabilities are modeled using two applications (one parameter pair \((\beta _0,\beta _1)\) for \(A=1\), and then a second pair \((\beta _0,\beta _1)\) for \(A=0\)) of the fitted equation:That is, we take the attestments, that is, the original qrels, as being assessor A’s decision and seek to simulate assessor B via this model for each value for A. These two probabilities correspond to the TPR and FPR values in the rank-biased qrel flips strategy. The experiments described in Section 5 use Disks 4 and 5 from TREC; we accordingly make use of the parameters derived by Webber et al. [63] for the TREC 6 collection, as shown in Table 1. Figure 1(b) then shows how TPR and FPR vary, given the two logistic regression models. At the left-hand (low) end of the meta-AP range shown on the horizontal axis, relevant documents (the blue line) have a probability of less than \(50\%\) of retaining their positive label when judged by the second B assessor; while at the right-hand (high) end of the meta-AP range, non-relevant documents (the red line) are predicted to be almost certainly (incorrectly) judged as being relevant when considered by a second B assessor. Note that in the rank-biased qrel flips approach TPR and FPR are independently parameterized, so—in contrast to random qrel flips and Figure 1(a)—it is not meaningful to plot them against each other.

Likely error rates on real data for both random and ranked models of error are discussed in Section 5.2.

Our interest is in how perturbed relevance judgments can affect system scores and system comparisons and in how sensitive measurements are to the level of perturbation. We therefore require multiple sets of judgoids, which we generate by using the random qrel flips and rank-biased qrel flips approaches, with different discrimination and bias factors, which in turn lead to varying rates for TPR and FPR. To achieve any desired levels of TPR and FPR, we make use of the relationships noted by Li and Smucker [30] (see Section 3.1) connecting disc and bias on the one hand and \({{ {TPR}}}\) and FPR on the other.

The overall process for generating perturbed query relevance judgments for each topic is shown by the pseudo-code in Figures 2 and 3. When the random qrel flips strategy is to be applied in Figure 3 the two sets \({ {wgts}}_0\) and \({ {wgts}}_1\) contain all-equal values; whereas if rank-biased qrel flips is to be applied, they contain the logistic-mapped (that is, via Equation (1)) meta-AP scores of each document that has been judged for the selected topic, as a set of values between zero and one that indicate the relative preference for that document being in any of the generated subsets.

Each set of judgoids for a topic is then computed as follows: For a given level of discrimination and bias, steps 3 –6 in Figure 2 first count the non-relevant judgments (set \(q_0\)) and the relevant judgments (set \(q_1\)); and then use the sizes of those two sets, plus the \({ {disc}}\) and \({ {bias}}\) parameters, to compute a false positive judgment count \(n_{0\rightarrow 1}\) and a true positive judgment count \(n_{1\rightarrow 1}\). Those two values are the respective target sizes for random subsets of \(q_0\) and \(q_1\), with that selection process undertaken at steps 13 –19. In between, two sets of weights are constructed, one containing a value between zero and one for each current non-relevant item, denoting its relative propensity to be flipped and thus become relevant; and one containing a value between zero and one for each current relevant item, similarly denoting its relative propensity to be retained as relevant. Those two \({ {wgts}}\) arrays are what drives the process shown in Figure 3, which uses them as guidance to bias the probabilities used to select a random non-uniform subset of its argument S.

Note that the two \({ {wgts}}[]\) arrays are not probability distributions and do not each sum to one—the values that they contain simply express relative emphases on the elements. Note also that under certain circumstances those emphases need to be moderated so the target subset sizes (\(n_{0\rightarrow 1}\) elements from \({ {wgts}}_0[]\), and \(n_{1\rightarrow 1}\) from \({ {wgts}}_1[]\)) can be achieved in expectation. The process shown in Figure 3 carries out that balancing and then performs the subset generation. A key component is that the “sense” of the subset extraction is reversed if the relative ratios between the weights would suggest that any element must be over-sampled. If such a condition arises (the test at step 4), then the statements through to step 7 create a complement set based on reverse weights (the difference of each from 1.0) and then upon return the sense is flipped back and the selected elements removed to make the desired subset.

Figure 4 provides two examples of how the selection is done, in both cases taking as input the same vector \({ {wgts}}[]\) of \(n=9\) relative selection weights. In Figure 4(a) a subset of \(n^{\prime }=2\) of the items is to be randomly selected. The computed \({ {multiplier}}\) (step 8 in Figure 3) is less than 1.0, and so the operation can proceed using the per-item selection probabilities shown in the corresponding green cells, which sum to 2.0. With each of the \(n=9\) selection decisions an independent random event, summed across the \(n=9\) items there will thus be, in expectation, \(n^{\prime }=2\) that are chosen. In Figure 4(b) a subset of size \(n^{\prime }=7\) is sought. Now, the computed \({ {multiplier}}\) is greater than 1, and per-item selection probabilities cannot be employed, because some might end up being greater than 1.0. To handle this situation, the vector of weights is complemented relative to the upper limit of 1.0 (step 5), and then the same process applied via a single recursive call (step 6) to determine a vector of exclusion probabilities (again, shown in green in the figure) of \(n^{\prime \prime }=n-n^{\prime }=2\) items to be removed (step 7), thereby forming the desired subset.

The random number generation process at step 12 of Figure 3 means that each time the function \({weighted}\_{subset}()\) is called, a different non-uniform sample of the attestments is selected and either flipped when taken from \(q_0\) or retained when taken from \(q_1\). Any desired number of alternative sets of judgoids can thus be constructed by iterating the mechanism described by the pseudo-code shown in the two figures.

We make use of the binary judgments created for two TREC² datasets: TREC 7 (Disks 4 and 5) and TREC Robust 2004 (Disks 4 and 5, minus the congressional records). The dataset statistics are summarized in Table 2. These judgments, which are created to a high standard by the TREC processes, are used in our experiments as attestments. They undoubtedly do still contain errors, but of necessity are assumed to be a usable approximation of true attestments.

All 50 queries associated with TREC 7 were used. The Robust 2004 collection has a much larger number of associated topics; these were divided into two sets, one with 50 queries and another with 199 queries, to allow consideration of topic set size upon experimental stability; in this collection, the mean judged relevant and irrelevant documents across the full set of 249 queries are 70 and \(1{,}181\), respectively. The sampling process used to obtain the two subsets was that every fifth query was assigned to the smaller query set. The runs from the top 50 systems in these two TREC rounds were selected, with “top” defined by system average for the metric rank-biased precision (RBP) [37] using a persistence value of \(\phi =0.95\), which corresponds to users who, on average, examine the first 20 documents in each run and who value relevance in rank position 1 approximately 2.65 times more highly than they value relevance in rank position 20. As a check, we also selected the top 50 systems using AP and NDCG, and across the four combinations of metric and collection the minimum top-50 overlap against RBP was 46 out of 50.

We also used RBP (again with \(\phi =0.95\), a value that should be assumed throughout unless otherwise noted) as a metric with which to evaluate runs, along with several other well-known approaches: average precision (AP); precision at 10 (P@10); normalized discounted cumulative gain (NDCG) [23] with log base 2; and reciprocal rank (RR). Note that the use of \(\phi =0.95\) broadly corresponds to the expected evaluation depth attained by AP and NDCG [41].

Rank-biased overlap (RBO) [62] was employed in experiments that required comparison of system rankings. Rank-biased overlap is a top-weighted list similarity measurement that assigns larger penalties to differences at the head of a ranking than it does to differences that occur further down; we use a parameter of \(\phi =0.90\) except where otherwise specified, which corresponds to a probabilistic viewer of the rankings pair who on average scans from the top of the two runs, exits after looking at an average of 10 pairs of documents, and then assesses the degree of overlap they have observed. We mainly use RBO to compare system orderings and hence evaluate it to depth 50, the number of systems employed in the experiments.

Rank-biased overlap is an overlap coefficient and is zero only if the two lists are disjoint. When the two lists are permutations of each other, there is an expected “background” RBO score that can be determined by a Monte Carlo technique that generates and scores a large number of random permutations. That process was used to establish the floor value applicable to these experiments: permutations of length 50 and a parameter of \(\phi =0.90\). The expected RBO for random orderings was found to be \(0.193887\pm 0.071999\)—that is, around 0.2, implying that reported RBO values of less than around 0.3 should be interpreted as meaning that two orderings each containing 50 systems are uncorrelated. We also computed Kendall’s tau in connection with system orderings, noting that it is not top-weighted.

Each experiment—involving a single combination of parameters such as discrimination and bias—involved the generation of 100 perturbed qrels files, with each topic treated independently. The range of plausible values for discrimination and bias was borrowed from previous research; in particular, Li and Smucker [30] suggest varying the assessor’s discrimination ability and bias in the ranges \(disc\in [0.5,3.0]\) and \(bias\in [-3.0,3.0]\), respectively. Smucker and Jethani [52] and Smucker and Jethani [53] had previously reported estimated average values for discrimination and bias of primary assessors (expert, or experienced) as \(disc=2.3\) and \(bias=0.37\) corresponding to \({{{ {TPR}}}}=0.78\) and \({{{ {FPR}}}}=0.06\); whereas secondary assessors (crowd workers) had less discrimination and were less conservative, with \(disc=1.9\) and \(bias=0.14\) corresponding to \({{{ {TPR}}}}=0.79\) and \({{{ {FPR}}}}=0.14\), respectively. Those estimates were made across 10 topics of the TREC 2005 robust track [52, 53].

In preliminary experiments, we explored parameter settings close to these latter values, seeking to simulate crowd workers. However, we shifted to more conservative settings after we saw the results that arose—with \({{ {FPR}}}=0.15\), we observed results very close to random. That level of variability occurs because there are far more “not relevant” judgments in the attestments than there are “relevant” ones, and flipping even 10%–20% of those “not relevant” outcomes swamps the “relevant” attestments. The net effect is to shuffle the set of systems. Indeed, we felt that the quality of evaluations that arose with \({{ {FPR}}}=0.15\) would be regarded by readers as absurd. Experiments with \({{ {FPR}}}=0.10\) also in some instances showed large differences between the robustness of utility-based metrics and the frailty of recall-based metrics.

As a result, we focus on the narrow range \({{ {FPR}}}\le 0.07\). Even so, there are clear measurement problems with these “expert level” judgments, as will be demonstrated shortly. We explored a range of \({{ {TPR}}}\) values but in many experiments report only \({{ {TPR}}}=0.93\) to allow direct comparison between results and between the behavior of ranked and random flipping of judgments.

Figure 6 shows the run score perturbations that result from the two judgoid generation protocols. To form the two graphs, the process shown in Figures 2 and 3 was applied to the TREC 7 qrels with a discriminating (\(disc=3.0\)) and neutral (\(bias=0.0\)) assessor assumed (that is, an “expert”), with 100 sets of judgoids generated. That resulted in a total of \(50 \times 50 \times 100 = 250{,}000\) individual system–query runs (systems by topics by judgoids). A random sample of \(5{,}000\) (that is, 2%) of those runs was then taken, and judgoid-derived and attestment-derived metric scores for AP and RBP computed and scatter-plotted.

In the left pane, the random qrel flips process generates RBP scores (blue dots) that are visibly consistent between judgoids and attestments, and hence well correlated. In contrast, the AP scores (red dots) are more dispersed and also show a clear pattern of shifting above the dotted mid-line, that is, of being numerically smaller with the perturbed judgments than with the attestments.

With the rank-biased qrel flips approach in the right pane, RBP scores tend to be below the mid-line; judgoid-based run scores tend to be larger than with the attestments, as popular non-relevant documents get flipped to relevant, a consequence of the biased subset selection employed in the \({{ {FPR}}}\) computation. In contrast, AP has many scores closer to the mid-line, but with a large mass of attestment scores that were near zero having much higher corresponding perturbed scores; that is, both of AP and RBP now show a high degree of dispersion.

However, the absolute run scores are in general not of direct interest, as their primary value is in system comparison; the key test is whether systems are affected in a consistent way by any pattern of drift in scores. The next two subsections examine what happens when the perturbed run scores are used in system-to-system comparisons.

In this experiment, we explore how perturbed judgments affect system ordering, in which each system is ranked from 1 to 50 by its average measured score with ties (which are rare) broken at random. Here, and in several of the other experiments discussed below, we report only results for AP and RBP as well-performing and widely used representative recall-based and utility-based metrics, respectively.

For each of 100 sets of judgoids, a system ranking is computed based on the mean metric score for each system across the corresponding set of topics, and then RBO is calculated, comparing that system ordering against the system ordering induced by the attestments. This process leads to 100 RBO scores for each parameter setting, each a measure of the similarity between results for one set of perturbed judgments and the original judgments. These scores are top-weighted, reflecting an interest in being able to identify a relatively small number of top-performing system. We then average the RBO values across the 100 sets of judgoids to obtain a single outcome value for each combination of parameters.

Before considering RBO scores, it is also helpful to look at rank ranges. These are computed out of the same experiment by tabulating, for each rank in each judgoid-induced system ordering, the corresponding system rank in the attestment-induced system ordering. This element of the reverse protocol yields information that helps answer the question, “if a system is at rank x using the judgoids, where might it have been ranked using the underlying attestments?” Figure 7 shows this type of output. A discriminating (\(disc=3.0\)) and neutral (\(bias=0.0\)) assessor is again assumed, the random qrel flips strategy is employed, and rank correspondences for AP are shown on the left and for RBP on the right. Each vertical box-whisker element shows the distribution of attestment rank positions corresponding to one judgoid rank position, with the solid area in each column depicting the middle two quartiles. The orange dots provide a guide that shows the location of the line of perfect consistency.

As is noted in the figure, these settings for disc and bias give a TPR of 0.93 and an FPR of 0.07. With around \(1,\!200\) judged irrelevant documents per query and 70 judged relevant, this corresponds to flipping to “off” around 4 previously positive judgments and flipping to “on” around 85 previously negative judgments. Those error rates intuitively feel quite small; as noted earlier, we would generally be happy with such an accurate assessment from human judges. However, as the raw numbers show, they do mean that the incorrect positive judgments risk swamping the correct ones. Nonetheless, RBP appears to be robust, with a notable consistency apparent in the plot, and as a corroboration, a corresponding mean RBO of \(0.9452607641\); whereas the corresponding RBO value for AP is \(0.6460283557\). That lower correlation is clearly evident in the left plot in Figure 7, with a wide range of attestment-induced ranks corresponding to each judgoid-induced rank, even among systems that score well (the low ranks in the horizontal axis of the plot).

When the same experiment was carried out using the 199 Robust 2004 queries and the corresponding 50 best systems, similar results emerged, with RBO correlations for AP and RBP of \(0.7417443778\) and \(0.9404976222\), respectively. These results are shown in Figure 8, using the same presentation as in Figure 7. This is somewhat surprising, since using a larger set of queries appears to not be helpful in terms of convergence of results; and suggests that the number of queries is not an important factor. That is, for a given error rate, we tentatively hypothesize that the instability in ranks under AP with random qrel flips is innate and is not dampened by increasing the size of the query set. The reliance of AP on normalization by the number of relevant documents for each topic may be a contributing factor here, since each set of judgoids ends up with its own actual \({{ {FPR}}}\) rate, with the process in Figure 3 delivering perturbations in expectation and not at a guaranteed rate that applies to every set of judgoids.

In other experiments with random qrel flips (results not shown here), RR and NDCG display a relatively wide range of variations in ordering similar to or worse than AP, while P@10 is more akin to RBP and is relatively stable.

Figure 9 presents the system rank relationships that arise when the rank-biased qrel flips approach is applied to the TREC 7 documents and queries, again for a smart neutral assessor. Now, both AP and RBP show severe degradation in system ordering, presumably due to the fact that rank-biased perturbation particularly affects top-ranking systems—they get to the top by returning some unpopular yet relevant documents, which are exactly the ones that have a relatively high likelihood of being flipped and deemed “not relevant” in the judgoid sets. In particular, both measures have RBO scores of approximately 0.2; as discussed in Section 5.1, that means that the degree of judgment distortion deployed to obtain Figure 9 would render an experimental outcome essentially meaningless.

The rank-biased qrel flips perturbation method also affects outcomes for the Robust 2004 data and topics (not shown in a figure), but not as gravely, with RBO values of 0.517 and 0.629 for AP and RBP, respectively. Analysis of the TREC 7 runs revealed a high level of diversity and more distinctiveness between runs in terms of documents returned, including in terms of relevant documents returned. However, the Robust 2004 runs had more overlap with each other and hence a lesser degree of vulnerability to the volatility introduced by rank-biased qrel flips. Even so, there was still wide variation in attestment rank for Robust 2004 systems that were top-scoring according to the judgoids, showing that even RBO scores of 0.5 or 0.6 are not necessarily a cause for celebration.

Table 3 presents a broader sweep of results, focusing on mean RBO scores between judgoid-induced system rankings and attestment-induced system rankings. Five effectiveness measures, two different true positive rates, and four different (and smaller) false positive rates are shown for each of four different experimental configurations. As previously noted, RBO is calculated with a persistence of \(\phi =0.90\) to depth 50; with the greatest RBO score for each combination of collection, \({{ {TPR}}}\), and \({{ {FPR}}}\) is highlighted in blue. Figure 10 shows the corresponding outcomes for the TREC 7 dataset. In Figure 10, the \({{ {TPR}}}=0.9\) results for each metric and for the four \({{ {FPR}}}\) values are laid down first via the bars in the solid colors, and then the \({{ {TPR}}}=0.7\) results are laid over via the hatching.

To confirm the validity of these results, we used a paired Student t-test to assess the significance of the differences between the results for RBP and AP. For random qrel flips, across the 24 cases in Table 3 and Figure 10 the computed p value was less than \(10^{-6}\) in all but three instances, in which the difference was not significant. For rank-biased qrel flips, AP is significantly better than RBP in one case but RBP is significantly better than AP in the other 23, always with p similarly close to zero. We thus conclude that RBP does indeed lead to smaller RBO values when rankings induced by perturbed qrels are being compared to the rankings induced by the original attestments.

Given that the results shown in Table 3 and Figure 10 can be regarded as being significant, our overarching observations are as follows:

As a confirmation, we also used Kendall’s \(\tau\) to compare system rankings to provide a non-top weighted perspective. Figure 11 shows the same trend of ranking correlations as observed for the RBO scores, namely, that utility-based metrics are more robust to judgment errors than are recall-based measures. We also observe again that the rank-biased qrel flips results in a faster onset of degradation.

To further explore the contrast between random qrel flips and rank-biased qrel flips, the collective behavior of the 10 top-scoring systems was compared to the remaining lower-scoring systems, noting that when the RBO persistence factor is 0.9 (used in Table 3 and Figure 10) the top 10 systems contribute the majority of the overlap calculation. We compared them to the remainder by building a consensus run for each topic that combined the 50 systems, ordering documents by decreasing meta-AP score. We then computed an RBO score for each topic and for each system relative to that consensus run. Because we are now comparing rankings of documents (rather than rankings of system), a much deeper overlap was sought, and RBO with a persistence factor of \(\phi =0.98\) was employed.

For TREC 7 the average RBO (\(\phi =0.98\)) score across all queries for the top 10 systems was \(0.357928\), with the average over the other 40 systems being much higher, at \(0.521761\). Investigating further, we found that the top systems in TREC 7 return documents that are relevant yet not popular; these documents do not gain a high meta-AP score and thus tend to be favored (Figure 1(b)) for flipping by the rank-biased qrel flips approach. In contrast, the Robust 2004 systems tend to behave similarly to each other, with slightly better RBO scores for the top-ranking systems compared to a consensus run, relative to the non-top systems. That is, the top systems and non-top systems retrieve similar proportions of popular and unpopular documents.

In results not included here, we also carried out experiments using the TREC-9 collection and relevance judgments. While those results showed up further patterns of behavior, in very broad terms they fell between the TREC-7 results and the Robust 2004 in terms of their sensitivity to disruption.

Our various results have several implications. First, they provide justification for the strategy of pooling; it is essential that all documents be considered equally, regardless of how many systems retrieved them. Second, it highlights the value of having a large number of systems contributing to the pool. Third, and most importantly, it suggests that popularity is only approximate as a predictor of assessor error. We have relied on it here and are satisfied that our results are robust—as is illustrated by the strength of confirmation between random and ranked flipping of judgments—but it does imply that the model should be used with awareness of its limitations.

Arguably the most pertinent previous work to that reported in this section was that reported by Bailey et al. [4] and Voorhees [59]. In these papers, the authors investigated the agreement between different sets of relevance judgments. However, the different designs in this prior work mean that there are confounds to a comparison. Where we have been able to examine many millions of sets of synthetic judgments generated under a range of parameters, the prior work was of necessity limited to only one or two sets of additional real judgments on one or two test collections on smaller numbers of systems and queries.

Bailey et al.’s results align well with ours, showing that substantial differences between systems can reverse when some of the judgments change. The methodology used by Voorhees makes direct comparison more difficult, as one of the sets of judgments is not based on pooling and thus many unjudged documents are assumed irrelevant in the score calculation, that is, there is an unknown error bound in the results, and the correlation function that was used (Kendall’s tau) is not top-weighted. The greater volume of data in our work has allowed us to draw more conclusive inferences, with the limited scope of the previous experiments meaning that outcomes were, at least to some extent, tentative.

A potential confound in the experiments reported in the previous subsection is that some disruptions to system orderings are uninteresting. When two systems have similar scores, or their differences in score are not statistically significant, changes in ordering may not be relevant to the hypotheses being tested. While we have noted consistent effects that vary monotonically in the degradation parameters, we nevertheless need to be alert to the possibility that what we have measured is the result of chance outcomes. To build confidence in the experiments reported in the previous subsection, we now restrict our attention to system differences that have been determined to be statistically significant. This is, after all, the core of a great many offline retrieval experiments: use of statistical significance tests to establish whether a new method can reasonably be claimed to be superior to alternatives. What we would hope to find is that when a pair of systems are significantly different when measured on a set of judgoids (which are all that is available in a typical experiment), then that significance would also be present in an evaluation based on the original attestments from which the judgoids were derived.

Consider the results shown in the two plots in Figure 12, which, as is the case throughout the rest of this subsection, concern the measures AP, NDCG, P@10, and RBP. The dataset is TREC 7, with judgoids created via the random qrel flips protocol. From among the \(125{,}500\) individual comparisons (that is, covering all pairs from among 50 systems, with 100 sets of judgoids applied to each pair), we extracted all “System A, System B, judgoids” triples where the two systems were found to be significantly different (according to those judgoids) with a p value in the range \(p\in [0.005, 0.015]\). Each of these outcomes is thus close to the “1 in 100” mistake level that would be regarded as being strong evidence of superiority. This filtering step is applied solely for the purposes of selecting for subsequent analysis a group of system pairs in which the relationship is highly significant and the p values are of comparable magnitude, and does not affect the “even more highly significant” outcomes derived when \(p\lt 0.005\). The number of such outcomes for each of the four effectiveness metrics is shown in the legend; for example, on the left-hand side the number for RBP is \(11{,}053\).

We then took the matching “System A, System B” attestment-based p-values (as a multi-set) corresponding to the selected triples and examined their distribution. The legend shows statistics of those four sets of corresponding p-values as computed via the attestments; on the left, for RBP it is \(0.02\pm 0.06\), for P@10 it is \(0.04\pm 0.11\), for AP it is \(0.32\pm 0.39\), and for NDCG it is \(0.30\pm 0.36\). The distribution of those attestment-based p-values is then shown by the inferred density curves.⁴

If system A outperforms system B on the judgoids, then it is still possible for B to outperform A on the attestments, and indeed for B to be so much better than A that the results are statistically significant. To capture this possibility, in Figure 12, we report what we call mapped p-values:

Thus, as a further subtlety in this experiment, p-values that are greater than 0.5 correspond to reversed system mean scores. If the perturbed observation was, say, that system A outperformed system B with \(p=0.01\), then an attestment observation of \(p=0.99\) would mean that B outperforms A with \(p=0.01\)—a strong contradiction of the finding that was inferred using the perturbed judgments.

As can be seen, in Figure 12(a), which shows results with a very low TPR selected as an extreme case, the RBP p-values remain reasonably tightly grouped at the left and show that it is behaving consistently: If the greatly perturbed judgoids suggest \(p\approx 0.01\) for some A-versus-B system comparison, then the corresponding attestments also heavily favor small p-values for A-versus-B. However, the matched p-values for AP and NDCG are, more or less, spread right across the whole range. That is, in this context AP and NDCG have failed: The observed results in the presence of error bear little or no correspondence to the results that would have been obtained if the attestments had been available. This implies that experimental outcomes assessed with AP and NDCG—presuming that judgment errors of the supposed magnitude were indeed being made—would be consistently incorrect, even though each of the comparisons using the judgoids yielded statistical significance and high confidence.

Unfortunately, the same observations also apply to Figure 12(b), where the error rates are lower and are at the “expert assessor” levels reported by previous authors. The distribution for AP and NDCG is not quite as poor, but it remains the case that, for most sets of judgoids and with the attestments taken as “truth,” what are apparently successful findings using the judgoids are at best unsubstantiated and at worst are false.

This finding has serious implications for use of AP and NDCG in practice. While RBP attestment results are centered on \(p=0.01\), in alignment with the judgoid results, those for AP and NDCG are centered on \(p=0.07\) and \(p=0.15\), respectively. That is, with even these low error rates the results show that a 1-in-100 chance of significance being a false positive corresponds to an underlying likelihood that the results are in fact not significant. Even a small volume of errors, well below that observed even among expert assessors let alone crowd workers, can lead to experimental outcomes that are simply wrong.

Experiments with random qrel flips on Robust 2004 (199) were very similar to those shown for TREC 7. A more interesting contrast is with the results for Robust 2004 (199) and rank-biased qrel flips perturbation, shown in Figure 13. (Note that in this figure the vertical scale is approximately one-tenth that of the previous figure.) This data reveals failure for all four effectiveness measures. While RBP and P@10 are slightly better than AP and NDCG, in that they offer a higher peak on the left, the results show that the presence of rank-influenced judgment errors of the type suggested by Webber et al. [63] has the potential to render meaningless any assessment of statistical significance when comparing systems. The plot in Figure 13(b), with a reduced error rate, is only marginally better; judgoid significance centered on \(p=0.01\) corresponds to attestment significance of \(p=0.14\), \(p=0.22,\) and \(p=0.34\) for P@10, RBP, and NDCG, respectively, even when the error rate is low.

A variant on the above results is shown in Figure 14. These two plots show the cumulative distribution of all attestment p-values that correspond (as a multi-set) to perturbed p-values using judgoids that are\({}\le 0.01\). In other words, these are the “true” p-values for all of the “plausible experiment in the presence of judgment errors” observed p-values in which there was high statistical confidence in the measured outcome. The left plot shows results with the random qrel flips approach, the right shows results with the rank-biased qrel flips perturbation mechanism.

In these figures, the intercept with the right vertical axis (which is at \(p=0.5\)) shows the fraction of system pairs that have the same ordering under both attestments and judgoids. For example, in Figure 14(b), for RBP, NDCG, and AP the proportions of system orderings that are consistent between judgoids and attestments are around 82%, 74%, and 67%, respectively. In Figure 14(a), effectively all of the p-values for attestments for RBP are well below 0.05 and over 90% are below 0.01, and the curve as a whole is in keeping with the expected false negative rate for a significance test. For the other effectiveness measures, the results are less consistent; NDCG is particularly poor, with less than 70% agreement at 0.05. Nor do the results improve much for lower error rates. It is clear that on this collection the recall-based measures are unacceptably vulnerable to errors in the judgments. However, again echoing the earlier results, none of the metrics has done especially well on the Robust 2004 (199) collection, and nor do any of them do especially well in the face of rank-biased qrel flips perturbations. If indeed judgment errors are made according to the patterns modeled by Webber et al. [63], we may well need to find judges who are more expert than experts.

Our work here has made extensive use of TREC-based resources, primarily qrels and system runs in which (for the most part) each system executed the same single query in response to each topic. That means that the current judgment pools risk being uni-dimensional with respect to query variations [40]; broadening them to cover multiple queries for each underlying topic has been shown to be at least as expensive in terms of judgments as is broadening the set of systems involved [36]. Investigation of the sensitivity of metric scores, system orderings, and system-versus-system comparisons in the presence of query variations is an important next step.

Our methodology and experiments are designed for binary judgments; we do not have error models for multi-level judgments and thus cannot simulate them in a principled way. However, assessor disagreement and error is not limited to binary judgments, and the evaluation measures have similar structures. It seems highly likely that the same kinds of issues will arise, but the sensitivity of the results to error, and indeed error rates, may be quite different in the multi-level case. Exploration of this issue would require perturbation models that are very different to those used here.

Note that our experiments are not designed to examine the relative validity of different methods of collecting judgments or the robustness of different kinds of judge. We used the same parameter settings for both methods of generating judgoids to ensure that the results were comparable, with the aim of showing that the same issues arise across different judgment methodologies.

Other researchers have considered the measurement reliability of even very shallow judgments when applied to a large number of topics [1, 14, 34]; this is another area where the reverse protocol that we have described might be an insightful way of exploring the dual areas of metric vulnerability and metric sensitivity in the face of judgment errors. However, it is already clear that current experiments have levels of vulnerability and sensitivity that risk undermining experimental outcomes, and which should be addressed if future research is to be robust.