Interface Design for Crowdsourcing Hierarchical Multi-Label Text Annotations

To both build and evaluate machine learning systems, researchers often rely on human-labeled datasets [13, 41, 54]. Gathering this labeled data efficiently and at high quality is a well-studied problem when labels are binary [25, 34, 55] or a flat list of choices [14, 33, 47], but labels can often be grouped into other structures as well, such as species in a taxonomy [50].

Concept hierarchies (or taxonomies; ontologies) are used in many applications ([10, 17, 18]) to describe concepts at a flexible granularity, and generally serve to help organize and structure both language and thought around a topic. In certain situations, they may become a target for data labeling itself, where multiple hierarchically-structured class labels can be chosen for a given instance, a setting known as hierarchical multi-label annotation [7, 56]. Given their usefulness in organizing thought, one might expect that leveraging the hierarchy during annotation may yield higher quality and efficiency. However, there are many design choices to consider: Does interface complexity increase cognitive load ([19, 40])? Will false negatives in an upper level of the hierarchy end up amplifying errors into annotations on a lower level [7]? Will presenting one part of the hierarchy while hiding the rest create a lack of context that leads to misinterpreted label definitions?

In this paper, we study how to incorporate the concept hierarchy into labeling schemes for crowdsource data annotation platforms such as Amazon Mechanical Turk (AMT). We focus on a difficult annotation task, assigning vaccine concerns ([46]) to text passages taken from anti-vaccination websites. The hierarchy of vaccine concerns includes labels such as “Health risks” and “Issues with research”. Small, purpose-built taxonomies are common in the domain of misinformation research [1, 10, 24, 42]. In the setting of difficult annotation tasks, we show that labeling schemes incorpor- ating hierarchies can help annotators perform better against ground-truth labels.

We investigate two separate design choices when annotating a single passage: (1) how to format the hierarchical labels when shown on the interface, and (2) the pass-logic that decides how to coordinate multiple workers towards labeling that passage. We compare two formats for presenting the hierarchy to annotators (see Figure 3):

For pass-logic, we look at three options (see Figure 1):

We compare all valid combinations of these formats and pass-logic options under a fixed-budget setting, which provides practical insights for research and engineering teams interested in data collection of hierarchical multi-label tasks. For multi-pass logic, we consider both randomly partitioning labels into smaller subsets or utilizing the groupings given to us by the hierarchy. Our results point to a few statistically significant factors:

Our results lead us to believe that difficult, high-subjectivity labeling tasks warrant new recommendations separate from crowdsource design guidelines in previous work ([7, 21]). We recommend considering incorporating hierarchies into the labeling process, and show a few options for how to do so. This is especially true if optimizing for individual worker performance, while choice of labeling scheme plays less of a role if using aggregation methods across several copies of annotations.

The reliance of supervised ML algorithms on labeled data has led to a great wealth of knowledge regarding efficient data labeling at large scale. Huge datasets have been constructed requiring immense human labeling time across many media. Among them are image and video datasets generally containing thousands of classes such as ImageNet (14M images) [13] and OpenImages (9M images) [30], but even with fewer classes, such as CelebFaces labeling 40 facial attributes (200k images) [34]. Also audio datasets, typically with hundreds of classes, for instance AudioSet (2M clips) [18], Free Music Archive (100k clips) [12], and OpenMIC-2018 (20k clips) [21] Lots of work is focused on allowing this scale of data collection while maintaining high quality [8, 14, 29, 51] or protecting crowdsource workers [4, 23].

Often, this labeling is done on tasks with low ambiguity or subjectivity, and minimal required training – which makes them suitable for large scale collection. For example, in ImageNet [13], labels are the names of well-known objects such as “ambulance”, “folding chair” or “snail.” Even in more difficult audio-annotation tasks such as labeling noise categories in a busy city ([7]), the labels (“jackhammer”, “car horn”) have strong, objective definitions.

Given the clarity on such label definitions, previous studies on user interface design for crowdsource annotation have recommended increasing annotation throughput, or the rate at which labels are collected from the annotation platform [4, 16, 37]. Throughput can be very quick for some tasks (minutes for hundreds to thousands of annotations), while other tasks may be much slower. Prior work found that single-pass methods have up to 9 times higher throughput if annotations are required to be fully labeled (assigning a value for every label) rather than sparse [7]. However, work in psychology has long known that there is a tradeoff between speed and accuracy for any information processing task a human performs [53]. Other HCI work also studies this tradeoff [35, 57]. This suggests optimizing for throughput could be harmful to annotation quality, particularly if the task is difficult.

The cognitive load theory [48] suggests that tasks with high cognitive load (the amount of mental effort) can induce errors and mistakes at higher frequency than tasks with lower cognitive load. Work on user interfaces which require some level of accuracy often tries to minimize unnecessary cognitive load [19, 40, 51]. Similarly, work in crowdsourcing recommends to chunk difficult tasks into smaller units of work [28]. Some work has shown that crowdsource platforms have great potential for rapidly collecting measurements in user studies [27]. Other work examines how long annotators remain on tasks, and characterizes differences between those that annotate few examples versus those that annotate many [15].

Recent efforts have also moved towards datasets for high-impact social issues such as: misinformation [10, 46], which attempts to classify common concerns regarding issues such as climate change or vaccines; fact-checking [49], which labels whether claims are verified by trusted sources; and claim review [2, 3], which determines if claims are worth fact-checking. Such labels inherently lend themselves to be a more difficult annotation task, given the subjective label definitions and necessary processing to parse written rationales or arguments in text.

In data labeling, it is common to collect multiple copies of annotations and aggregate them using a majority vote [5, 54]. Some work studies how to perform aggregation more effectively [52]. This is said to reduce the impact of low-quality annotations during collection. Some old work in aggregation methods such as EM uses weightings from estimates of worker skill [11], while other work incorporates question difficulty through parametric approaches [26] or non-parametric approaches [44].

However, recent trends in NLP have began questioning aggregation, arguing that subjective labels should not be aggregated if multiple opinions are valid. Rather, this line of work ([38, 58]) suggests predicting the distribution of human opinions, rather than the majority vote. One implication that follows is that individual annotator performance becomes more important, since one cannot aggregate away labeling error using a simple majority vote.

Labels are not always in the form of lists. There has been a large amount of work on labeling hierarchical multi-label annotations [7, 21, 45, 56], where the task is to select any relevant option from labels in a hierarchical structure. While most work employs a small group of experts to build the concept hierarchies before it gets labeled by workers, some research attempts to build these hierarchies through crowdsourcing methods [6, 9].

In considering the performance of crowdsource workers, a lot of effort has been spent to introduce gamification of the labeling task [22, 31, 36, 39, 43], but we note that this requires significant overhead efforts to build the games, which may not be feasible when data collection is time-sensitive.

We collected three copies of annotations for each passage, for each of the 6 labeling schemes (§3.4.3) through AMT. In this section we compare the labeling schemes against each other on performance and examine the reasons for why performance varied across labeling schemes.

There is a large body of work which has studied crowdsource annotations. How to make it scalable, how to keep quality high, and how to increase throughput. There is also a deep wealth of knowledge and best practices regarding user interface design. Much of this work, however, has been done for labeling tasks that are low-subjectivity, have clearly defined label definitions, and low cognitive load.

We studied six candidate labeling schemes in annotation of a taxonomy of vaccine concerns. Our motivation was to study whether the hierarchy itself could aid workers as they perform the annotation task, and how to design the annotation task for high-quality labels under a fixed budget. We found that integrating the hierarchy into the labeling scheme helps with improving annotation quality, whether explicitly in the interface or through logical passes made behind the scenes. Our analysis showed that a hierarchical multi-pass multi-label scheme performs best when considering individual worker performance. We believe individual worker performance to carry more importance when the tasks are inherently subjective, since a growing body of work is interested in predicting label distributions. Much like in [7] and [45], we find that workers assign more labels per passage on average when they are in multi-pass schemes versus single-pass ones.

However, if the priority is to collect high confidence labels rather than distributions of human opinion, we found that employing the single-pass hierarchical multi-label along with a majority vote achieves highest performance. Unlike [7] and [21], we don’t see a drop-off in performance when using single-pass labeling schemes. Although we don’t conduct a qualitative study, we did not receive notably different amounts of complaints from workers in any of the labeling schemes. Largely, complaints were not about the interface designs at all, but rather about being allowed to annotate more data after workers finished the batch or were banned for failing attention checks. When using majority vote, the choice of labeling scheme matters less than it does for individual worker performance. The ease of setup with single-pass options should not be undervalued either. Such labeling schemes are already supported natively in AMT’s requester user interface, making it a strong option for smaller projects with a necessary quick turnaround.

Overall, we find that introducing the hierarchy helps almost universally across our experiments. In comparisons between partitioning the labels into groups randomly versus using the hierarchies structure, we find that using the hierarchy dominates across all performance metrics. Exposing the hierarchy explicitly helped performance on single-pass schemes by increasing worker performance on particularly difficult passages. We used a taxonomy specifically designed to achieve high agreement among crowdsource workers. While some previous work has indicated that integrating hierarchies into the data labeling process may harm performance ([51]), we find that it boosts it. The contexts here are different: our task is higher difficulty and therefore the hierarchy may aid in completing the task, but using a taxonomy specifically designed for high agreement among crowdsource annotations may also indicate that for these methods to work you may need a well-designed hierarchy.

We investigate various labeling schemes for crowdsourcing text annotation of difficult, high-subjectivity tasks and measure impact on worker performance against ground-truth labels. We find that integrating hierarchies into the labeling scheme helps with boosting performance.

Through analysis, we explore three potential indirect causes for improvement against ground-truth labels: (1) They group similar concepts together, improving F1 scores to 0.50 from 0.34 as compared to random groupings. (2) They allow relative increases in performance on difficult passages, leading to an increase in as much as + 0.40 on F1 score on high difficulty examples. (3) They boost the true positive frequency, thereby increasing precision of workers without detriment to recall. We recommend considering incorporating hierarchies into the labeling process if optimizing for individual worker performance, while using a majority vote setting if solely optimizing for F1 score (achieving 0.70 with single-pass hierarchical multi-label).

We thank Pardis Emami-Naeini and anonymous reviewers for feedback. This work was supported by NSF award IIS-2211526 and an award from Google.

Figure 1: “Three diagrams are shown describing single-pass, multi-pass, and hierarchical multi-pass routing logic. For single-pass, all set of labels are given to one worker. For multi-pass, the labels are partitioned into groups (1,2,...) and given to separate workers (A,B,...). For hierarchical multi-label, the top level labels are given to one worker, who’s annotations determine whether or not the child labels will be given as a new group to annotators downstream. This example shows the case where the first worker labels TFFTF, and downstream there are two tasks set up for new workers to label the children of label 1 and label 4, respectively.” Figure 2: “The figure shows an example passage that reads: ’The minister of fear (the CDC) was working overtime peddling doom and gloom, knowing that frightened people do not make rational decisions — nothing sells vaccines like panic.’” Figure 3: “Four diagrams are shown side by side. In each diagram there are a set of checkboxes or radio buttons indicating how the labels will be presented to the user. Binary label (the leftmost diagram) contains a simple question ’1.1?’ and below it a radio button reading ’yes’ or ’no’. Multi-label contains a simple list of checkboxes labeled ’1.1, 1.2,...’. Hierarchical multi-label v1 contains staggered checkboxes where the leftmost checkboxes read ’1, 2’ and the boxes immediately under these are tucked under them, reading ’1.1,...’ for the parent label ’1’, and ’2.1,...’ for the parent label ’2’. Hierarchical multi-label v2 contains both the radio button setup from the leftmost diagram, as well as the checkboxes from multi-label underneath them.” Figure 4: “A table is shown with column headers reading ’interface design’, ’greater than or equal to 1 tutorial Q’, ’greater than or equal to full tutorial’, ’greater than or equal to took exam’, and ’greater than or equal to 1 datapoint’. The table shows values for all 6 labeling schemes.” Figure 5: “A table is shown with values for precision, recall, and F1 score. These metrics are given for each of the 6 labeling schemes, and a random baseline is shown at the bottom. Hrchl-pass multi has bold font at the f1 score indicating it is the highest value in that column: 0.56.” Figure 6: “A two-part table is shown with column headers ’model factor’, ’estimate’, ’95\(\%\) CI’, ’SE’, and ’p-value’. The first part of the table (top) has a subheading that reads ’labeling_scheme (baseline=multi-pass random multi-label)’, and the second part of the table (bottom) has a subheading that reads ’additional numerical factors’. The first part includes 5 of the 6 labeling schemes, while the bottom includes new factors such as ’time_started’, ’percentage_easy’, and ’true_positive_freq’. Some values in the table are denoted ’*’ which represents a p-value below 0.001.” Figure 7: “The figure shows an example passage that reads: ’Pregnant women given vaccine have babies with more health problems’” Figure 8: “A bar chart is shown giving the value for worker F1 score on the X axis ranging from 0 to 1, and the difficulty on the Y axis. The labels, from top to bottom, on the Y axis read ’immediate author agreement’, ’author agreement after providing rationale’, and ’author agreement after discussion’.” Figure 9: “A scatter plot shows orange and blue dots generally following a linearly positive relationship. The orange dots are labeled ’hrchl-pass’ and the blue dots ’multi-pass’. On the X axis: ’Frequency of True Positives shown to workers’. On the Y axis: Worker F1. Each blue dot is paired with an orange dot through an arrow which is drawn between them pointing towards the orange dot.” Figure 10: “A line plot is shown with dashed lines between three dots. The dots are lined up at tickmarks labeled ’sensitive’, ’majority’, and ’unanimous’. This is repeated for all 6 labeling schemes, leaving 6 connected dotted lines all in different colors. Every one of the lines follows an inverted V shape, with their highest point being over the ’majority’ tick mark.”

We ignore all data collected for tutorial paragraphs, entrance exam, and quality checks when assessing workers. In the future we may assess how to leverage any information about worker performance on the tutorial towards improving data collection quality.

We remove all data collected from a worker if they failed the entrance exam or a quality check. Such workers were also banned in live time during data collection to avoid spending any further of our research budget on their data collection. To be clear, when setting the budgets for each labeling scheme, we do not factor in additional cost due to such workers. Rather, we re-annotate all the passages which they had been paid for. For certain labeling schemes, consistency between level 1 and level 2 is forced by the annotation platform. For example, in single-pass hrchl-label, workers cannot submit a positive value for a level 2 label without also submitting a positive value for its parent node in the taxonomy. This is achieved through some javascript in the annotation platform and is visually confirmed to the workers while labeling. However, for other labeling schemes such as multi-pass multi-label, the separation of L1 and L2 labels into separate screens leads to collecting data which is not necessarily consistent. Since any real-world use of this latter type of labeling scheme would include corrections for consistency, we correct for this during post-processing. This ensures fair comparison between the labeling schemes such that we don’t disadvantage multi-pass schemes. To further ensure comparison is consistent, we make sure to look at performance on level 2 labels on its own during our analysis.

We measure annotator performance using the F1 score, a harmonic mean of precision and recall. This score is commonly used in machine learning tasks and is a well understood and cared about metric in research communities such as natural language processing (NLP). Examining this metric gives more utility for NLP researchers, since the positively labeled examples will be the most informative during model training. One can achieve very high accuracy simply by marking all passages as negative. Since each worker’s performance can be evaluated across each label in the taxonomy individually, we must use some aggregation technique when presenting our scores. We employ a macro-level average of F1, which is computed by first finding the F1 score on every taxonomy label, and then averaging across all these labels. In the case where we give an F1 score for each worker, the macro-averaging process happens in parallel for each worker. In the case where we give a single F1 score for all workers, we take all the annotations and treat them as if a single worker had filled out all annotations and we then follow the macro-averaging process. This method of averaging is preferable in our setting as opposed to a samples-based average which would compute F1 over a single passage, and then aggregate across passages. Semantically, we use this method since we also care about worker performance on each individual label and can thereby inspect these values. Inspecting worker performance on each passage is less important to us, since the set of passages used are meant to be a sample of the types of passages annotated in any application of our work.

We train each worker before they complete any real annotations. Workers get paid for this training in order to ensure a fair and non-predatory ([4]) employment. This leads to additional costs to those interested in paying for the data labeling, since some workers can go through some or even all of the training, yet submit no actual annotations. Such workers never become “productive.” Below we report the task uptake, that is the percentage of workers who became productive, out of all the workers which completed at least one tutorial example. We also report an inefficiency number, which is the percentage of extraneous annotations collected from training and attention checks (discussed in detail in §3.2). The inefficiency number compares the extraneous annotations to the total number of “useful” annotations collected. For multi-pass schemes, the workers were allowed to complete one partition (group) of the labels and then begin a new partition. This occurred seamlessly, whereas there was a multi-day delay for the hrchl-pass multi scheme, meaning that there were less return workers for this task and thus decreasing the total task uptake.⁶

Task inefficiency (which is directly proportional to additional incurred cost) is lowest for hrchl-pass multi. The task uptake is the highest for multi-pass random multi at \(80\%\), which contrasts the performance trends shown in §4.1. One potential explanation for this could be that annotators underestimate the difficulty of the task when they’re presented randomly partitioned labels.

To find the confidence intervals included in any results of the paper, we use bootstrap confidence intervals. This is done directly on the raw data we collected, where a datapoint is a single submission by a worker. That is, for single-pass schemes the datapoint will include all labels from the taxonomy, but for multi-pass schemes the datapoint will only include a subset of the labels. We draw N=10,000 samples with replacement from the original data, then find the performance metric using the process outlined in the paper (including all preprocessing). We then use these 10,000 measurements of each performance metric to compute 99% confidence intervals.

We do this sampling on the subset of annotations which ultimately contribute to the performance metric, rather than all the annotations we receive. This ensures we are not sampling (for example) tutorial annotations which we will ignore in the final calculations anyway.

Suppose the team wants to collect approximately 10, 000 fully labeled passages in order to provide high-quality training data. How much will it cost them to use each labeling scheme?

If the team assumes the longest part of labeling is due to reading, and want to guarantee a strong hourly wage for workers, then they can fix the reward at $0.10 per passage (this is a toy example, but loosely this is in line with paying above minimum wage in the United States). Then, the cost of data collection has to do with how many times they ask workers to read a single passage, just to collect a full set of labels. If chunking the question into smaller subsets (multi-pass), the cost will be greater. The extreme is when you ask a new worker to read the passage once for every label in the taxonomy. See Figure 14 for a breakdown of the cost to carry out this annotation.

We see that binary labeling schemes are prohibitively expensive. For this reason we do not include binary labeling schemes in further comparisons.

Overall, single-pass options are the cheapest (both below $2, 000), including the cost of training the workers (paying them for completing the tutorial examples). Multi-pass options have a higher range, $5, 000 − 8, 000. Hrchl-pass multi balances cost (under $3, 000) but still uses multiple workers to complete one passage’s annotations.

Instant agreement (“easy”) are the passage-label pairs for which the entire lab (3 authors plus 3 more students) gave the same value while individually annotating. During this step, the team looked up any terms we were unfamiliar or unclear about, just like the AMT workers are instructed to do.

Agreement after writing rationales (“medium”): highlight specific parts of the passage and justify why a label should or shouldn’t apply. We only went through this process for passages where we had disagreed in the first step. During this process, we include any level 1 label if there is disagreement within any of its children, even if all team members agreed on the level 1 label.

For the remaining disagreement, we had a brief (1-2 minute) discussion for every passage-label pair, reading everyone’s given rationale to see whether one of us could convince the others. This process included both reading the rationales written in 1 as well as generating new rationales (in discussion). The passage-label pairs agreed upon in this stage are referred to as “hard”.

See Figure 15 for a few plots examining the worker performance metrics as each difficulty category varies.

The remaining passages are marked using each of our individual votes, and placed aside (“no agreement”). We consider these passages to be too subjective to give a gold label for, and don’t evaluate workers on them since any label could be valid so long as they have a strong justification. In future work we may consider looking at the justification/rationale of an AMT worker to assess their performance on highly subjective passage-label pairs.

We make note of which category each passage-label pair was resolved in, such that we can perform an analysis into how the “difficulty’’ of passages affect labeling performance. We recognize that this is not necessarily a direct measurement of how difficult it is to label a passage, but we make the assumption that any passage that requires increasingly more thought or discussion to reach consensus will imply this passage is more difficult.

See Tables in Figure 16 for each labeling scheme detailing the performance broken down by each individual label of the taxonomy. Final scores are computed for each metric by taking averages across these taxonomy labels.