Longitudinal Loyalty: Understanding The Barriers To Running Longitudinal Studies On Crowdsourcing Platforms

The research community has focused from a longitudinal perspective, for instance, on (mis)information assessment. Roitero et al. [68, 69] run a truthfulness labeling task repeated four times at a distance of one month each inviting both new and previously participating workers. They observe that returning workers spend more time on the task as compared to fresh workers who have not done the task before. Fan et al. [23] repeated the same crowdsourcing task multiple times inviting the same group of participating workers each day for 20 days observing a sharp decline in return rates over time. Mensio et al. [49] proposed a tool for the longitudinal assessment of the misinformation shared by Twitter accounts.

Longitudinal studies often address health-related issues and challenges. Strickland and Stoops [75] conducted a study on alcohol use, using a weekly survey over 18 weeks. The study involved an initial task that took 21 minutes to complete, followed by regular 2-minute follow-up tasks. High response rates (64.1%-86.8%) were observed across the 18 weeks. Active participation was incentivized through entry into a raffle for one of five $50 bonuses if participants completed 14 or more weekly surveys.

Mishra and Carleton [50] described a study aimed at gathering data on gambling-related behaviors, tendencies, and traits. They conducted three crowdsourcing experiments and a fourth two-wave longitudinal study, which included 13.5% of Study 1 participants and 14.8% of Study 2 participants. This longitudinal study demonstrated acceptable test-retest reliability for the identified problem. Similarly, Brooks and Clark [9] conducted a longitudinal study involving 636 young adults to investigate the gambling-related issue of loot boxes in video games.

Strickland and Stoops [76] provided an overview of using Amazon Mechanical Turk to conduct longitudinal studies for addiction science. They show a fourfold increase in the number of papers utilizing this platform for participant recruitment from 2014 to 2017. Goodwin et al. [30], on the other hand, examined the potential of Reddit as a recruitment strategy for addiction science research, arguing that it could be useful for conducting longitudinal follow-up surveys.

Ogata et al. [55] explored the relationship between domestic pets and their owners during the COVID-19 pandemic through a four-staged longitudinal study involving 4,237 workers. In a related context, Dayton et al. [17] investigated testing hesitancy and disclosure stigma in a four-wave study with 355 workers, while Dang et al. [16] studied COVID-19’s progression characteristics and recovery patterns by collecting audio samples from 212 individuals. Mun et al. [52] conducted a two-year longitudinal study on 1,453 adults with chronic pain, surveying them three times to explore pain severity, interference, emotional distress, and opioid misuse during the pandemic. Additionally, Mun et al. [51] investigated the impact of insomnia severity and evening chronotype on chronic pain in 884 adults over 21 months. They found that insomnia may be a stronger predictor of changes in pain and emotional distress.

The literature review by Cho et al. [14] examined crowdsourcing-based approaches in ophthalmology, analyzing 17 longitudinal studies. Schober et al. [72] investigates pollen allergies through a longitudinal study, analyzing approximately 25,000 crowdsourced search queries from citizens spanning 2017 to 2020. Rajamani et al. [67] used a longitudinal crowdsourcing approach to gather ideas and feedback for enhancing electronic health record systems, collecting 294 responses between 2019 and 2022.

Other researchers address human-related aspects while employing longitudinal-based crowdsourcing approaches. Daly and Nataraajan [15] conducted three studies. The first focused on a two-month re-response rate among a US Amazon Mechanical Turk sample (n = 752; 75%). The second study (n = 373) explored four- and eight-month re-response rates among US immigrants (56% and 38%, respectively). The third study examined a thirteen-month re-response rate (47%), all involving a 23-minute task.

Qiu et al. [66] explored human memorability in the context of information retrieval on the web in a longitudinal study spanning 2 sessions across 7 days with at least a 3-day gap between the two sessions. The authors recruited participants from Amazon Mechanical Turk, and measured knowledge gain and long-term memorability of participants in their study. Tolmeijer et al. [77] investigated trust development in a house recommendation system through a Prolific study spanning three sessions within a week. Initially, 255 workers participated, with 83% returning for the second session two days later. Of those, 96% completed the third session, resulting in 203 participants who finished all three sessions, representing a nearly 80% retention rate throughout the study. Li et al. [46] conduct a large-scale longitudinal study about recruitment and retention in remote research. They recruit 10,000 workers across two phases, gathering 12 weeks of daily surveys and passive smartphone data, resulting in 330,000 days (equivalent to 900 years) of observation.

Wang et al. [80] introduced a two-week game with a purpose. Through longitudinal studies, they examine individuals’ experiences with hedonic and social factors in early stages and expand to include hedonic, social, and usability-related factors in later stages. Leung et al. [45] surveyed 1,000 Amazon Mechanical Turk workers to uncover factors influencing continued participation. Their findings highlight two main triggers: external regulation, such as monetary rewards, and workers’ intrinsic motivation.

Grant et al. [31] explored fairness in crowdsourcing through two theoretical lenses: organizational justice and institutional logic. They conduct a longitudinal netnographic study to understand workers’ perceptions of fairness.

Aljohani and Jones [4] presented initial findings from recruiting qualified yet anonymous workers for hacking experiments involving defensive cyber deception. These experiments are part of a longitudinal study examining malicious cybersecurity experiments on crowdsourcing platforms [3].

Gurung et al. [32] designed a crowdsourcing platform for a longitudinal study analyzing incorrect answers from 2015-2020 academic years across two mathematics courses, aiming to understand how to enhance student learning through remediation.

Sometimes, the specific (micro-task) commercial crowdsourcing platform chosen can hamper the overall worker experience. Peer et al. [60] showed that Amazon Mechanical Turk shows a lower population replenishment rate and tends to have more dishonest workers compared to platforms like Prolific. In a subsequent study, Peer et al. [61] highlighted Prolific’s data quality across various measures relevant to behavioral research. Given the relevance of longitudinal studies to behavioral research [50, 75, 76], platform choice becomes a crucial consideration.

Hata et al. [35] analyzed longitudinal crowdsourcing platform data and found that work quality remains stable over time for the same worker, suggesting that long-term work quality can be predicted after the first five tasks. Additionally, Huang et al. [40] explored the motivations behind continued participation of crowd workers in crowd logistics platforms, confirming the importance of monetary incentives as well as workers’ trust and cooperation.

Retention rates of workers vary significantly across longitudinal studies and decrease as time passes [10, 37, 44, 52, 73], starting from the 80% obtained by Shapiro et al. [73] after a week to the 56% over an year obtained by Mun et al. [52].

Various studies used different reward schemes and incentives to increase retention rates, with strategies predominantly revolving around payment schemes. A common approach involves incentivizing worker retention through supplementary payments.

Difallah et al. [19] showed that offering a bonus to achieve a milestone is the most effective method for retaining workers up to a predefined milestone within a continuous series of tasks with no interruptions. Auer et al. [6] compared traditional work to crowd work in longitudinal studies regarding performance payment effects. They found no difference in performance but emphasized the importance of ethically rewarding workers due to their limited bargaining power. Pay significantly affects attrition (i.e., single task abandonment) but not retention in the second wave of longitudinal studies.

Benbunan-Fich [8] investigated the question of whether workers who quit a study before its completion should receive monetary compensation. They propose that determining an appropriate partial payment, especially for longitudinal studies, involves complex considerations beyond simple monetary compensation.

To interpret our results correctly, it should be noted that some survey questions required multiple responses based on workers’ past experiences with longitudinal studies, while others required only a single response, as described in Section 5.2.

Most questions in the P1 part require answers for each past experience, while questions in the P2 part and one question from the P1 part require only a single answer. Recruiting 300 workers, the maximum number of answers in the former case is 900 (assuming three experiences per worker). In the latter case, the maximum is 300. The results (Section 6.1.1) show that the number of reported experiences is 547.

In result analysis, we often break down results by the platform used to recruit workers who answered the survey. For instance, a worker recruited from Amazon Mechanical Turk but reporting on a longitudinal study that took place on Prolific would be included in the Amazon Mechanical Turk breakdown.

We use bar charts for closed-ended multiple-choice questions and univariate distribution charts for numerical questions in our quantitative analysis. Results are broken down by crowdsourcing platform to highlight differences visually. A color scheme (blue for Amazon Mechanical Turk, orange for Prolific, and green for Toloka) is introduced in the legend of the first figure and consistently applied in subsequent figures to prevent repetition and information overload.

In our bar charts, the x-axis shows available answers, and the second row shows their relative frequencies across platforms. The y-axis represents answer frequencies, with absolute frequencies displayed above each bar. Total absolute frequencies equal 547 or 300 based on question requirements, denoted with \(E\) or \(W\), respectively. These values are shown in the chart’s lower left corner. If a question allows for providing non-mutually-exclusive answers, the top chart is marked with \(A\). Additionally, total answers and experiences/workers are reported in the lower left corner of the chart.

In univariate distribution charts, the y-axis represents the probability density function for three continuous random variables, representing the answers provided by workers across each considered crowdsourcing platform. The x-axis ranges from the minimum value to a cutoff, filtering out outliers. Dashed lines indicate mean values for each platform, using the established color scheme. Total data used is reported in the lower left corner, marked with a corresponding letter. In some cases, outliers are filtered out, noted by an additional label beneath the data count.

We used a conventional qualitative content analysis approach [39] to analyze open-ended responses. This inductive method describes phenomena with limited existing research or theory, unlike deductive qualitative analysis, which relies on predetermined themes from literature.

Two authors of this article acted as expert researchers, reviewing all responses to the open-ended mandatory questions and those allowing free-text input. For each response, they created a custom “code” by highlighting key phrases capturing significant insights using a predefined keyword. For instance, if a worker mentions participating in the longitudinal study because it was interesting and provided self-discovery, the initial code chosen by the authors might be the keyword task_interest. As analysis progresses, multiple core concepts emerge, forming the foundation of the initial overall coding scheme.

The qualitative analysis phase involved merging initially identified codes based on their inter-dependencies through multiple iterations and discussions. For instance, codes like task_interest, task_payment, and task_easiness were merged into the overarching theme of task_features. This process led to the emergence of seven themes, detailed in Table 1, with sample answers and initial codes. Due to expert involvement and iterative refinement, internal agreement is not reported here; interested readers can refer to McDonald et al. [48].

Table 2 details the distribution of additional free texts provided by workers while responding to each mandatory non-text-based question. For P1 part questions, the table reports both the total number of experiences with text and the number of workers providing it. For P2 part questions, only the latter is provided since workers are asked once per question. These texts augment the thematic analysis, offering additional insights to the quantitative analysis of provided answers.

Finally, it is important to note that if a question, such as question 1.1.X.9.2, is not included in the result analysis, it is because it specifically required text-based answers, and unfortunately, the workers did not provide any useful responses for analysis.

In the six cases where the answer provided by the workers was numeric, such as for question 1.1.X.1 of the P1 part (Section 6.1.2), we used ANOVA [56] to determine if there was a statistically significant difference (\(p \lt 0.05\)) between the means of the groups.

Specifically, we used a one-way ANOVA to compare the means of the three groups of workers (i.e., Amazon Mechanical Turk, Prolific, and Toloka). In the cases where we found a statistically significant difference at the \(p \lt 0.05\) level, we performed posthoc tests using Tukey’s HSD method [2] to determine which groups differed significantly from each other. Tukey’s HSD is a multiple comparison test that controls for Type I error rate by adjusting the significance level based on the number of pairwise comparisons.

For the nine closed-ended questions, which required choosing a mutually-exclusive answer from a predefined set, such as question 1.1.X.5 of the P1 part (Section 6.1.6), we used chi-squared tests to determine if there were statistically significant differences between the groups.

Specifically, we calculated the observed contingency table of frequencies and used the chi-squared test to compare it to the expected contingency table under the null hypothesis of no difference between the groups. To account for and correct multiple comparisons, we used the false discovery rate (FDR) correction [70], which controls the expected proportion of false discoveries among the rejected null hypotheses. If we encountered zero expected frequencies while performing the chi-squared test, we excluded the comparison from the analysis.

Similarly to the previous case, for the seven questions that allowed choosing multiple non-mutually-exclusive answers, such as question 7 of the P2 part (Section 6.1.21), we also used chi-squared tests to determine if there were statistically significant differences between the groups.

However, unlike the previous case, we had to address the situation where a respondent could select multiple options, resulting in overlapping categories. To accommodate this, we calculated the observed contingency table of frequencies using a modified approach that allowed for overlapping categories. Then, we then employed the chi-squared test and FDR correction, as in the previous case, to determine if there were significant differences between the groups.

To begin the investigation, we analyzed the previous experiences with longitudinal studies in which each worker reported having taken part, reported in Table 3. We recall that the charts shown in the following figures (Figures 1–22) should be interpreted as described in Section 5.3.2.

A total of 300 workers were recruited, with each platform contributing 100 workers. They reported 547 previous experiences with longitudinal studies, averaging 1.82 experiences per worker. Prolific workers reported the most experiences (193), followed by Amazon Mechanical Turk (187) and Toloka (167). Prolific had the highest proportion of workers with previous experience (35.28%), followed by Amazon Mechanical Turk (34.19%), while Toloka workers had less experience (30.53%). Additionally, 97 workers (32.3%) reported experiences from a different crowdsourcing platform than their recruitment platform (see also Figure 6).

Figure 1 details workers’ previous experiences with longitudinal studies from Table 3. The analysis shows that 45% of workers reported one experience, while 27.67% and 27.33% reported two and three experiences, respectively. These proportions varied across platforms. For Amazon Mechanical Turk, 42% reported one experience, 29% reported two, and 29% reported three. In Prolific, 43% reported one experience, 21% reported two, and 36% reported three. In Toloka, 50% reported one experience, 33% reported two, and 17% reported three. No statistically significant differences were observed across platforms.

The analysis suggests that workers on Prolific are more likely to report multiple previous experiences compared to those on other platforms, validating the recruitment criterion described in Section 5.2. Workers on Amazon Mechanical Turk and Toloka seem accustomed to longitudinal studies, indicating the need for a higher HIT completion threshold to recruit them effectively.

Figure 2 describes the time elapsed in terms of months since each previous experience reported, with a particular focus on participation in longitudinal studies that occurred up to 12 months earlier.

The majority of the reported experiences (87%), indeed, occurred within the 12 months preceding participation in the survey, while the remaining 13% occurred earlier. The distribution of participation that took place within the previous year, however, is rather homogeneous, with roughly 13% of participation for each crowdsourcing platform occurring more than 12 months earlier. This indicates that on Amazon Mechanical Turk and Prolific, workers were able to commit to longitudinal studies throughout the whole year before participating in this survey, while on Toloka, the experiences reported have been more recent (Amazon Mechanical Turk vs. Toloka statistically significant, adjusted p-value < 0.05).

Figure 3 details, for each previous experience with longitudinal studies reported, how many sessions composed the overall study referred.

The longitudinal studies in which workers participated on Amazon Mechanical Turk and Toloka have an average of about six sessions, while those on Prolific have seven sessions on average. In general, it appears that task requesters tend to publish slightly longer longitudinal studies on Prolific, although we did not obtain statistically significant comparisons across platforms.

Figure 4 details the time elapsed, in terms of days, between the sessions of the longitudinal study to which the reported experiences refer, focusing on ranges from 1 day to more than 30 days.

The timespans ranging from 1 day to 9 days, encompass the majority of the longitudinal studies referred to by the reported experiences (63.45%). By extending the considered range up to 30 days, the vast majority of previous experiences (90%) are comprised. Summarizing, most requesters schedule the next session of a study starting from the following day up to a month later, with ten days being the most common timespan (Amazon Mechanical Turk vs. Toloka statistically significant, adjusted p-value < 0.01).

Figure 5 details the duration of sessions in the longitudinal study to which the reported experiences refer, measured in minutes or hours.

Almost half of the longitudinal studies had sessions lasting 15 minutes (48.09%), while 22.89% lasted for 30 minutes, 12.72% for 45 minutes, and 12.41% for 60 minutes. The vast majority of sessions, thus, take place within an hour of work (96.11%). There is a small but not negligible number of sessions in longitudinal studies available on Toloka that last for 2 hours (13), along with two sessions on Amazon Mechanical Turk and a single session on Prolific. Furthermore, two workers reported Amazon Mechanical Turk sessions lasting 3 hours or more.

In general, the vast majority of task requesters on Prolific tend to publish longitudinal studies with shorter sessions, primarily 15 minutes (72%) or 30 minutes (20%), compared to other platforms. The answer distribution is more uniform when comparing Amazon Mechanical Turk and Toloka, although requesters on the latter platform tend to publish studies with longer sessions (Amazon Mechanical Turk vs. Prolific, Amazon Mechanical Turk vs. Toloka, and Toloka vs. Prolific statistically significant; adjusted p-value < 0.01).

Figure 6 describes on which previous experiences with longitudinal studies were conducted, as a worker recruited on a platform might have worked also elsewhere. Roughly, the same number of experiences took place on Amazon Mechanical Turk (38.16%) and Prolific (39.47%), while fewer experiences (22.37%) happened on Toloka.

Breaking down the responses by platform, the majority of experiences reported by Amazon Mechanical Turk and Prolific workers occurred on their respective platforms (around 90%). However, there were instances of cross-platform participation: 9% of Amazon Mechanical Turk workers reported experiences on Prolific, while 6% of Prolific workers reported experiences on Amazon Mechanical Turk and 4% on Toloka. Additionally, although experiences reported by Toloka workers primarily occurred on Toloka (63%), a notable portion also occurred on Amazon Mechanical Turk (17%) and Prolific (19%).

Summarizing, the distribution of the collected answers shows that Toloka workers tend to work on other platforms more frequently than those recruited from Amazon Mechanical Turk and Prolific, particularly in the context of longitudinal studies. However, this trend can also be observed on the remaining platforms (Amazon Mechanical Turk vs. Prolific, Amazon Mechanical Turk vs. Toloka, and Prolific vs. Toloka are statistically significant with an adjusted p-value < 0.01).

Figure 7 investigates the payment model adopted by the longitudinal studies in which the recruited workers reported participating.

The majority of reported previous experiences (70.31%) involved longitudinal studies where workers were paid after each session, while 21.84% reported experiences with a final reward as the only form of payment. Only 7.84% of the reported experiences described studies relying on a combination of both payment approaches.

The distribution of the answers collected shows that the majority of previous experiences reported were part of longitudinal studies in which the workers were paid after each session, particularly on Amazon Mechanical Turk (75%). Using a final reward is also a viable option, as in 25% of the experiences reported by workers recruited on Prolific and Toloka. Furthermore, 9% of the experiences reported by Amazon Mechanical Turk workers and 7% of those reported on the remaining platforms refer to longitudinal studies that employed a combination of both approaches (Amazon Mechanical Turk vs. Prolific, Amazon Mechanical Turk vs. Toloka and Prolific vs. Toloka statistically significant; adjusted p-value < 0.01).

Figure 8 investigates the workers’ satisfaction after having participated in the longitudinal study referred to by each reported experience.

The vast majority of workers (91.59%) express their interest in participating again in the same longitudinal study. When breaking down the data across each platform, such opinion is consistent for both Prolific and Toloka workers, with a percentage of positive answers of 98% and 93%, respectively, while it lowers to 83% for Amazon Mechanical Turk workers (Amazon Mechanical Turk vs. Prolific and Prolific vs. Toloka statistically significant; adjusted p-value < 0.01).

The mandatory open question 1.1.X.7.2 (P1 part) asks workers to specify what drove them to return for a second session after completing the first one in the longitudinal study referred to by the reported experience. Also, the workers must explain why they would refuse to participate in the same study altogether.

The workers provided 485 answers among the 547 previous experiences with longitudinal studies reported (88.66%). The distribution of answers collected across different themes is as follows: 272 out of 485 (56.08%) addressed aspects related to the task performed (task_features), while 101 (20.82%) focused on workers’ own beliefs and motivations (worker_features). Additionally, 10 (2.06%) were about the longitudinal study as a whole (ls_features), 9 (1.86%) about the requester (requester_features), and 2 (0.41%) about the platform (platform_features). Lastly, 91 (18.76%) answers were deemed unusable (answer_useless). Table 7 (Appendix B) shows a sample of such answers.

The majority of responses (272 out of 485, 56.08%) highlight how task attributes influence their decisions. Some workers find tasks interesting (100 out of 272, 36.76%), easy (54 out of 272, 19.85%), or well-paid (112 out of 272, 41.58%), which motivates their return. Others (15 out of 272, 5%) mention the perceived reliability of securing rewards in subsequent sessions as a driver to return. Many workers (58 out of 272, 41.58%) appreciate the task’s agency for expressing their views and getting paid in return. Conversely, issues like low or unfair rewards, worker unavailability during follow-up sessions, or device-specific requirements are common reasons for abandonment or refusal to participate in longitudinal studies after a session. About 20.82% of responses (101 out of 485) come from workers who believe their preferences and attributes influence their decision to return for subsequent sessions in longitudinal studies.

A few workers (4 out of 101, 3.96%) mentioned the sunk costs of completing the first session as a motivating factor to return [5]. Additionally, 45 out of 101 workers (44.55%) expressed satisfaction with completing the initial session, citing the commitment required (12 out of 101, 11.88%); overall involvement; or the chance to gain insights, learn, and develop skills throughout the studies (15 out of 101, 15%).

A small number of workers (9 out of 485, 1.86%) discussed aspects and characteristics of the task requester that impact loyalty and commitment to the longitudinal study. They highlghted communication with the requesters and their ability to remind participants of subsequent study sessions as crucial factors. Additionally, 10 workers out of 485 (2%) touched on aspects of the longitudinal study as a whole. They described the type of study they enjoy and explained how longitudinal studies provided guaranteed work without the need to compete for tasks.

Figure 9 addresses the underlying motivations that drive workers’ participation in the previous experiences with longitudinal studies reported.

Monetary aspects such as rewards and bonuses are the most important incentives for the participation in the majority of reported experiences (70.42%). Workers’ personal interest in the task proposed by the requester in the longitudinal study is an incentive for roughly 19% of experiences. Roughly 6% of participation in the reported experiences occurred because the worker found the task proposed educative, while the workers’ altruism, in terms of helping the overall research, has a lower but not negligible importance, considered by 4.71% of the respondents.

When considering each platform, it is interesting to note that 17% of Toloka workers found the task proposed in the reported experience with longitudinal study educative, while this component is almost absent from Amazon Mechanical Turk (1%) and Prolific. Furthermore, Prolific is the platform that published the majority of experiences that took place due to workers’ personal interest (26%) or willingness to help the research (7%). This may be due to the fact that such platforms are mostly focused on academic-related research projects, and task requesters are often researchers [58].

Generally, even though monetary aspects are the most popular incentives that drove workers to participate in the previous experiences with longitudinal studies reported, the remaining factors should not be overlooked when designing the overall longitudinal study (Amazon Mechanical Turk vs. Prolific, Amazon Mechanical Turk vs. Toloka, and Prolific vs. Toloka statistically significant; adjusted p-value < 0.01).

Figure 10 investigates whether workers completed the overall longitudinal study to which each reported experience refers. Specifically, they claim completion of almost every previous experience (97.65%), with only 13 experiences out of 547 (2.35%) dropped.

When considering each platform, workers claim completion of almost every experience on Prolific and Toloka (99%), while this amount is slightly lower for Amazon Mechanical Turk, particularly 95% (no statistically significant comparisons across platforms obtained). Even though the crowdsourcing platforms do not provide any means of verifying this data, we recall that we recruit workers with certain task completion rates (i.e., experienced workers), as described in Section 5.2. Thus, we argue that they have little incentive to provide inaccurate information about their previous completions.

Figure 11 addresses the underlying motivations that drive workers to complete the previous experiences with longitudinal studies reported and should be compared with the answers provided for question 1.1.X.8, analyzed in Section 6.1.10, which focuses on the ones that drive workers to participate. Indeed, while the set of possible answers is the same, this question restricts the focus to completed experiences and attempts to grasp the changes in workers’ perception of the overall experience. Thus, the 11 experiences from which workers dropped participation (i.e., those reported in the right half of Figure 10) are marked using a separate string, that is “Participation Dropped”, to allow a direct comparison of the bar charts.

Monetary aspects such as rewards and bonuses remain the most important factors for the majority of the previous experiences reported (68.3%), with a slight decrease (2.12%). The impact of workers’ personal interest in the task proposed by the requester (18.49%) remains almost unchanged, as does their opinion about the task being educative. Most of the answers that shift from monetary aspects, indeed, end up describing workers’ willingness to help with the overall research, from 4.71% to 6.06%.

When considering each platform, the overall distribution of answers does not change in terms of relative comparisons. The most noticeable difference is found for Prolific, where workers’ personal interest in the proposed task drops from 26% to 19%, becoming comparable with that of other platforms. A similar phenomenon occurs for Amazon Mechanical Turk, where interest in the final reward shifts from 49% to 55% (Amazon Mechanical Turk vs. Prolific, Amazon Mechanical Turk vs. Toloka and Prolific vs. Toloka statistically significant; adjusted p-value < 0.01).

The mandatory open-ended question 2 (P1 part) is used to ask workers about the adequacy and suitability of the crowdsourcing platform of provenance in the support they provide for longitudinal studies.

The majority of workers (273 out of 300, 91%) provided an answer that allows us to draw some kind of consideration. The distributions of the answers collected across different themes is as follows: 244 out of 273 (89.34%) addressed aspects related to the crowdsourcing platform (platform_features), while 11 (4.03%) focused on workers’ own beliefs and motivations (worker_features). Lastly, 18 (6.59%) answers were deemed unusable (answer_useless). Table 8 (Appendix B) shows a sample of such answers.

The vast majority of answers directly relate to the crowdsourcing platform of origin (244 out of 273, 89.34%). Breaking down the respondents across each platform reveals 98 workers from Amazon Mechanical Turk, 100 from Prolific, and 76 from Toloka. The majority of Amazon Mechanical Turk workers (70 out of 98, 71.43%) believe the platform is generally adequate, with few providing additional details. Three of them (3.06%) specifically mention the ease of sending reminders for upcoming longitudinal study sessions. Only seven (7.14%) find the platform inadequate in supporting longitudinal studies. One worker suggests that the platform needs design improvements to facilitate scheduling tasks for longitudinal studies, while another highlights the challenge for requesters to ensure worker honesty.

Nearly all Prolific workers (97 out of 100, 97%) consider the platform adequate for supporting longitudinal studies, with many providing detailed responses. Some mention the platform’s detailed task reports, facilitating tracking throughout the study. Others (7 out of 100, 7%) highlight the diverse backgrounds and skills of available individuals. Factors such as ease of contacting or sending reminders to workers using their identifier are noted by 16 out of 100 workers (16%). Additionally, two workers (2 out of 100, 2%) emphasize worker motivation and reliability as important considerations for researchers. Notably, one worker mentions being recruited from the platform via a third-party application that relies on the platform’s API.

The majority of Toloka workers (68 out of 76, 89.47%) consider the platform adequate overall, with few providing specific details. Two workers (2 out of 76, 2.63%) mention worker availability and the ease of contacting them using their identifier. One worker’s response is notable; they believe the platform cannot adequately support a longitudinal study due to residing in a country with poor network infrastructure.

When considering workers who are uncertain or outright deny the adequacy of the platform, several cross-platform factors become apparent. These workers are more likely to drop out of longitudinal studies due to perceived inadequacies. They express difficulties in assessing requester honesty, which can lead to skepticism about participating in such studies. Additionally, respondents believe that workers typically do not actively seek out longitudinal studies, suggesting a need for platforms to better distinguish these studies from standard crowdsourcing tasks.

Figure 12 investigates the reasons that limit the availability of longitudinal studies on crowdsourcing platforms according to workers’ opinions. The most prevalent reasons, chosen roughly the same number of times, are that workers dislike the required commitment (32.85%) and that the provided rewards and incentives are insufficient. Several answers indicate that longitudinal studies are not optimally supported by current popular crowdsourcing platforms (24.85%), and 9.07% of answers point out that usually requesters do not need longitudinal participation since most tasks deal with static data to annotate.

The distribution of answers changes when considering each platform. Specifically, 44% of the answers provided by Prolific workers indicate their dislike of the required commitment, while this factor is less important for Amazon Mechanical Turk workers (29%) and Toloka workers (26%). The lack of adequate technical support is prevalent among the answers provided by Toloka workers (35%), while for Prolific, this is reported by only 12% of the answers. The percentage of answers indicating that rewards and incentives are insufficient is slightly higher for Amazon Mechanical Turk (36%) compared to Toloka (33%), which in turn is slightly higher than Prolific (29%). Among the answers describing that often crowdsourcing tasks do not need longitudinal participation, those from Prolific are prevalent (15%).

Summarizing, workers indeed dislike the required commitment and find monetary aspects and related incentives insufficient. Also, they think that longitudinal studies are not adequately supported by crowdsourcing platforms (Amazon Mechanical Turk vs. Prolific, Amazon Mechanical Turk vs. Toloka, and Prolific vs. Toloka statistically significant; adjusted p-value < 0.01).

Figure 13 investigates the number of days workers would be happy to commit to for a longitudinal study, hypothesizing a single session having a duration of 15 minutes per day.

By considering each platform, Amazon Mechanical Turk and Toloka workers show rather similar trends, with mean numbers of days around 19 and 17, respectively. Turning to Prolific, this number increases to an average of almost a month (29 days).

Generally, Prolific is the platform that allows for finding workers willing to commit to longitudinal studies for longer periods, at least when compared with Toloka (Prolific vs Toloka statistically significant with adjusted p-value < 0.05).

Figure 14 investigates which are the reasons that drive workers to decline participation in longitudinal studies.

The majority of the answers provided by workers indicate that the length of the longitudinal study, in terms of the number of sessions and thus the time elapsed in days or even months since its start, is the most important factor (70.79%). The remaining answers (29.03%) indicate that the frequency of the sessions of the longitudinal study is also a reason that can lead to declining participation and should not be overlooked.

By considering each platform, the vast majority of answers provided by Prolific workers (85%) consider study length as a major concern, and this holds also when considering Toloka, albeit to a lesser extent (71%). As for Amazon Mechanical Turk, the trend is more nuanced, since the gap between answers that consider study length (57%) and study frequency (43%) is smaller (Amazon Mechanical Turk vs. Prolific, Amazon Mechanical Turk vs. Toloka and Prolific vs. Toloka statistically significant; adjusted p-value < 0.01).

Figure 15 investigates the preferred participation frequency in longitudinal studies according to the workers, in terms of time periods.

The vast majority of workers prefer frequent studies, having a daily to weekly participation commitment. Particularly, a daily participation is the most popular option overall (42.78%). Only a niche of 11 workers (6.68%) would prefer longer time periods.

There are some nuances among the preferences of the workers recruited from each platform. Particularly, Toloka workers prefer, for the most part, a daily participation frequency (53%). Prolific workers, on the other hand, have a slightly higher preference for a weekly frequency (40%), followed by a daily frequency (35%). For Amazon Mechanical Turk workers, the trend is the opposite, as they prefer a daily participation frequency (40%), shortly followed by a weekly frequency (38%). Regarding longer frequencies, it is worth noting that six Toloka workers (6%) prefer a biweekly frequency, and five Amazon Mechanical Turk workers (5%) along with three Prolific workers (3%) prefer a monthly frequency.

These findings can be aligned with those described in Figure 14, as indeed the study length is a major concern for workers (Amazon Mechanical Turk vs. Prolific, Amazon Mechanical Turk vs. Toloka and Prolific vs. Toloka statistically significant; adjusted p-value < 0.01).

Figure 16 investigates the preferred session duration in hours for longitudinal studies according to workers.

Prolific workers prefer short sessions of less than 1 hour on average, while Amazon Mechanical Turk and Toloka workers share a more uniform preference, indicating an average of about two hours. The figure does not show 9 outliers who provide non-reasonable durations (i.e., between 15 and 50 hours), thus they are removed.

Generally speaking, Amazon Mechanical Turk and Toloka workers are thus keen to work for a longer time within a single session when compared with Prolific workers (Amazon Mechanical Turk vs. Prolific and Prolific vs. Toloka statistically significant; adjusted p-value < 0.05).

Figure 17 investigates the acceptable hourly payment rate in USD$ for participating in longitudinal studies on the recruitment platform, as reported by the workers.

Amazon Mechanical Turk workers aim to receive the highest hourly payment on average (about $13), while for Prolific workers, this amount lowers to about $10.50. On the other hand, Toloka workers indicate the lowest acceptable amount of money (about $8.5). The figure does not include eight outliers who provided unreasonable amounts (i.e., amounts ranging between $80 and $100) and were thus removed.

To interpret the provided answers, one must consider that the payment models of Amazon Mechanical Turk and Toloka differ from that of Prolific. On the first two platforms, a task requester proposes a unitary amount of money for each work unit performed, which can be arbitrarily high. On the other hand, the Prolific platform requires requesters to estimate the task completion time and propose, instead of a unitary amount, a minimum amount of money based on the hourly estimate. Thus, this difference may impact the workers’ perception of the acceptable payment amount (Amazon Mechanical Turk vs. Prolific, Amazon Mechanical Turk vs. Toloka statistically significant; adjusted p-value < 0.05).

Figure 18 investigates the preferred amount of time in hours that workers are available to allocate for participating in longitudinal studies on a daily basis.

The workers recruited on Toloka are those keen to work more per day, being available to allocate up to almost four hours on average (3.81). Then, Amazon Mechanical Turk workers prefer working up to almost three hours (2.85), while Prolific ones expect to work less, with roughly an hour and a half (1.66). The figure does not show 18 outliers who provided non-reasonable amounts of hours per day (i.e., between 20 and 25), and were thus removed.

In general, Toloka workers are those who are keen to work more within a day and expect to be rewarded less. This is evident not only in the time they allocate daily for participation, as shown in Figure 18, but also when asked about their preferred session duration (Figure 16) or their ideal daily payment (Figure 17). As for Amazon Mechanical Turk and Prolific workers, they expect to work less on average, particularly the latter ones (Amazon Mechanical Turk vs. Prolific, Prolific vs. Toloka statistically significant; adjusted p-value < 0.05).

Figure 19 investigates the underlying motivations that drive participation in new longitudinal studies.

In general, the type of reward/payment mechanism is the most important incentive, according to the vast majority of answers provided by workers (81.86%). Among them, the preferred alternative is providing payment after each session (32.07%). As for the remaining ones, 24.22% indicate a final bonus to be awarded after the last session, while 20.38% prefer a progressive incremental payment after each session. A progressive decremental payment (2.51%) or eventual penalization for skipping one or more sessions (2.43%) have a small but not negligible influence on participation chances in new studies.

Beyond the reward/payment mechanism, 12.04% of answers indicate working on different task types to increase engagement diversity, while 6.18% suggest experimental variants of the same tasks to reduce repeatability. When considering each crowdsourcing platform, no particular trends emerge (Amazon Mechanical Turk vs. Prolific, Amazon Mechanical Turk vs. Toloka, and Prolific vs. Toloka statistically significant; adjusted p-value < 0.01).

Figure 20 investigates the tasks that workers would like to perform in a longitudinal study. We acknowledge that the predefined set of answers we provided might not have been perceived as exhaustive. Indeed, they were given the opportunity to provide a free-text response to further elaborate.

By surveys, we refer to surveys about various aspects that are usually crowdsourced, like demographics (22.71%). Verification and validation tasks require workers in the crowd to either verify certain aspects as per the given instructions, or confirm the validity of various kinds of content (17.99%). Interpretation and analysis tasks rely on the wisdom of the crowd to use their interpretation skills during task completion (17.92%). Information finding tasks delegate the process of searching to satisfy one’s information need to the workers in the crowd (16.51%). Content access tasks require the crowd workers to simply access some content (14.59%) and content creation tasks require the workers to generate new content for a document or website (10.28%).

It is worth noting that two workers mentioned in their free text responses other types of tasks, namely gamified tasks and content editing, which indeed is an option that we did not consider along with content access and content creation.

Summarizing, workers are willing to perform any of the task types proposed across each platform, with a rather homogeneous answer distribution. However, this distribution still accounts for statistical significance (Amazon Mechanical Turk vs. Prolific, Amazon Mechanical Turk vs. Toloka, and Prolific vs. Toloka statistically significant; adjusted p-value < 0.01).

Figure 21 investigates which are the benefits of being involved in longitudinal studies according to workers.

In general, workers think that the most important benefit characterizing longitudinal studies is increased productivity due to their more operational nature (32.1%). They also appreciate the time-saving aspect, as longitudinal studies eliminate the need for regular task searching (26.64%). Furthermore, workers think that receiving intermediate payments, after each session of the longitudinal study, would increase trust in the requester (25.81%). Additionally, some workers find value in avoiding the need to re-learn tasks when participating in longitudinal studies (15.45%).

The trends are homogeneous across all platforms, with no factor considered more important than others. However, the only exception is increased productivity, which is more prominent for Amazon Mechanical Turk workers (36%) and Toloka workers (37%) compared to Prolific (24%). Nonetheless, this distribution still accounts for statistical significance (Amazon Mechanical Turk vs. Prolific, Amazon Mechanical Turk vs. Toloka, and Prolific vs. Toloka statistically significant; adjusted p-value < 0.01).

Figure 22 investigates which are the downsides of being involved in longitudinal studies according to workers.

The answers provided by workers indicate that a reward provided only at the end of the longitudinal study is the most important downside (30.87%). The lack of flexibility in the study schedule and the long term commitment required have roughly are indicated by roughly the same amount of answers, namely 27.48% and 27.63%. The lack of diversity in terms of the work to be performed during each session of the overall study plays a minor role (14.02%).

By considering each platform, the trends are rather homogeneous for Amazon Mechanical Turk and Prolific. However, it is interesting to notice how the lack of diversity is a more prominent downside for Toloka workers (20%), while at the same time, the long-term commitment is less of an issue (21%) when compared to the remaining platforms (Amazon Mechanical Turk vs. Prolific, Amazon Mechanical Turk vs. Toloka, and Prolific vs. Toloka statistically significant; adjusted p-value < 0.01).

The last and optional question 11 (P2 part) asked workers to provide any suggestions to requesters that aim to design a longitudinal study.

There are 201 out of 300 (67%) workers who provide some kind of answer. The distribution of answers collected across different themes is as follows: 139 out of 201 (69.15%) addressed aspects related to the task performed (task_features), 9 (4.48%) focused on requesters (requester_features), and 7 (3.48%) focused on workers’ own beliefs and motivations (worker_features). Additionally, 5 (2.49%) were about the longitudinal study as a whole (ls_features), and 2 (1%) were about the platform (platform_features). Lastly, 2 (1%) answers were deemed unusable (answer_useless). Furthermore, 37 (18.41%) workers explicitly stated that they did not have any suggestions (no_suggestion). Table 9 (Appendix B) shows a sample of such answers.

The majority of workers (139 out of 201, 69.15%) suggest improvements related to the features of the task to be performed within each session of the longitudinal study, including its design, scheduling, and participant filtering. Six out of 139 workers (4.32%) propose allowing a reasonable window for completion, considering other activities in workers’ schedules. One worker suggests the option to skip a session if unable to commit occasionally. Additionally, a few workers (3 out of 139, 2.16%) emphasize the importance of conducting pilot tests for the tasks, which can help both requesters find suitable participants and retain workers throughout the study. A worker suggests offering different systems for participating in the study (e.g., desktop devices, smartphones) and another worker advises against requiring downloads. This resonates with prior work that has revealed diverse work environments that workers are embedded in [25]. Workers emphasize the need for clear instructions and user interface, an understandable sequence of events, identifying changes over time, and providing insight into cause-and-effect relationships. Some believe variability could help maintain interest in the study.

Regarding the overall structure of a longitudinal study (5 out of 201, 2.49%), workers suggest planning all sessions in advance while remaining flexible with the schedule, especially when involving multiple geographic time zones. They also recommend establishing a sense of progression, such as highlighting differences in previous responses at the end of each session.

A few workers (7 out of 201, 3.48%) provide personal insights. One worker notes that many are self-employed and must pay taxes on their earnings from crowdsourcing platforms, so rewards should reflect this. Another worker prefers small payments with a bonus for completing all sessions.

Considering aspects related to the task requesters (9 out of 201, 4.48%), workers think regular feedback from requesters is important. They suggest that requesters should be communicative and friendly, leave spaces for feedback in each study, send reminders when needed, and provide clear upfront information.

The workers’ answers for the P1 part of the survey are summarized in Table 4, while those provided for P2 in Table 5. Both tables provide a detailed summary of the answers, along with the code used to classify each question and a breakdown of responses across each crowdsourcing platform considered.

Table 6 shows the outcome of statistical tests performed by comparing the groups of answers provided across each platform. The table includes the name and answer type of each question. A checkmark (\(\checkmark\)) indicates a statistically significant comparison with the adjusted p-value provided, while its absence indicates that a given comparison was not statistically significant.

Finally, we summarize the key findings with a list of take-home messages, starting from the perception of longitudinal studies’ according to workers’ previous experiences (messages 1-9, P1 part questions), then moving to workers’ opinions about future longitudinal studies (messages 10-17, P2 part questions). For each message, we report a reference to the corresponding section where the analysis is reported.

Communication is a critical factor in encouraging workers’ retention and decreasing the abandonment rate, as emerges by their answers about what drove them towards returning to longitudinal studies and their suggestions to task requesters, described in Section 6.1.9.

According to the workers, task requesters should inform them about upcoming sessions, the progress made throughout the study, and, eventually, contact the workers explicitly to invite them to participate in newly published studies, also considering that several days can pass, as shown in Figure 4. Requesters should also provide information about the overall progress of the longitudinal study and feedback concerning the quality of the work performed up to the current session.

When asked to provide additional suggestions, as reported in Section 6.1.25, they also pointed out that alerts, e-mails, or notifications should be sent according to a regular schedule. Sending them randomly could be detrimental to the worker experience. Furthermore, they noted that platforms like Prolific provide only an internal notification system, without any way to send a standard e-mail to the worker.

Workers have free time to dedicate to participation in crowdsourcing tasks during different days of their working week, as reported in Section 6.1.25, and generally, it consists of roughly 2 hours on average (Figure 18). Scheduling the work required properly, thus, is particularly important for longitudinal studies, also considering that they can be composed of a potentially high number of sessions, as shown in Figure 3.

Determining a priori and explicitly stating the overall number of sessions is useful since it would allow the worker to estimate the amount of commitment required, especially given that they are available to commit for up to a month, as shown in Figure 13. Communicating when the subsequent session is going to happen will provide some flexibility to the workers.

Furthermore, task requesters should be careful when recruiting workers from multiple geographic time zones. For one worker, the session might start in the morning, while for another, it may be during the night. It could be beneficial to split the work required in multiple batches spread across the whole 24-hour timespan. Alternatively, requesters could provide a time frame that is large enough for workers to complete a session, with some of them suggesting 24 to 48 hours. Also, too much time should not pass between each session, as workers prefer a daily participation frequency (Figure 15). We thus argue that the workers may become bored or not recall the overall study, and thus drop participation halfway through.

The requester could also consider allowing workers to skip one or more sessions to provide additional flexibility, especially considering that the presence of eventual penalizations is not an aspect to further motivate workers in participating, according to them (Figure 19).

Crowdsourcing platforms measure worker performances and quality using various metrics and indicators, such as the time elapsed between accepting a given HIT and its successful submission and the overall completion rate. These indicators can be used by task requesters to filter the pool of available workers as, indeed, we ourselves have done (Section 5.2).

Workers suggest that they might avoid participating in longitudinal studies because they somehow believe that this could increase the odds of being rejected at any time after a given session, once completed, thus impacting the completion rates and performance as measured by the platform. In other words, workers fear performance measurement, especially in the context of longitudinal studies (Section 6.1.9).

A way to address such an issue is by disclosing and clarifying the whole study’s workflow, having a particular focus on the rejection criteria. They should be described accurately along with the behaviors and causes that may trigger them.

Even though longitudinal studies might increase the fear of performance indicators, task requesters should remember that workers find such kinds of studies more reliable than other types of crowdsourcing-based studies, as they point out when asked about their loyalty and commitment in Section 6.1.9.

Such reliability refers to the fact that workers find longitudinal studies to be more operational, as the same work is repeated over time, allowing for better productivity. Also, they think that longitudinal studies allow for avoiding spending time searching for new tasks, as shown by Figure 21.

They also suggest, in the answers analyzed in Section 6.1.25, that a successful longitudinal study demonstrates researcher honesty, increasing the overall trustworthiness. Hence, task requesters should employ a well-documented task design which is as consistent as possible across sessions, having a sound and understandable sequence of events. Turning back to Figure 21, it can be seen that several workers also find that a way to increase trust in a requester is planning intermediate payments.

Crowdsourcing platforms allow task requesters to recruit people from all over the world. This may include workers from countries known for not having adequate network infrastructure. For instance, when considering the Toloka platform, it is rather easy to find people from CIS countries [43] (Commonwealth of Independent States), as reported by a worker.

Task requesters should carefully consider where to recruit each worker since their provenance can profoundly affect their availability, loyalty, and commitment. For instance, when asked about the suitability of the platform for longitudinal studies in Section 6.1.13, a worker specifically points out that it needs further improvements for this specific studies such as for scheduling sessions, and infrastructural factors may further exacerbate issues which are platform intrinsic.

Workers may use various devices to perform crowdsourcing tasks. For instance, the Prolific platform offers task requesters a user interface control to explicitly allow the usage of certain device classes. Moreover, a worker could start working on a given device and then switch to another one, at a later time. This can be particularly true for longitudinal studies since they are made of different sessions that can be performed over an arbitrary amount of days.

The requester should thus design and build a layout as cross-platform as possible, thus offering the possibility of using different devices. However, workers do not necessarily agree with being required to download additional software to perform a crowdsourcing task, as they point out both in Sections 6.1.9 and 6.1.25. Task requesters should aim to provide a single (and possibly web-based) interface where the workers can perform the work required, whenever possible.

The most important incentives to foster longitudinal studies’ availability on crowdsourcing platforms and motivate workers in participating and completing them are those related to monetary aspects such as reward and bonuses, as shown in Figure 19.

While in a crowdsourcing setting, indeed, both terms refer to some kind of monetary compensation, the differences lie in the modalities by which they are provided. Usually, the reward is the payment planned at the task’s completion, while a bonus might be implicit or provided based on workers’ performance, among other factors. When further narrowing the focus to crowdsourcing-based longitudinal studies, the bonus is provided after completing all the sessions of a longitudinal study, or parts thereof, as done by Strickland and Stoops [75].

Task requesters should thus consider planning a reward after each individual session and one or more bonuses scattered throughout the study to minimize worker drop-off. The partial reward could be a fixed amount or initially low, increasing as the study progresses, as an incentive for consistent participation. Such a decision might help reduce the workers’ abandonment rate by further motivating them, and using an incremental form of payment helps contain the expenses during the initial stages.

Piloting a task to be performed helps reduce worker attrition due to errors and unexpected scenarios within its business logic, and longitudinal studies are no exception. In Section 6.1.25, the workers suggest that using good screeners can both help requesters find participants that fit the needs of the study, as well as participants that are less likely to quit part-way through.

Related to this, longitudinal studies may involve recruiting novice workers during subsequent sessions, as done by Roitero et al. [68]. Task requesters may consider deploying a lightweight training version of the task to be performed. This will help first-timers and prepare them to perform the overall study as expected.

One of the most pressing issues reported by the workers is the need of being reminded of an upcoming session when committing to a longitudinal study (Sections 6.1.13 and 6.1.25). For instance, a worker answered by reporting that they enjoyed participating because he had been reminded daily. Several workers also believe that longitudinal studies are not optimally supported in general, and this could be part of the problem. Hence, the crowdsourcing platform should allow task requesters to remind workers somehow.

A solution could involve allowing automatic reminders to be scheduled. These reminders could be scheduled after each session or after a fixed amount of time, and they should include a customizable message if needed. The reminders could be sent as notifications on the platform’s user interface or as simple email messages.

Workers often express a desire to perceive and understand their progress within a longitudinal study (Section 6.1.25). This desire is further motivated by the fact that some of them feel incentivized by their personal interest in the task, both in participating, as shown in Figure 9, and in completing it (Figure 11).

Similarly to reminding workers, allowing them to understand their progress within a longitudinal study seems a reasonable requirement at a first glance, yet it is hardly achievable on the platforms considered, as they generally provide feedback to the worker only within a single crowdsourcing task (i.e., a single session of the overall study).

One solution to provide feedback to the worker and build a sense of progress could be allowing requesters to display in advance the number of sessions of the whole study. Also, workers reported that they enjoy participating in longitudinal studies to monitor the changes in answers over time. This could thus be another interesting piece of information to be summarized and shown as a performance indicator.

Roitero et al. [68] designed and conducted a longitudinal study that involved asking workers to fact-check statements related to the COVID-19 pandemic delivered by public figures, such as politicians. A particular aspect of their study is that they republished a fixed set of HITs four times. Each time, they contacted the workers who previously participated, asking them to repeat the fact-checking activity. They also recruited novice workers to compare the work of the two groups.

Given that workers are willing to commit to a longitudinal study for roughly 22 days on average, as shown in Figure 13, it is natural to assume that many of them will drop out of participation, also considering that they consider study length as a major reason for doing so (Figure 14), and even though almost everyone claim completion of previous longitudinal studies (Figure 10). Indeed, Roitero et al. [68] measured task abandonment [33], reporting a 50% abandonment ratio on average.

In light of the case considered, the crowdsourcing platform should, first and foremost, offer a simple method to facilitate the recruitment of workers based not only on demographic criteria but also on their previous participation in the study. Additionally, it should provide a way to compensate for the reduced number of returning workers by explicitly asking the requester whether they want to recruit novice workers as well. As of today, Prolific somehow mitigates this by allowing for saving lists of worker groups that can be used to select the exact same participants for new studies.

When designing and publishing a study on a crowdsourcing platform, it is not possible to indicate that it will be conducted in a longitudinal fashion, by publishing additional sessions over time. Given that Figure 21 shows that several workers believe longitudinal studies enable them to avoid spending regular time searching for new tasks and allow them to be more productive, we suggest that platforms provide workers with a user interface filter to separate longitudinal studies from standard tasks. Consequently, platforms should offer requesters the option to choose whether their studies will be longitudinal or not.

While the idea of adding adequate user interface filters may seem obvious, the workers on every platform considered can only guess or rely on the study descriptions provided by the requester to understand whether they are going to participate in some kind of longitudinal study. Implementing this change will thus raise their awareness and facilitate participation, allowing task requesters to optimize the time needed to recruit the required number of workers.

Workers use a multitude of devices to participate in crowdsourcing tasks of any kind. Among the platforms considered, only Prolific allows task requesters to indicate the type of device class (i.e., mobile, desktop, or tablet) required to perform a given task, and workers can filter the available tasks accordingly. Specifically, a worker reported participating in a longitudinal study that involved maintaining a log on an Android device, along with collecting certain health data (e.g., heartbeat, etc.), which indeed took place on Prolific, while answering about platform suitability (Section 6.1.13).

Thus, crowdsourcing platforms should provide task requesters with a way to design a layout suitable for each device class. This could be achieved by offering a set of predefined and responsive user interface components, as done to some extent by MTurk with its Crowd HTML elements,⁵ or by Toloka with its template builder.⁶ The issue with these two approaches, however, is that they require considerable web development skills. Prolific, on the other hand, started moving in October 2023 towards such a direction by rolling out a survey builder that can be used to design simple polls consisting of 1-5 questions as of today.⁷

To further improve support for as many devices as possible, the platform could provide a way to design different layouts for the same task, one for each device class supported. Then, the workers should be allowed to choose studies compatible with a certain device class using an appropriate filter, similar to the choice between participating in a longitudinal or standard study. This best practice is general and not limited to longitudinal study design.