Nothing Special   »   [go: up one dir, main page]
Next Article in Journal
Study on Impact Mechanisms of Resident Adaptability in Old Communities Based on Complex Adaptive System Theory: Theoretical Construction and Empirical Analysis of Xuzhou City Center
Previous Article in Journal
Review of the Role of Urban Green Infrastructure on Climate Resiliency: A Focus on Heat Mitigation Modelling Scenario on the Microclimate and Building Scale
You seem to have javascript disabled. Please note that many of the page functionalities won't work as expected without javascript enabled.
 
 
Search for Articles:
Title / Keyword
Author / Affiliation / Email
Journal
Article Type
 
 
Advanced Search
 
Section
Special Issue
Volume
Issue
Number
Page
 
Journals
Urban Science
Volume 8
Issue 4
10.3390/urbansci8040222
urbansci-logo
Submit to this Journal Review for this Journal Propose a Special Issue
► Article Menu

Article Menu

share Share announcement Help format_quote Cite question_answer Discuss in SciProfiles thumb_up
...
Endorse textsms
...
Comment
first_page
settings
Order Article Reprints
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Article

Unlocking Blended Emotions and Underlying Drivers: A Deep Dive into COVID-19 Vaccination Insights on Twitter Across Digital and Physical Realms in New York, Using ChatGPT

by
Li Yin
*,
Mo Han
and
Xuanyi Nie
Department of Urban and Regional Planning, University at Buffalo, The State University of New York, 3435 Main Street, 116 Hayes Hall, Buffalo, NY 14214, USA
*
Author to whom correspondence should be addressed.
Urban Sci. 2024, 8(4), 222; https://doi.org/10.3390/urbansci8040222
Submission received: 17 October 2024 / Revised: 15 November 2024 / Accepted: 18 November 2024 / Published: 20 November 2024
Browse Figures
Figure 1
<p>Analysis flow chart.</p> "> Figure 2
<p>Results from the sentiment and emotion analysis: primary and secondary sentiments.</p> "> Figure 3
<p>Results from the topic analysis: factors contributing to the primary and secondary sentiments.</p> "> Figure 4
<p>Sentiment distributions by region and by borough (New York City): before and after the <span class="html-italic">We Can Do</span> campaign.</p> ">
Versions Notes

Abstract

:
The conclusions drawn from commonly used topic modeling and sentiment analysis of COVID-19 vaccination discussions on social media often hinge on researchers’ interpretation. These methods inadequately capture the nuanced real-world human emotions and struggle with identifying sarcasm and handling mixed sentiments. This study uses OpenAI API and its Large Language Models (LLM) to analyze tweets to further the discussion on improving vaccination literacy and fostering public trust. We employed LLM to uncover underlying topics associated with non-polarized sentiments to understand public concerns and factors eroding public confidence in vaccination. In addition, the city and regional level analysis provides a more detailed breakdown of spatial differences in the physical realm. Our results showed a blend of positive sentiments toward COVID-19 vaccination in New York State, with an underlying sense of concern. Our topic analysis reveals that social media platforms, which facilitate personal experience sharing, can influence both vaccination acceptance and hesitancy in positive and negative ways. Our method was able to better capture the nuanced emotions of real-world individuals. This approach is less subjective and more consistent than traditional models as it employs ChatGPT’s extensive pre-trained databases instead of relying on individual researchers’ judgments.

1. Background

The World Health Organization has identified vaccine hesitancy as a significant global health threat, potentially influenced by health information sourced from various channels, including the Internet and social media [1,2]. Misinformation about vaccination side effects is among the barriers that hinder acceptance. Ref. [1] (p. 2568) found that “there are considerable public health concerns raised by the anti-vaccine messaging on such platforms and the consequent potential for downstream vaccine hesitancy”.
Globally, social media has emerged as a vital conduit for medical information dissemination by connecting individuals, governments, and organizations across the cyber landscape [3,4]. This was particularly evident during the early stages of the COVID-19 pandemic, when social distancing and quarantine policies drastically curtailed in-person contact [5]. Sentiment analysis and emotion detection, as a critical area of natural language processing, are important for interpreting unstructured data such as those generated on social media platforms [6].
Amid the increasing body of research on social media data analysis related to COVID-19 vaccination, the widely used conventional sentiment analysis and topic modeling findings can diverge significantly depending on researchers’ interpretation. These techniques can be broadly classified into three categories based on the machine-learning approaches utilized: lexicon-based, machine-learning-based, and deep-learning-based methods. They span from associating specific emotions and topics with each word to categorizing text into various sentiments and topics and even extracting features from textual data [7,8,9].
However, existing research leveraging these methods was predominantly focused on recognizing polarity in texts with categorized positive and negative attitudes. This opinion-mining method inadequately captures the nuanced nature of real-world human emotions. It struggles with identifying sarcasm, and is ineffective in handling mixed sentiments, which are crucial for interpreting subjective and biased sentiment analysis [10]. Studies have shown that humans often experience multiple emotions simultaneously, with these emotions blending and mixing together [11,12]. Many prevalent methods rely on analyzing keyword frequencies, which is insufficient for capturing the richness and complexity of human emotions. Therefore, a multi-layer emotion analysis that considers mixed emotions is essential for a more comprehensive understanding of sentiments.
The recent emergence and advancement of Large Language Models (LLMs) that support OpenAI ChatGPT can serve as potent tools for a more in-depth analysis of sentiments and diverse opinions regarding COVID-19 vaccination. This study uses OpenAI API and its LLM to analyze the position of Twitter, a social media platform, in promoting vaccine benefits or propagating against COVID-19 vaccine hesitancy to further the discussion on the improvement of vaccination literacy and foster public trust. The multi-class and multi-layer sentiment and emotion analysis, enabled by the LLM, can help uncover underlying topics associated with various sentiments and emotions to understand public concerns and factors contributing to the compromise of public confidence in vaccination. Using this method, we examined ten sentiments and emotions across layers of both primary sentiments and secondary sentiments. The ten sentiments include four positive ones—hopefulness, happiness, trust, and relief—each corresponding to a specific positive emotion. Similarly, there are four negative sentiments—concern, skepticism, anger, and frustration—each linked to a distinct negative emotion. Additionally, there is a neutral sentiment, and a touch of sarcasm to capture more nuanced sentimental and emotional expressions. Finally, we investigate whether there are spatial differences in vaccination acceptance and hesitancy across regions in New York State and boroughs in New York City to identify and understand the regional and city-specific differences.
Our results from the multi-class and multi-layer sentiment and topic analysis can help identify specific triggers for negative sentiments, such as concern or skepticism to develop targeted interventions and design educational information to evoke positive emotions. Our results from the city and regional level analysis can provide a more detailed understanding of spatial differences in the physical realms, as opposed to cyberspace. While the overall patterns of sentiments and emotions at the aggregate level in cyberspace are observed, specific physical communities can exhibit distinct variations. Such results help policymakers spot shifts in public sentiment, thus making informed decisions and developing strategies for better resource allocations and crisis communications. Our study will push the discussion on using ChatGPT models and other LLMs to address vaccine hesitancy more effectively and foster public trust in vaccinations by decision-makers, practitioners, and researchers.

2. Literature Review

2.1. COVID-19 Pandemic and Vaccination Hesitancy and Acceptance

“Vaccination is one of the most successful public health interventions and a cornerstone for the prevention of communicable infectious diseases” [1] (p. 2568). The past century has seen major advances in vaccination. To curb the COVID-19 outbreak, achieving herd immunity through vaccination is crucial, and public acceptance plays a vital role in this effort. According to a survey conducted across 23 countries in 2022, the willingness to accept COVID-19 vaccination was 79.1%, marking a 5.2% rise from 2021, while the hesitancy rate went up in eight countries compared to 2021 [2]. Vaccine hesitancy, often depicted as anti-vaccination sentiment in the media that leads to the delay of acceptance or refusal of vaccination, has the potential to endanger global health [13]. Studies have shown a link between vaccine hesitancy and vaccine uptake [14].
The literature has identified factors linked to hesitancy through surveys and literature reviews, including low education level and unemployment, mistrust in science, government, health authorities, and health care systems [15,16], concerns about side effects, vaccine safety and efficacy [16,17], and misinformation [2,18]. The factors linked to higher acceptance included belief in vaccine effectiveness, vaccine trust, and perceived high susceptibility to fear of infection [3].
Many studies have suggested strategies to enhance vaccine acceptance, including improving health literacy, communication campaigns focusing on vaccine safety and effectiveness, building trust, addressing concerns and misinformation, fostering a supportive environment by celebrating and engaging influences and promoting vaccination, successful administration logistics, and implementing vaccine mandates [16,18,19]. Studies have found connections between local events and the messages trending on social media [20]. The active intervention to debunk vaccine misinformation on social media by the government and health organizations such as New York State and the CDC [21,22] can help increase the acceptance of COVID-19 vaccination. The national public education campaigns such as We Can Do, which started on 5 April 2021, using a mix of traditional and new media channels have impacted positively vaccine uptake in the United States [19].

2.2. Sentiment and Emotion Analysis: Applying LLM to Social Media Data

Ref. [23] found that the hashtag “#COVID” was heavily mentioned on social media during the COVID-19 pandemic, and COVID-19 vaccination is a topic that permeates conversations on social media [5]. As a convenient and accessible platform for medical information [24], social media played a pivotal role in expanding communication and interactions from the traditional physical space to the digital realm, where people freely express their feelings, opinions, and feedback [5,25]. Social media data have thus been widely used in the understanding of public perceptions of spaces, events, and policies, among others [26,27,28].

2.2.1. Social Emotion and Vaccination Acceptance

Emotional states can spread to others through emotional contagion, causing people to feel the same emotions unconsciously [29]. Messages on the Internet or social media can influence our emotional experiences, potentially impacting our behaviors and emotions, including vaccination acceptance [30].
On one hand, emotions have fundamental traits that allow them to be distinguished and categorized [31]. Ref. [32] identified six primary and 25 secondary emotions. Ref. [31] outlined six basic emotions: sadness, happiness, fear, anger, surprise, and disgust. These six were later expanded to include additional emotions [33]. Other studies have explored different emotion categories, such as eight discrete emotions, including anger, fear, anticipation, trust, surprise, sadness, joy, and disgust [34,35], and nine categorical emotions, identified by [36].
On the other hand, human emotions are rich and complex. Studies suggest that people can experience more than one emotion at a time, with emotions blending and mixing together [11]. Ref. [12] (p. 1) found that people “experienced positive and negative emotions simultaneously relatively frequently”. This highlights the complexity and multi-faceted nature of human emotions.
People feel emotions when they interpret an instance of affective sensation. How they conceptualize their emotional state depends on their understanding of the emotions, which is influenced by the context and the language used to describe them [37]. AI and sentiment analysis can help identify social emotions [38].

2.2.2. Sentiment and Emotion Analysis

“Emotions have never been alien in AI” [38] (p. 6). Sentiment and emotion analysis has served as a research method to potentially bridge natural intelligence with artificial intelligence in understanding social emotions [38].
The prevalent sentiment analysis and topic modeling techniques can be broadly classified into three categories: lexicon-based, machine-learning-based, and deep-learning-based methods. The lexicon-based approach, which includes tools such as the National Research Council Canada Lexicon (NRCL) and the Valence Aware Dictionary for Sentiment Reasoning (VADER), uses dictionaries and corpora to link each word with specific emotions and topics [9,39,40] Machine-learning-based methods use conventional algorithms such as SVM [41], Bidirectional Encoder Representations from Transformers (BERT), and Naïve Bayes to sort the text into various sentiments and topics [8,42,43]. Another example is Latent Dirichlet Allocation (LDA) [44]—a popular machine-learning-based technique used to identify underlying topics within texts [45]. Deep-learning-based methods deploy sophisticated neural networks to identify complex patterns from the text [46] (Lee and Dernoncourt, 2016).
Recent studies incorporated tools for the detection of a wider range of emotions, such as NRCL and BERT [3,35] (Hu et al., 2021; Wang and Chen, 2023). Ref. [43] used lexicons to analyze eight emotions and two sentiments. Ref. [3] used VADER to analyze three sentiments (positive, neutral, and negative) from Twitter data and used the NRCL to recognize four pairs of emotions (joy, sadness, anger, fear, trust, disgust, surprise, and anticipation). Ref. [46] used BERT to detect six emotions from COVID-19 tweets. The NRC Emotion Lexicon, as presented by Wang and Chen (2023), is extensively employed in the text emotion assessment. This analytical tool determines emotions by examining how frequently certain keywords appear, and where each keyword is linked to a specific emotion. However, this method falls short, as it cannot capture the complexity of sentiments going beyond simple word counts.
While the lexicon-based approach is straightforward—relying on keyword frequency to determine sentiment—this method can be context-insensitive and might produce erroneous results because a word could carry different meanings in different contexts. The widely used LDA relies on subjective input from the researchers by trial and error for identifying the optimal number of topics, which is inherently subjective. Lexicon-based and machine-learning-based methods often presuppose that sentiment analysis is a binary or straightforward linear problem that does not align with the complexity of human emotions and sentiments. Deep-learning-based methods require significant data to train the models effectively, aiming to recognize nuanced emotions in specific contexts.
Real-world sentiments often blend multiple elements. One single categorization struggles to handle mixed sentiments effectively. Real-world communication is rich and varied, often conveying emotions through complex constructs, such as idioms, irony, or sarcasm. Moreover, sentiment can be heavily context-dependent, with the same word carrying different emotional weights in different settings. Current sentiment analysis approaches are often insufficient for detecting mixed emotions and sarcasm. They also struggle to account for context, which can result in the misclassification of sarcastic or humorous sentiments [47].
Recent LLM methods are particularly potent in comprehending human language. Ref. [48] showcase the efficacy of LLMs in the analysis of public sentiment with the potential to significantly reduce both time and costs in interpreting various forms of sentiments. Different from traditional methods, the LLM approach offers a more human-like interpretation of complex sentiments. From the topic modeling perspective, LLM does not require a specific topic number, thereby permitting an unfettered and freeform selection of topics, according to the context. These all result in more accurate and insightful text analysis, which is particularly beneficial for interpreting the multifaceted nature of communication on social media.
Advanced Natural Language Processing (NLP) [49] and LLM [50] provide three advanced analytical capabilities beyond the traditional methods. First, the LLM can categorize sentiments into a wider range of emotions, based on the broader context within a tweet. This includes identifying the score and intensity of different emotions. Second, the LLM can employ its robust reasoning capabilities to analyze sentiments related to specific aspects of COVID-19 vaccinations. This analysis helps pinpoint which aspects are viewed positively or negatively, and importantly, it can point out the reasons behind these sentiments. Third, the LLM can summarize large volumes of tweets to obtain an overview of public sentiment without needing to read through each tweet. By utilizing these advanced analytical capabilities that traditional NLP models cannot do, we can extract richer, more detailed insights from Twitter data.

3. Method

We propose a method that leverages an advanced LLM model which has been pre-trained on a vast corpus of text data from the Internet, encompassing a wide array of contexts and styles. This extensive training and its alignment to human language behavior enable the LLM to interpret text with an understanding of nuance and context that goes well beyond traditional methods. We conducted multi-class and multi-layer sentiment analyses, followed by topic analysis and city and regional analysis (see Figure 1).

3.1. Data Collection

We collected COVID-19 vaccine-related discourse from Twitter’s streaming API. Since our study focused on public discussion about COVID vaccination, we used the names of laboratories such as Moderna, Pfizer, and Johnson & Johnson, along with various spellings of “COVID”. This approach helped filter out irrelevant tweets discussing COVID-19 in a broader context, resulting in more targeted and relevant data on vaccination.
The Twitter data were from the start of January 2021, to the end of June 2021, and the messages were sent from New York State. According to [51], this period has seen the most notable variations or increases in vaccination coverage among adults aged 18 years and older in the United States.
We used place ID information to identify locations within the state at the city level. The dataset encompasses tweets from a total of 392 different cities in New York state, with the highest number of tweets originating from Manhattan, NY, USA. We also consolidate the number of tweets into the regions used by the New York State Department of Health for analysis. Just under 5000 tweets were from the metropolitan region. Statewide data on daily COVID-19 vaccination at the city and county levels were collected from the New York State Department of Health for the year 2021.

3.2. Multi-Class and Multi-Layer Sentiment and Emotion Analysis

We utilized OpenAI’s pre-trained Generative Pre-trained Transformer (GPT), a cutting-edge language processing model for multi-class and multi-layer sentiment analysis. Our choice was driven by GPT’s superior ability to interpret and generate text that closely mimics human language, significantly outperforming other conventional language processing methods [52,53]. OpenAI’s models have been benchmarked to deliver high-quality results, which is pivotal for the accurate analysis of complex datasets such as those composed of tweets. The second advantage is the practicality of integration; compared to other open-source LLMs, OpenAI’s API allows for straightforward utilization without the requirement for extensive GPU resources on our part. By accessing this model through OpenAI API, we can leverage its advanced capabilities for the project. These factors collectively influenced our decision to employ this model, ensuring efficiency without sacrificing performance.
From the language understanding perspective, we used a more granular approach for sentiment and emotion analysis, which incorporates a wide range of sentiment and emotion labels to capture nuances beyond simple polarity. We used four positive sentiments, four negative sentiments, a sarcasm sentiment, and a neutral sentiment to represent ten distinct emotions for the sentiment and emotion analysis.
Before categorizing sentiments, we utilized the LLM to objectively extract the emotional tone of each tweet without any pre-established categories or limitations. We developed a set of sentiments and emotions by having the LLM analyze, summarize, and group the open-ended, organically derived sentiments from this initial run. We also incorporated insights from the literature review to align with the unique characteristics of our tweet dataset. This includes complex sentiments such as sarcasm, which traditional methods might miss. This tailored approach enhances the accuracy and relevance of our sentiment analysis. It allows us to provide a more precise interpretation of the data, as it reflects the actual sentiments expressed in the tweets, including those with multiple layers of meaning or non-obvious emotional expressions.
We asked GPT to identify primary and secondary sentiments in the texts using the ten discrete emotions. The scores for primary and secondary sentiments added up to 1. If there was no clear secondary sentiment identified, the primary sentiment received the full score of 1. If a secondary sentiment was present, it was assigned a score, but the total of both scores was still equal to 1, with the primary sentiment’s score being higher than that of the secondary.

3.3. Topic Analysis

The sentiments were then broken down by specific aspects or topics within a text to provide richer insights. While using the LLM to analyze the sentiment and emotion of each tweet, we simultaneously instruct the LLM to produce an additional set of outputs: the entity and the aspect related to each sentiment. In sentiment analysis, entities typically denote the names of products, services, people, events, and organizations, while aspects generally pertain to the characteristics and elements of these entities [54]. This dual output—entity and aspect—succinctly captures the content and focus of each tweet, allowing us to grasp the topic without reading the entire tweet. This is particularly useful when aligning specific sentiments with their corresponding topics, a task that traditional methods, which can only cluster topics, fail to achieve, as they do not distill the sentiment-specific topics.
In a process similar to how we analyzed tweet sentiment and emotion, we employed the LLM to freely identify the topic of each input, i.e., the entity and aspect pair linked to each sentiment per tweet. Once we have all these topic outputs extracted, we then instructed the LLM to cluster them into broader groups and clusters, without imposing a limit on the number of these groups. Finally, we categorized the sentiment of each tweet into the following 6 topic groups: vaccine efficacy, vaccine side effects, vaccine administration logistics, vaccine policy guidance, vaccine personal decisions and experiences, and others.

3.4. Model Validation

We validated the performance of the LLM in detecting sentiments and emotions to assess its reliability and effectiveness. Two sample sets, each comprising 5% of the total dataset, were randomly selected. We engaged two planning researchers and one student to manually determine the sentiments and emotions of the sample data at both the primary and second layers.
The human annotators, who had experience in qualitative data and were unaware of the LLM’s annotations, assessed emotions in this randomly selected set of tweets. Each tweet was assigned one of the eight emotions, plus sarcasm and neutral sentiments, for both primary and secondary sentiments. We then evaluated the consistency between the emotion classifications made by the human annotators and those identified by the LLM. To quantify this comparison, we used a simple percentage match, which is the ratio of instances where the human decision matched the LLM’s decision.
It is important to note that human annotators were limited to using one of these ten sentiments and emotions to label both the primary and secondary sentiments. Once the human annotations were completed, the annotators themselves conducted the matching process. They reviewed the classifications, comparing theirs and those by the LLM to determine alignment.

3.5. City and Regional Analysis

We further analyzed the four regions in New York State and five boroughs of New York City using the results from the sentiment and emotion analysis and vaccination data. The four regions of New York State, as defined in the vaccination data, are the metropolitan area, capital district, central New York, and Western New York. Each of the five boroughs of New York City is coextensive with a respective county of the State of New York: The Bronx corresponds to Bronx County, Brooklyn to Kings County, Manhattan to New York County, Queens to Queens County, and Staten Island to Richmond County.
We charted the number of people vaccinated daily for our study period in these boroughs and regions and summarized the number of positive, negative, neutral, and sarcasm sentiments across primary and secondary layers. A national public education campaign called We Can Do, which started on April 5, 2021, was found to have a positive impact on COVID-19 vaccine uptake in the United States. We conducted two analyses before and after the start of the campaign to explore whether this impact is reflected in Twitter sentiments across the regions in New York State and the boroughs of New York City.

4. Results and Discussions

4.1. Muti-Class and Multi-Layer Sentiment and Emotion Analysis

The results from the three reviews of the annotations on the sample dataset showed an average agreement of 94% between human annotations and LLM annotations. This high level of agreement indicates that our model is robust for our study purpose and demonstrates its potential for high reliability in nuanced emotional analysis within urban planning contexts.
Figure 2 shows the results from the multi-class, multi-layer sentiment and emotion analysis. The layer one/primary sentiments displayed in the first column are connected to the layer two/secondary sentiments listed in the second column. The sizes of the sentiment and emotion circles correspond to their ranks and percentages, summarized from all tweets, with larger circles indicating higher ranks and percentages. Every primary sentiment circle has three lines originating from it, linking to the top three secondary sentiments. The thickness of the line indicates the strength of the connection between the circles, with thicker lines representing a higher percentage of tweets associated with those circles.
This figure shows that the predominant primary sentiment and emotion associated with COVID-19 vaccination in New York State is relief. The predominant secondary sentiment and emotion is hopefulness, which is also connected to relief in the first layer. Concern appears in the top three sentiments across both layers, standing as the only negative sentiment among the top three. The primary sentiment of concern is intertwined with both trust and skepticism, resulting in a mixed negative sentiment. Concern is also linked to relief either as a primary or secondary sentiment, suggesting a mixed positive sentiment when there is a sense of worry. Trust ranks third among the primary sentiments. All three associated secondary sentiments are positive, including hopefulness, relief, and happiness. Among the primary sentiments, a small percentage corresponds to negative emotions such as skepticism, anger, and frustration, which are further linked to other negative emotions at the secondary layer. One exception is frustration, which is linked to relief.
Figure 2 illustrates that positive sentiments prevail among the public in New York State, as indicated by both primary and secondary sentiments. This aligns with other studies conducted during a similar time period in various countries around the world. In Spain, trust was identified as the predominant emotion [43], while in [55], the majority of sentiments were positive. In addition, Figure 2 depicts a blend of positive sentiments toward vaccination, accompanied by an underlying sense of concern. This aspect of blended emotions was not addressed in previous studies, underscoring our unique contribution. Our method was able to more effectively capture the nuanced and blended human emotions found in real-world scenarios, moving beyond simple positive and negative categorization.

4.2. Results from Topic Analysis

Figure 3 illustrates examples of the topics that underlie the dominant primary and secondary sentiments, centered around concern across both layers of sentiments and emotions. The topmost section provides a breakdown of the underlying topics and the corresponding percentages of tweets associated with each of those topics for the linked primary sentiments. The middle sections list the primary and secondary sentiments, respectively. The bottom section outlines the underlying topics and their associated percentages for the linked second sentiments.
The first graph shows that 33.4% of the Twitter discussion with relief as the top-ranked primary sentiment revolves around efficacy, followed by 22.2% discussing administration logistics. The secondary sentiment associated with relief is concern, with over 37% of the concern sentiment focusing on personal experience and approximately 20% on side effects. This suggests that, while people experience relief due to efficacy and efficient administration logistics, they also harbor concerns related to their personal experiences and vaccine side effects, which can be reactions after receiving the vaccine.
The central and last graphs illustrate the relationships between the second-ranked primary sentiment, its associated secondary sentiments, and the underlying topics. The central graph highlights the relationship between concern and relief. It shows that 40.2% of the discussion contributing to the concern emotion pertains to side effects, followed by a little less than 20% related to administration logistics. The concern is mixed with a sense of relief, driven by personal experience (38.7%) and administration logistics (15.7%). This implies that when the primary sentiment is concern, side effects play a significant role in generating this negative sentiment, even though people’s first-hand encounters with the vaccine and effective administration logistics contribute to alleviating it.
The last graph depicts the relationship between concern and trust. It shows that when there is a mixed feeling of concern and trust, the concern comes from administration logistics (33%) and efficacy (22%), while the trust comes from efficacy (26.2%) and administration logistics (23.8%). In this case, vaccine efficacy and administration logistics can significantly influence both negative and positive sentiments. On one hand, people may trust the vaccine’s ability to prevent infection and reduce the severity of the disease. On the other hand, they may be concerned about the presented numbers and the vaccine’s protective capacity.

4.3. Results from City and Regional Analysis

Figure 4 displays the sentiments across four regions of New York State (top part) and the five boroughs of New York City (bottom part). Every borough or region is represented by a pair of bars: the first corresponds to the period before the We Can Do public education campaign launched in April 2021 to increase vaccine confidence through digital outreach, while the second corresponds to the period after the campaign. The blue bars represent positive sentiments from both primary and secondary categories, while purple represents negative sentiments. In the middle section, the two-line graphs depict the daily count of vaccinated individuals for these regions (left) and boroughs (right), from January to June 2021. Notably, these charts reveal a peak in April following the launch of the We Can Do campaign.
This figure shows that the public’s emotions are dominated by positive sentiments across all boroughs in New York City and all regions in New York State, although a small percentage of sarcastic tweets exist in all boroughs and regions. In New York City, Staten Island has higher percentages of negative sentiments compared to other boroughs. It echoes results from a survey conducted by the City University of New York, which showed that Staten Island has the lowest percentage of people trusting the science behind COVID-19 vaccines [56].
The disparity between the bars within each pair reflects the shifts in sentiment and emotion before and after the We Can Do campaign. Our analysis results showed a slight decrease in negative sentiments and a slight increase in positive sentiments after the campaign across all New York City boroughs, except for Staten Island, where another study has found lower trust levels in the scientific information about COVID-19 vaccination. At the regional level, the Central New York region experienced an increase in negative sentiments after April. A more comprehensive analysis is necessary to comprehend this deviation from the national trend in vaccination rates observed in several studies. Nevertheless, these results largely corroborate the findings from other studies on the positive impact of the digital education campaign [19,56].

5. Discussion and Conclusions

This study aimed to examine Twitter discussions on COVID-19 vaccinations using multi-class and multi-layer sentiment and emotion analysis, along with topic analysis, to investigate public perceptions of vaccination benefits, side effects, and delivery, along with their influence on opinions about vaccinations. Even though positive sentiments prevail among the public in New York State, our results also depict a blend of positive sentiments toward vaccination, accompanied by an underlying sense of concern. Our approach allowed us to detect and quantify mixed sentiments. Rather than labeling them using one single label, we were able to identify both primary and secondary sentiments. Our model was also able to recognize sarcasm and irony. Using contextual cues and linguistic patterns can aid in distinguishing these nuanced expressions.
The outcomes of our topic analysis are less subjective and more consistent than those derived from prevalent models because the results rely on ChatGPT’s vast training databases rather than individual researchers’ judgments. Our findings suggest that social media platforms such as Twitter, which enable the sharing of personal experiences about vaccination, can impact both vaccination acceptance and hesitancy in both positive and negative ways. Improving individual first-hand experiences and effectively managing the distribution and delivery of the vaccine can help vaccination acceptance.
Prior research has highlighted that the main reason for willingness to accept COVID-19 vaccination is the desire to protect oneself or others, while concerns about side effects and safety are the primary reasons for unwillingness [57]. Systematic reviews and meta-analyses have shown that interventions such as educational programs and public health campaigns effectively increased vaccination rates across various populations, including adolescents and underserved and vulnerable populations [58,59,60].
Traditional methods to boost vaccination rates rely on infrastructure development, such as Immunization Information Systems [61] and the involvement of pharmacists [62]. These approaches can be relatively expensive and time-consuming to implement. Social networks have been increasingly used to disseminate information about pandemics, response plans, and public health measures [63], demonstrating a cost-effective communication strategy for public health promotion through social media [64].
Given social media’s ability to influence public opinions and actions (Wang and Vergeer, 2004), it becomes an effective pathway to encourage compliance with preventative initiatives and public health norms [65,66]. Personal experiences can be shared with vast populations on social media via retweets [43]. Therefore, emphasizing positive experiences and rationalities through social media can achieve public health promotion goals. Our results also suggest the importance of communication and education campaigns through digital channels, including social media, that focus on vaccine efficiency, trust-building, fostering a supportive environment by involving influencers, and ensuring efficient administration logistics.
The multi-class and multi-layer sentiment and emotion analysis we used provides a more detailed understanding of people’s feelings. This method can more effectively capture the nuanced and blended human emotions found in real-world scenarios, moving beyond simple positive and negative categorization to understand public perception, opinion, and experience of vaccination. By leveraging the pre-trained LLM that can capture context, this approach allows for consideration of surrounding words and phrases to improve sentiment classification. It can also help generate less subjective and more consistent results to help identify specific pain points or areas for improvement and problem-solving and promote specific features that the public cares about. The results can inspire uplifting strategies based on emotional responses and help inform issue-focused content for dealing with vaccination hesitancy.
The city and regional level analysis provides a more detailed breakdown of spatial differences in the physical realm, as opposed to cyberspace. While the results at the city and regional levels exhibit similar patterns of both positive and negative sentiment and emotion compared to those at the aggregate level, certain areas show variations. These discrepancies warrant further comprehensive study to examine the underlying reasons.
This paper focuses on exploring a pre-trained LLM for multi-class and multi-layer sentiment and emotion analysis, as well as for topic analysis, over a six-month period suggested by the literature, with significant increases in vaccination coverage among adults aged 18 years and older in the United States. To comprehensively analyze changes in sentiment trends, future studies should collect data over several years and throughout the different phases of the vaccination process to identify hotspots and areas of trend change, thereby illustrating the trajectory of sentiment change more effectively [43].

Author Contributions

Conceptualization, L.Y.; methodology, L.Y. and M.H.; software, L.Y. and M.H.; validation, L.Y., M.H. and X.N.; formal analysis, L.Y. and M.H.; investigation, L.Y. and M.H.; resources, L.Y. and M.H.; data curation, L.Y.; writing—original draft preparation, L.Y. and M.H.; writing—review and editing, L.Y., M.H. and X.N.; visualization, L.Y., M.H. and X.N.; supervision, L.Y.; project administration, L.Y. All authors have read and agreed to the published version of the manuscript.

Funding

This research received no external funding.

Data Availability Statement

Data available from the authors upon request.

Acknowledgments

We greatly appreciate the constructive comments from all three reviewers. We thank Selena Han for the work on the model validation, results interpretation, and paper editing.

Conflicts of Interest

The authors declare no conflict of interest.

References

  1. Puri, N.; Coomes, E.A.; Haghbayan, H.; Gunaratne, K. Social media and vaccine hesitancy: New updates for the era of COVID-19 and globalized infectious diseases. Hum. Vaccines Immunother. 2020, 16, 2586–2593. [Google Scholar] [CrossRef] [PubMed]
  2. Lazarus, J.V.; Wyka, K.; White, T.M.; Picchio, C.A.; Gostin, L.O.; Larson, H.J.; Rabin, K.; Ratzan, S.C.; Kamarulzaman, A.; El-Mohandes, A. A survey of COVID-19 vaccine acceptance across 23 countries in 2022. Nat. Med. 2023, 29, 366–375. [Google Scholar] [CrossRef] [PubMed]
  3. Hu, T.; Wang, S.; Luo, W.; Zhang, M.; Huang, X.; Yan, Y.; Liu, R.; Ly, K.; Kacker, V.; She, B.; et al. Revealing public opinion towards COVID-19 vaccines with Twitter data in the United States: Spatiotemporal perspective. J. Med. Internet Res. 2021, 23, e30854. [Google Scholar] [CrossRef] [PubMed]
  4. Garcia, K.; Berton, L. Topic detection and sentiment analysis in Twitter content related to COVID-19 from Brazil and the USA. Appl. Soft Comput. 2021, 101, 107057. [Google Scholar] [CrossRef]
  5. Saud, M.; Mashud, M.; Ida, R. Usage of social media during the pandemic: Seeking support and awareness about COVID-19 through social media platforms. J. Public Aff. 2020, 20, e02417. [Google Scholar] [CrossRef]
  6. Li, M.; Hua, Y.; Liao, Y.; Zhou, L.; Li, X.; Wang, L.; Yang, J. Tracking the Impact of COVID-19 and Lockdown Policies on Public Mental Health Using Social Media: Infoveillance Study. J. Med. Internet Res. 2022, 24, e39676. [Google Scholar] [CrossRef]
  7. Blei, D.M.; Ng, A.Y.; Jordan, M.I. Latent dirichlet allocation. J. Mach. Learn. Res. 2003, 3, 993–1022. [Google Scholar]
  8. Johnson, R.; Zhang, T. Effective Use of Word Order for Text Categorization with Convolutional Neural Networks. arXiv 2014, arXiv:1412.1058. [Google Scholar]
  9. Birjali, M.; Kasri, M.; Beni-Hssane, A. A Comprehensive Survey on Sentiment Analysis: Approaches, Challenges and Trends. Knowl.-Based Syst. 2021, 226, 107134. [Google Scholar] [CrossRef]
  10. Nandwani, P.; Verma, R. A review on sentiment analysis and emotion detection from text. Soc. Netw. Anal. Min. 2021, 11, 81. [Google Scholar] [CrossRef]
  11. Berrios, R.; Totterdell, P.; Kellett, S. When Feeling Mixed Can Be Meaningful: The Relation Between Mixed Emotions and Eudaimonic Well-Being. J. Happiness Stud. 2018, 19, 841–861. [Google Scholar] [CrossRef]
  12. Trampe, D.; Quoidbach, J.; Taquet, M.; Avenanti, A. Emotions in Everyday Life. PLoS ONE 2015, 10, e0145450. [Google Scholar] [CrossRef] [PubMed]
  13. Callender, D. Vaccine hesitancy: More than a movement. Hum. Vaccines Immunother. 2016, 12, 2464–2468. [Google Scholar] [CrossRef] [PubMed]
  14. Wagner, A.L.; Porth, J.M.; Wu, Z.; Boulton, M.L.; Finlay, J.M.; Kobayashi, L.C. Vaccine hesitancy during the COVID-19 pandemic: A latent class analysis of middle-aged and older US adults. J. Commun. Health 2022, 47, 408–415. [Google Scholar] [CrossRef]
  15. Shakeel, C.S.; Mujeeb, A.A.; Mirza, M.S.; Chaudhry, B.; Khan, S.J. Global COVID-19 vaccine acceptance: A systematic review of associated social and behavioral factors. Vaccines 2022, 10, 110. [Google Scholar] [CrossRef]
  16. Majid, U.; Ahmad, M.; Zain, S.; Akande, A.; Ikhlaq, F. COVID-19 vaccine hesitancy and acceptance: A comprehensive scoping review of global literature. Health Promot. Int. 2022, 37, daac078. [Google Scholar] [CrossRef]
  17. Zhang, Y.; Banga Ndzouboukou, J.L.; Gan, M.; Lin, X.; Fan, X. Immune evasive effects of SARS-CoV-2 variants to COVID-19 emergency used vaccines. Front. Immunol. 2021, 12, 4842. [Google Scholar] [CrossRef]
  18. Rossi, M.M.; Parisi, M.A.; Cartmell, K.B.; McFall, D. Understanding COVID-19 vaccine hesitancy in the Hispanic adult population of South Carolina: A complex mixed-method design evaluation study. BMC Public Health 2023, 23, 2359. [Google Scholar] [CrossRef]
  19. Williams, C.J.; Kranzler, E.C.; Luchman, J.N.; Denison, B.; Fischer, S.; Wonder, T.; Ostby, R.; Vines, M.; Weinberg, J.; Sayers, E.L.P.; et al. The Initial Relationship Between the United States Department of Health and Human Services’ Digital COVID-19 Public Education Campaign and Vaccine Uptake: Campaign Effectiveness Evaluation. J. Med. Internet Res. 2023, 25, e43873. [Google Scholar] [CrossRef]
  20. Gunaratne, K.; Coomes, E.A.; Haghbayan, H. Temporal trends in anti-vaccine discourse on Twitter. Vaccine 2019, 37, 4867–4871. [Google Scholar] [CrossRef]
  21. New York State. Governor Hochul Announces #GetTheVaxFacts Campaign to Combat COVID-19 Vaccine Misinformation; NYS Governor’s Press Office: New York, NY, USA, 2021. Available online: https://www.governor.ny.gov/news/governor-hochul-announces-getthevaxfacts-campaign-combat-covid-19-vaccine-misinformation (accessed on 1 March 2024).
  22. CDC. Connecticut Uses Social Media to Engage Long-Term Care Residents. COVID-19 Vaccine Community Features; CDC: Atlanta, GA, USA, 2021. Available online: https://archive.cdc.gov/www_cdc_gov/vaccines/covid-19/health-departments/features/index.html (accessed on 20 February 2022).
  23. Chen, E.; Lerman, K.; Ferrara, E. Tracking social media discourse about the COVID-19 pandemic: Development of a public coronavirus Twitter data set. JMIR Public Health Surveill. 2020, 6, e19273. [Google Scholar] [CrossRef] [PubMed]
  24. Hua, Y.; Jiang, H.; Lin, S.; Yang, J.; Plasek, J.M.; Bates, D.W.; Zhou, L. Using Twitter Data to Understand Public Perceptions of Approved versus Off-label Use for COVID-19-related Medications. J. Am. Med. Inform. Assoc. 2022, 29, 1668–1678. [Google Scholar] [CrossRef] [PubMed]
  25. Wang, C.; Yin, L. Defining Urban Big Data in Urban Planning: Literature Review. J. Urban Plan. Dev. 2023, 149, 04022044. [Google Scholar] [CrossRef]
  26. Plunz, R.A.; Zhou, Y.; Carrasco Vintimilla, M.I.; Mckeown, K.; Yu, T.; Uguccioni, L.; Sutto, M.P. Twitter sentiment in New York City parks as measure of well-being. Landsc. Urban Plan. 2019, 189, 235–246. [Google Scholar] [CrossRef]
  27. Roberts, H.; Sadler, J.; Chapman, L. The value of Twitter data for determining the emotional responses of people to urban green spaces: A case study and critical evaluation. Urban Stud. 2019, 56, 818–835. [Google Scholar] [CrossRef]
  28. Shin, E.J. What Can We Learn from Online Reviews? Examining the Reviews of Los Angeles Metro Rail Stations. J. Plan. Educ. Res. 2023, 43, 254–267. [Google Scholar] [CrossRef]
  29. Zuboff, S. Surveillance Capitalism and the Challenge of Collective Action. New Labor Forum 2019, 28, 10–29. [Google Scholar] [CrossRef]
  30. Kramer, A.D.I.; Guillory, J.E.; Hancock, J.T. Experimental evidence of massive-scale emotional contagion through social networks. Proc. Natl. Acad. Sci. USA 2014, 111, 8788–8790. [Google Scholar] [CrossRef]
  31. Ekman, P. An argument for basic emotions. Cogn. Emot. 1992, 6, 169–200. [Google Scholar] [CrossRef]
  32. Shaver, P.; Schwartz, J.; Kirson, D.; O’Connor, C. Emotion knowledge: Further exploration of a prototype approach. J. Pers. Soc. Psychol. 1987, 52, 1061–1086. [Google Scholar] [CrossRef] [PubMed]
  33. Ekman, P. Handbook of cognition and emotion. In Handbook of Cognition and Emotion; John Wiley & Sons: Hoboken, NJ, USA, 1999; pp. 226–232. [Google Scholar]
  34. Plutchik, R. A psychoevolutionary theory of emotions. Soc. Sci. Inf. 1982, 21, 529–553. [Google Scholar] [CrossRef]
  35. Wang, Y.; Chen, Y. Characterizing discourses about COVID-19 vaccines on Twitter: A topic modeling and sentiment analysis approach. J. Commun. Health 2023, 16, 103–112. [Google Scholar] [CrossRef] [PubMed]
  36. Tomkins, S.S.; McCarter, R. What and where are the primary affects? Some evidence for a theory. Percept. Mot. Ski. 1964, 18, 119–158. [Google Scholar] [CrossRef] [PubMed]
  37. Barrett, L.F. Solving the Emotion Paradox: Categorization and the Experience of Emotion. Pers. Soc. Psychol. Rev. 2006, 10, 20–46. [Google Scholar] [CrossRef] [PubMed]
  38. Navarro, J.; Marijuán, P.C. Natural intelligence and the ‘economy’ of social emotions: A connection with AI sentiment analysis. Biosystems 2023, 233, 105039. [Google Scholar] [CrossRef]
  39. Casadei, P.; Lee, N. Global cities, creative industries and their representation on social media: A micro-data analysis of Twitter data on the fashion industry. Environ. Plan. A Econ. Space 2020, 52, 1195–1220. [Google Scholar] [CrossRef]
  40. Wang, X.; Vergeer, M. Effect of Social Media Posts on Stock Market During COVID-19 Infodemic: An Agenda Diffusion Approach. SAGE Open 2024, 14, 21582440241227688. [Google Scholar] [CrossRef]
  41. Do, H.J.; Lim, C.-G.; Kim, Y.J.; Choi, H.-J. Analyzing emotions in twitter during a crisis: A case study of the 2015 Middle East Respiratory Syndrome outbreak in Korea. In Proceedings of the 2016 International Conference on Big Data and Smart Computing (BigComp), Hong Kong, China, 18–20 January 2016; pp. 415–418. [Google Scholar]
  42. Khan, S.M.; Chowdhury, M.; Ngo, L.B.; Apon, A. Multi-class twitter data categorization and geocoding with a novel computing framework. Cities 2020, 96, 102410. [Google Scholar] [CrossRef]
  43. Turón, A.; Altuzarra, A.; Moreno-Jiménez, J.; Navarro, J. Evolution of social mood in Spain throughout the COVID-19 vaccination process: A machine learning approach to tweets analysis. Public Health 2023, 215, 83–90. [Google Scholar] [CrossRef]
  44. Jelodar, H.; Wang, Y.; Yuan, C.; Feng, X.; Jiang, X.; Li, Y.; Zhao, L. Latent Dirichlet allocation (LDA) and topic modeling: Models, applications, a survey. Multimed. Tools Appl. 2019, 78, 15169–15211. [Google Scholar] [CrossRef]
  45. Lee, J.Y.; Dernoncourt, F. Sequential Short-Text Classification with Recurrent and Convolutional Neural Networks. arXiv 2016, arXiv:1603.03827. [Google Scholar]
  46. Singh, M.; Jakhar, A.K.; Pandey, S. Sentiment analysis on the impact of coronavirus in social life using the BERT model. Soc. Netw. Anal. Min. 2021, 11, 33. [Google Scholar] [CrossRef] [PubMed]
  47. Maynard, D.G.; Greenwood, M.A. Who cares about sarcastic tweets? investigating the impact of sarcasm on sentiment analysis. In Proceedings of the Lrec 2014 Proceedings. Language Resources and Evaluation Conference (LREC), Reykjavik, Iceland, 26–31 May 2014. [Google Scholar]
  48. Fu, X.; Sanchez, T.W.; Li, C.; Reu Junqueira, J. Deciphering Public Voices in the Digital Era: Benchmarking ChatGPT for Analyzing Citizen Feedback in Hamilton, New Zealand. J. Am. Plan. Assoc. 2024, 90, 728–741. [Google Scholar] [CrossRef]
  49. Nadkarni, P.M.; Ohno-Machado, L.; Chapman, W.W. Natural language processing: An introduction. J. Am. Med. Inform. Assoc. 2011, 18, 544–551. [Google Scholar] [CrossRef]
  50. Ouyang, L.; Wu, J.; Jiang, X.; Almeida, D.; Wainwright, C.; Wainwright, C.; Mishkin, P.; Agarwal, S.; Slama, K.; Ray, A.; et al. Training language models to follow instructions with human feedback. Adv. Neural Inf. Process. Syst. 2022, 35, 27730–27744. [Google Scholar]
  51. Kriss, J.L.; Hung, M.-C.; Srivastav, A.; Black, C.L.; Lindley, M.C.; Lee, J.T.; Koppaka, R.; Tsai, Y.; Lu, P.-J.; Yankey, D.; et al. COVID-19 Vaccination Coverage, by Race and Ethnicity—National Immunization Survey Adult COVID Module, United States, December 2020–November 2021. CDC Morbidity and Mortality Weekly Report. 2022. Available online: https://pubmed.ncbi.nlm.nih.gov/35679179/ (accessed on 1 March 2024).
  52. Zhao, W.X.; Zhou, K.; Li, J.; Tang, T.; Wang, X.; Hou, Y.; Min, Y.; Zhang, B.; Zhang, J.; Dong, Z.; et al. A survey of large language models. arXiv 2023, arXiv:2303.18223. [Google Scholar]
  53. Liu, B. Aspect and Entity Extraction. In Sentiment Analysis: Mining Opinions, Sentiments, and Emotions; Cambridge University Press: Cambridge, UK, 2015; pp. 137–188. [Google Scholar]
  54. Xu, H.; Liu, R.; Luo, Z.; Xu, M. COVID-19 vaccine sensing: Sentiment analysis and subject distillation from twitter data. Telemat. Inform. Rep. 2022, 8, 100016. [Google Scholar] [CrossRef]
  55. City University of New York Graduate School of Public Health & Health Policy. COVID-19 Survey—August 2022. Available online: https://sph.cuny.edu/research/covid-19-survey-august-2022/ (accessed on 20 April 2024).
  56. Denison, B.; Dahlen, H.; Kim, J.C.; Williams, C.; Kranzler, E.; Luchman, J.N.; Trigger, S.; Bennett, M.; Nighbor, T.; Vines, M.; et al. Evaluation of the “We Can Do This” Campaign Paid Media and COVID-19 Vaccination Uptake, United States, December 2020–January 2022. J. Health Commun. 2022, 28, 573–584. [Google Scholar] [CrossRef]
  57. Wang, Q.; Yang, L.; Jin, H.; Lin, L. Vaccination against COVID-19: A systematic review and meta-analysis of acceptability and its predictors. Prev. Med. 2021, 150, 106694. [Google Scholar] [CrossRef]
  58. Schumacher, S.; Salmanton-García, J.; Cornely, O.A.; Mellinghoff, S.C. Increasing influenza vaccination coverage in healthcare workers: A review on campaign strategies and their effect. Infection 2021, 49, 387–399. [Google Scholar] [CrossRef]
  59. Siddiqui, F.A.; Padhani, Z.A.; Salam, R.A.; Aliani, R.; Lassi, Z.S.; Das, J.K.; Bhutta, Z.A. Interventions to Improve Immunization Coverage Among Children and Adolescents: A Meta-analysis. Pediatrics 2022, 149 (Suppl. S5), e2021053852D. [Google Scholar] [CrossRef] [PubMed]
  60. Norman, G.; Kletter, M.; Dumville, J. Interventions to increase vaccination in vulnerable groups: Rapid overview of reviews. BMC Public Health 2024, 24, 1479. [Google Scholar] [CrossRef] [PubMed]
  61. Groom, H.; Hopkins, D.P.; Pabst, L.J.; Morgan, J.M.; Patel, M.; Calonge, N.; Coyle, R.; Dombkowski, K.; Groom, A.V.; Kurilo, M.B.; et al. Immunization Information Systems to Increase Vaccination Rates: A Community Guide Systematic Review. J. Public Health Manag. Pract. 2015, 21, 227–248. [Google Scholar] [CrossRef] [PubMed]
  62. Isenor, J.E.; Edwards, N.T.; Alia, T.A.; Slayter, K.L.; MacDougall, D.M.; McNeil, S.A.; Bowles, S.K. Impact of pharmacists as immunizers on vaccination rates: A systematic review and meta-analysis. Vaccine 2016, 34, 5708–5723. [Google Scholar] [CrossRef] [PubMed]
  63. Rufai, S.R.; Bunce, C. World leaders’ usage of Twitter in response to the COVID-19 pandemic: A content analysis. J. Public Health 2020, 42, 510–516. [Google Scholar] [CrossRef]
  64. Liang, H.; Fung, I.C.-H.; Tse, Z.T.H.; Yin, J.; Chan, C.-H.; Pechta, L.E.; Smith, B.J.; Marquez-Lameda, R.D.; Meltzer, M.I.; Lubell, K.M.; et al. How did Ebola information spread on twitter: Broadcasting or viral spreading? BMC Public Health 2019, 19, 438. [Google Scholar] [CrossRef]
  65. Bernhardt, J.M. Communication at the core of effective public health. Am. J. Public Health 2004, 94, 2051–2053. [Google Scholar] [CrossRef]
  66. Lezine, D.A.; Reed, G.A. Political Will: A Bridge Between Public Health Knowledge and Action. Am. J. Public Health 2007, 97, 2010–2013. [Google Scholar] [CrossRef]
Urbansci 08 00222 g001
Figure 1. Analysis flow chart.
Figure 1. Analysis flow chart.
Urbansci 08 00222 g001
Urbansci 08 00222 g002
Figure 2. Results from the sentiment and emotion analysis: primary and secondary sentiments.
Figure 2. Results from the sentiment and emotion analysis: primary and secondary sentiments.
Urbansci 08 00222 g002
Urbansci 08 00222 g003
Figure 3. Results from the topic analysis: factors contributing to the primary and secondary sentiments.
Figure 3. Results from the topic analysis: factors contributing to the primary and secondary sentiments.
Urbansci 08 00222 g003
Urbansci 08 00222 g004
Figure 4. Sentiment distributions by region and by borough (New York City): before and after the We Can Do campaign.
Figure 4. Sentiment distributions by region and by borough (New York City): before and after the We Can Do campaign.
Urbansci 08 00222 g004
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.

Share and Cite

MDPI and ACS Style

Yin, L.; Han, M.; Nie, X. Unlocking Blended Emotions and Underlying Drivers: A Deep Dive into COVID-19 Vaccination Insights on Twitter Across Digital and Physical Realms in New York, Using ChatGPT. Urban Sci. 2024, 8, 222. https://doi.org/10.3390/urbansci8040222

AMA Style

Yin L, Han M, Nie X. Unlocking Blended Emotions and Underlying Drivers: A Deep Dive into COVID-19 Vaccination Insights on Twitter Across Digital and Physical Realms in New York, Using ChatGPT. Urban Science. 2024; 8(4):222. https://doi.org/10.3390/urbansci8040222

Chicago/Turabian Style

Yin, Li, Mo Han, and Xuanyi Nie. 2024. "Unlocking Blended Emotions and Underlying Drivers: A Deep Dive into COVID-19 Vaccination Insights on Twitter Across Digital and Physical Realms in New York, Using ChatGPT" Urban Science 8, no. 4: 222. https://doi.org/10.3390/urbansci8040222

APA Style

Yin, L., Han, M., & Nie, X. (2024). Unlocking Blended Emotions and Underlying Drivers: A Deep Dive into COVID-19 Vaccination Insights on Twitter Across Digital and Physical Realms in New York, Using ChatGPT. Urban Science, 8(4), 222. https://doi.org/10.3390/urbansci8040222

Article Metrics

Back to TopTop