G Model
COLEGN-700; No. of Pages 8
ARTICLE IN PRESS
Collegian xxx (2020) xxx–xxx
Contents lists available at ScienceDirect
Collegian
journal homepage: www.elsevier.com/locate/coll
Effectiveness of the Mental Health First Aid programme for general
nursing students in Hong Kong: A randomised controlled trial
Maria S.Y. Hunga,∗ , Meyrick C.M. Chowa , Wai Tong Chienb , Prudence Y.K. Wongc
a
b
c
Tung Wah College, Hong Kong Special Administrative Region, China
Nethersole School of Nursing, Faculty of Medicine, The Chinese University of Hong Kong, Hong Kong Special Administrative Region, China
The Mental Health Association of Hong Kong, Hong Kong Special Administrative Region, China
a r t i c l e
i n f o
Article history:
Received 6 July 2019
Received in revised form
10 December 2019
Accepted 15 April 2020
Available online xxx
Keywords:
Nursing students
Undergraduate students
Mental wellbeing
Mental Health First Aid
a b s t r a c t
Background: Insufficient mental health related knowledge among healthcare professionals and students
may lead to higher levels of stigma towards and increased social distance from mentally unwell patients.
Moreover, university students enrolled in healthcare programmes are particularly susceptible to stress,
depression, anxiety and other mental health problems.
Objective: This study evaluated the effectiveness of the Mental Health First Aid (MHFA) Programme for
undergraduate general nursing students in improving their mental health literacy, decreasing their social
distance from persons with mental health issues, increasing their mental health first-aid intention, and
enhancing their confidence in assisting others. The programme was also extended the effectiveness to
help students to maintain their own mental wellbeing.
Methods: A randomised controlled trial study with a pre-test, post-test, and follow-up research design was
conducted. Three hundred and fifty-eight students were recruited. The experimental group comprised
182 students, of whom 168 completed MHFA training and a post-test questionnaire in mid-2017, and
167 students completed a follow-up questionnaire early in 2018. Meanwhile, 175 and 165 of the 176
students in the control group completed the post-test and follow-up questionnaires, respectively.
Results: The effectiveness of the MHFA programme was supported by the results of repeated measures
analysis of variance with Greenhouse-Geisser correction. Significant interaction effects were found in
mental health knowledge in recognition of depression (p2 = 0.17; 95% CI: 0.24, 0.42), recognition of
schizophrenia (p2 = 0.16; 95% CI: 0.33, 0.47), depression social distance (p2 = 0.20; 95% CI: −0.41.
−0.22), schizophrenia social distance (p2 = 0.31; 95% CI: −0.58, −0.37), confidence in helping (p2 = 0.04;
95% CI: 0.11. 0.14), mental first-aid actions (p2 = 0.08; 95% CI: 1.75, 2.90), and mental wellbeing
(p2 = 0.15; 95% CI: −0.53, −0.19).
Discussion: Mental illnesses present substantial challenges to healthcare professionals worldwide. MHFA
training should be promoted locally and integrated within compulsory training and extracurricular activities in curricula developed for healthcare and general university students.
Conclusion: This study confirms the benefits of MHFA for general nursing students in developing their
professional understanding of and willingness to assist people with mental health problems, and in
developing, maintaining, and improving their understanding of their own mental health.
© 2020 Australian College of Nursing Ltd. Published by Elsevier Ltd.
1. Introduction and background
According to the World Health Organization (World Health
Organization, 2017), approximately one in four people worldwide
∗ Corresponding author at: School of Nursing, Tung Wah College, 31
Wylie Road, Homantin, Kowloon, Hong Kong Special Administrative Region.
Tel.: +852 34686804.
E-mail addresses: mariahung@twc.edu.hk (M.S.Y. Hung), wtchien@cuhk.edu.hk
(W.T. Chien).
are affected by mental illness during their lifetimes. The WHO recommends global collaborative efforts to promote mental health
and prevent mental illness. Most mental health problems first arise
during adolescence and are highly prevalent among college and
university students, particularly those in the field of healthcare
(Eisenberg, Hunt, & Speer, 2012; Kelly et al., 2011). Prolonged academic study, together with a heavy workload and high-pressure
clinical practicum, may have adverse effects on the mental and
physical well-being of these students (He, Turnbull, Kirshbaum,
Phillips, & Klainin-Yobas, 2018; IsHak et al., 2013; Lee, Lee, & Lee,
https://doi.org/10.1016/j.colegn.2020.04.006
1322-7696/© 2020 Australian College of Nursing Ltd. Published by Elsevier Ltd.
Please cite this article in press as: Hung, M. S. Y., et al. Effectiveness of the Mental Health First Aid programme for general nursing
students in Hong Kong: A randomised controlled trial. Collegian (2020), https://doi.org/10.1016/j.colegn.2020.04.006
G Model
COLEGN-700; No. of Pages 8
2
ARTICLE IN PRESS
M.S.Y. Hung et al. / Collegian xxx (2020) xxx–xxx
Summary of relevance
Problem
• Little is known about the effect of the Mental Health First
Aid (MHFA) programme in improving the mental health knowledge, behaviour, attitudes, and wellbeing of general nursing
students.
What is already known?
• The MHFA programme is a psycho-educational training
programme that has been shown to enhance mental health
knowledge in diverse populations in various countries.
What does this paper add?
• The findings show that completing the MHFA programme
increases general nursing students’ mental health knowledge,
confidence, and intention to help, which will improve their
ability to care for patients with mental and physical comorbidities, as well as benefiting their friends, families, and the
public. Furthermore, MHFA training has the secondary effect
of enhancing general nursing students’ self-awareness and
mental wellbeing.
The Mental Health Review Report (Food & Health Bureau,
HKSAR, 2017) published by the Hong Kong government noted that
some forms of psychosis, such as schizophrenia, are typically more
evident in late adolescence and early adulthood. Tertiary institutions are thus optimally positioned to organise peer support
training to promote and improve adolescents’ mental wellbeing.
For nurses, knowledge regarding the early identification of mental health problems and effective interventions, together with a
positive attitude and willingness to offer help and support for individuals with mental health disorders, should be a fundamental
attribute of patient care.
This waitlist-based randomised controlled trial aimed to evaluate the effectiveness of MHFA training for undergraduate nursing
students. The objectives were as follows: 1) to increase the participants’ mental health literacy; 2) to decrease the social distance
between the participants and people with mental health problems;
3) to increase the participants’ confidence in helping others; 4) to
enhance participants’ mental health first-aid intentions; and 5) to
improve the participants’ mental wellbeing as a secondary effect of
the MHFA programme.
2. Methods
2018; Sun et al., 2016), and indicates the need to provide extensive
support for this population (Cheung et al., 2016; He et al., 2018;
Mospan, Hess, Blackwelder, Grover, & Dula, 2017; Rotenstein et al.,
2016). Moreover, healthcare students and nurses who face mental health problems have expressed concerns about disclosure and
lack of acceptance from their colleagues (Fokuo et al., 2017; Knaak,
Mantler, & Szeto, 2017).
In addition, it has been reported in several studies that
insufficient mental health related knowledge among healthcare
professionals and students might lead to higher levels of stigma,
prejudice, and social distance when these individuals are faced with
mentally unwell patients (Bingham & O’Brien, 2018; Chernomas
& Shapiro, 2013; Fokuo et al., 2017; Giandinoto, Stephenson, &
Edward, 2018; Knaak et al., 2017; Mak et al., 2015). In particular, nurses often spend considerably more time providing patient
care than other healthcare professionals. These obstacles likely hinder nurses’ motivation and ability to provide quality holistic care.
Accordingly, relevant pre-registration education for nursing students that establishes a holistic and positive view of people with
mental disorders may better facilitate the needs of these patients.
The MHFA programme is an effective standardised psychoeducational training programme that was established in Australia
in 2000. It aims to enhance attendees’ knowledge of general mental
health and common mental problems such as anxiety, depression, and psychosis, and the self-harming behaviours that can be
triggered by these problems, with the aim of supporting people
with mental health problems (Kitchener & Jorm, 2002). In the last
decade, several evaluations of diverse target populations in various countries, including nursing students, have demonstrated the
effects of MHFA training (Bond, Jorm, Kitchener, & Reavley, 2015;
Burns et al., 2017; Davies, Beever, & Glazebrook, 2018; Hung, Lam,
& Chow, 2019; Kelly & Birks, 2017; Svensson & Hansson, 2014;
Wong, Lau, Kwok, Wong, & Tori, 2017). A recent systematic review
and meta-analysis of 18 studies of MHFA training with 5936 participants demonstrated that the training significantly improved
the participants’ knowledge and identification of mental illnesses,
increased their awareness of effective treatments for these illnesses, and improved their confidence when assisting individuals
with mental health problems (Morgan, Ross, & Reavley, 2018). In
addition, in a recent qualitative study of nursing students’ experience of MHFA training conducted by the first two authors of this
paper, the participants reported gaining a greater awareness of the
importance of their own mental wellbeing from the programme
(Hung et al., 2019).
2.1. Participants and procedures
The eligible participants were all students who (a) were at least
18 years of age, (b) were enrolled in an undergraduate general
nursing programme, and (c) could understand both Chinese and
English language materials. The study setting was one of the largest
tertiary institutions in Hong Kong, which provides several undergraduate general nursing programmes for more than 1600 nursing
students. The estimated minimum sample size was 128 per group
in a two-group statistical analysis, assuming an effect size of 0.5, a
desired power of 0.8 and an ␣ level of 0.05 (Cunningham & McCrumGardner, 2007).
2.2. Ethical considerations
After receiving ethical approval for the study from the Research
Ethics Sub-Committee of the participating institution, general nursing undergraduate students were invited to join the study through
class announcements and emails in April 2017. Three hundred and
fifty-eight eligible students were enrolled within one week during May 2017. The participants were supplied with an information
sheet that clearly described the purpose, procedure, risks, and benefits of the study, and the voluntary nature of participation. They
were also informed of their ability to withdraw from the study at
any time without any negative consequences. Informed consent
was then obtained from all of the participants.
2.3. Data collection and intervention
The first author (HSYM) generated a random allocation
sequence table. Two research assistants were responsible for then
enrolling and assigning the participants to either the experimental
group or a waitlisted control group. Self-administrated questionnaires were used to collect data immediately after randomisation
(pre-test), after participation in the 12-hour MHFA programme
(post-test), and six months after completion (follow-up). Fig. 1
shows a flow diagram of different stages of the study. During
the face-to-face MHFA training sessions (Hong Kong version) conducted by certified MHFA instructors in Cantonese with English
supplementation, the participants were given an overview of the
main classifications of mental health disorders, such as anxiety,
depression, psychosis, and related manifestations, such as substance abuse.
Please cite this article in press as: Hung, M. S. Y., et al. Effectiveness of the Mental Health First Aid programme for general nursing
students in Hong Kong: A randomised controlled trial. Collegian (2020), https://doi.org/10.1016/j.colegn.2020.04.006
G Model
COLEGN-700; No. of Pages 8
ARTICLE IN PRESS
M.S.Y. Hung et al. / Collegian xxx (2020) xxx–xxx
Fig. 1. The stages of the study.
The Chinese version of MHFA manual (Hong Kong 3rd Edition) (The Mental Health Association of Hong Kong, 2015) and
an information booklet about local mental health resources were
distributed to each participant. Each three-hour training session
comprised small-group interactive teaching and learning activities, and four sessions were administered over a one-month period.
During these sessions, the participating students learned not only
the signs and symptoms, possible risk factors, and crises associated with these mental health problems, but also strategies to help
those suffering from these problems. In the experimental group,
168 of the 182 students completed MHFA training and the posttest questionnaire during mid-2017, and 167 students completed
the follow-up questionnaire early in 2018. Meanwhile, 175 and 165
of the 176 students in the control group completed the post-test
and follow-up questionnaires, respectively. After completing the
follow-up questionnaire, the participants in the control group also
underwent MHFA training, as this was expected to benefit all the
participants. Overall, 20 12-hour MHFA training classes comprising
approximately 15–20 students were provided for the experimental
and waitlisted groups in this study.
3
tal illness. Their knowledge and beliefs of students regarding the
‘helpfulness’ of different professionals, such as general physicians,
clinical psychologists, and counsellors, who can offer assistance to
people with mental health problems (i.e., professional consensus)
and appropriate treatment decisions (i.e., beliefs about treatment)
were also assessed. Each correct answer received one mark, and a
higher score indicated better mental health knowledge.
For the social distance scale, five types of interpersonal relationship were used to describe the participant’s willingness to have a
relationship with a person who has mental health problems, such
as living in a neighbourhood or marrying into a family with a person
who has mental health problems. These items were scored on a 4point Likert scale ranging from 1 (definitely would) to 4 (definitely
would not). A higher mean score indicated greater social distance.
The reliability coefficients for the social distance scale ranged from
0.86 to 0.93, which demonstrated good internal consistency.
In addition, nursing students were asked about their confidence
in their ability to provide assistance to individuals with mental
health disorders. They were required to respond to this item on
a 5-point Likert scale ranging from 1 (not confident at all) to 5
(extremely confident).
For the assessment of mental health first-aid intentions, the participants were asked whether they had observed mental health
problems themselves during the past six months, in their family
members or friends or anyone else they knew, and if so, what kinds
of assistance or MHFA actions they had been able to offer. One
mark was given to each action that was based on the ‘ALGEE’ action
plan (Kitchener, Jorm, & Kelly, 2017), comprising assessment of the
risk of harm or suicidal plan, listening non-judgementally to the
person’s problems and encouraging the person to talk with family members or others. A higher score indicated that more MHFA
actions had been performed.
2.6. Chinese Health Questionnaire (CHQ)
The CHQ was selected to measure the students’ mental health.
This tool was developed to screen for minor mental health problems
(e.g., symptoms of general psychological distress or minor psychiatric disorders) in the general population, including adolescents
in community settings (Cheng, Wu, Chong, & Williams, 1990). It
assesses the participant’s mental health status at the time of the
interview relative to his or her usual state. The CHQ comprises 12
items, such as ‘Have you been suffering from a headache or pressure in your head?’ Each item is scored on a 4-point Likert scale
ranging from 1 (never) to 4 (more than usual), and summed to calculate a mean score. A lower score indicates better mental health,
and a higher score represents more psychological distress. The reliability coefficients for the CHQ ranged from 0.83 to 0.86, displaying
satisfactory internal consistency.
2.4. Outcome measures
2.7. Data analysis
The outcomes were measured in terms of local relevance
and application, according to the participants’ self-completed
responses to two well-validated questionnaires: the Mental Health
Literacy Scale (Chinese version) and the Chinese Health Questionnaire (CHQ).
IBM SPSS-Statistics 23 was used for the data analysis, which was
conducted on an intention-to-treat basis. The missing data for every
outcome measure were imputed by using the last-observationcarried-forward method.
2.5. Mental Health Literacy Scale (Chinese version)
3. Results
Mental health literacy was evaluated using the Mental Health
Literacy Scale (Chinese version), which comprises two scenarios of
major depression and schizophrenia that fulfil the indicative standards in the Diagnostic and Statistical Manual of Mental Disorders
(Fourth Edition) (Wong, Lam, & Poon, 2010). The scale assessed the
participants’ recognition of mental health disorders, awareness of
mental health problems, and knowledge of treatments for men-
The participants comprised 358 nursing students (63 (17.6%)
male and 295 (82.4%) female). Table 1 shows the descriptive statistics for the experimental and control groups, respectively. The
358 students were randomised into either the experimental group
(n = 182) or the control group (n = 176). Twenty-five (13.9%) participants in the experimental group and 14 (8%) in the control
group had previously undergone mental health training, and 59
Please cite this article in press as: Hung, M. S. Y., et al. Effectiveness of the Mental Health First Aid programme for general nursing
students in Hong Kong: A randomised controlled trial. Collegian (2020), https://doi.org/10.1016/j.colegn.2020.04.006
G Model
COLEGN-700; No. of Pages 8
ARTICLE IN PRESS
M.S.Y. Hung et al. / Collegian xxx (2020) xxx–xxx
4
Table 1
Descriptive statistics for the experimental and control group
Experimental group
Mean ± SD
N
Gender
Male
Female
Age
Previous training on mental health training
Mental health issues in self
Control group
Frequency
Percentage
Mean ± SD
182
Frequency
Percentage
176
36
146
19.9%
80.1%
25
59
13.9
32.8
20.80 ± 1.91
27
149
15.3%
84.7%
14
68
8.0%
36.8
20.76 ± 1.91
(32.8%) and 68 (38.6%) participants in the experimental and control groups, respectively, had experienced problems with their
own mental health. A chi-square test and independent samples
t-test found no significant differences between the experimental
and control groups in terms of sex [ 2 (1) = 1.27, p = 0.26], age
[t (354) = 0.27, p = 0.82], or previous training in mental health [2
(1) = 3.32, p = 0.07].
A repeated-measures analysis of variance with GreenhouseGeisser correction was used to investigate the effectiveness of
MHFA training. Table 2 provides the means, standard deviations,
within-subject variances, and effect sizes of the outcome measures. Significant interaction effects were observed across the
pre-test, post-test, and follow-up assessments of the depression
scenario, including recognition of depression [F (2,355) = 36.56,
p < 0.01, p2 = 0.17], professional consensus [F (2,355) = 3.82, p <
0.05, p2 = 0.02], beliefs about treatment [F (2,355) 4.28, p <
0.05, p = 0.02], and social distance [F (2,355) = 47.01, p < 0.01,
p2 = 0.20]; the schizophrenia scenario, including recognition of
schizophrenia [F (2,355) = 33.94, p < 0.01, p2 = 0.16], professional
consensus [F (2,355) = 6.28, p < 0.01, p2 = 0.03], beliefs about treatment [F (2,355) = 7.50, p < 0.01, p2 = 0.04], and social distance [F
(2,355) = 81.35, p < 0.01, p2 = 0.31]; confidence in the ability to
help [F (2,355) = 7.53, p < 0.01, p2 = 0.04]; mental health first-aid
action [F (2,355) = 5.92, p < 0.01, p2 = 0.08]; and mental health
status in the CHQ [F (2,355) = 31.69, p < 0.01, p2 = 0.15].
Furthermore, a post hoc Bonferroni correction revealed that
the MHFA training improved the participants’ scores between the
pre-test and post-test for the depression scenario, including recognition of depression (Cohen’s d = −1.10, p < 0.01) and social distance
(Cohen’s d = 0.88, p < 0.01), and the schizophrenia scenario, including recognition of schizophrenia (Cohen’s d = 1.44, p < 0.01), social
distance (Cohen’s d = 1.15, p < 0.01), confidence in the ability to provide help (Cohen’s d = −0.34, p < 0.01), and the CHQ score (Cohen’s
d = 0.76, p < 0.01). This effectiveness was sustained at the six-month
follow-up assessment for all of the measures except the CHQ score.
Although no significant differences were observed between the
pre-test and post-test in beliefs about treatment for the depression
scenario, professional consensus for the schizophrenia scenario, or
mental health first aid action, these measures showed significant
improvement at the six-month follow-up assessment.
In terms of mental health first-aid action, 125 participants (86
[68.8%] in the experimental group and 39 [31.2%] in the control
group) reported having had contact with individuals with mental
health problems within six months of receiving the MHFA training.
Of these students, only ten (four and six students in the experimental and control groups, respectively) revealed that they had
not offered any help or support to these individuals. The ALGEE
skills commonly used by the study participants included listening
to the individual’s problems in a non-judgemental manner (80 in
the experimental group; 36 in the control group), encouraging the
individual to discuss their concerns (80 in the experimental group;
35 in the control group), encouraging the individual to seek support
from their relatives (56 in the experimental group; 21 in the control
group) and professionals (51 in the experimental group; 17 in the
control group), and accompanying the individual to seek help and
advice (30 in the experimental group; 5 in the control group).
4. Discussion
The results of this study indicate the effectiveness of MHFA
training for undergraduate nursing students in terms of enhancing
their mental health literacy, confidence in helping others, mental
health first-aid action and mental wellbeing, and reducing their
social distance from people with mental disorders. This face-to-face
MHFA training course, which comprised small-group interactive
teaching and learning activities, aroused the participants’ learning interest and active involvement and enhanced their knowledge
and practical skills, which echoed the findings of previous studies
(Hung et al., 2019; Kelly & Birks, 2017). The feedback from a pilot
randomised controlled e-learning MHFA study of UK medical students expressed difficulties in practising new skills via eLearning
(Davies et al., 2018); however, another study recommended that a
blended strategy that combined face-to-face training with online
learning might be more favourable and effective than an approach
based only on face-to-face or e-learning (Bond et al., 2015; Liu et al.,
2016).
4.1. Mental health literacy
Notably, this study demonstrated the ability of MHFA training to
increase the mental health knowledge of the participating nursing
students, as evidenced by the increased recognition of mental illness (depression and schizophrenia) and improved understanding
of the roles of relevant professionals, medication, and treatment
available to assist people with mental health problems. The greatest gain was observed in the recognition of mental illness. The
improvement was expected because the MHFA programme had
been specifically developed to enhance the recipients’ recognition
of mental illness.
These results are consistent with those of local and foreign
studies conducted with healthcare students (Bond et al., 2015;
Burns et al., 2017; Davies et al., 2018; Hung et al., 2019; Kelly &
Birks, 2017, with the Hong Kong public (Wong et al., 2017) and in
Chinese-speaking communities in Australia (Lam et al., 2010). The
programme also emphasised the importance of early treatment and
clearly delineated the roles of various healthcare professionals in
treating people with mental disorders (Wong et al., 2017). However, in line with other studies (Burns et al., 2017; Jorm, Kitchener,
Sawyer, Scales, & Cvetkovski, 2010), a slight decrease in knowledge
of treatments for mental illness and helpfulness from different professionals was observed in the post-test compared to the six-month
follow-up assessment. This attenuation reflects the need for regular refreshment or further professional development to reinforce
the learned knowledge.
Mental illnesses present substantial challenges to healthcare
professionals worldwide, especially in general hospital settings in
which patients often present with mental and physical comorbidities (Vasiliadis, Lamoureux-Lamarche, & Guerra, 2017). Although
Please cite this article in press as: Hung, M. S. Y., et al. Effectiveness of the Mental Health First Aid programme for general nursing
students in Hong Kong: A randomised controlled trial. Collegian (2020), https://doi.org/10.1016/j.colegn.2020.04.006
G Model
ARTICLE IN PRESS
COLEGN-700; No. of Pages 8
Pre–follow- up
Cohen’s d
−1.46**
−0.10
0.05**
0.78**
−1.28**
−0.03*
−0.09
0.92**
−0.28**
−1.32**
0.48
Pre– Post
Cohen’s d
−1.10**
−0.26
−0.17
0.88**
−1.44**
−0.25
−0.21
1.15**
−0.34*
0.06
0.76**
M.S.Y. Hung et al. / Collegian xxx (2020) xxx–xxx
5
MHFA training is not intended to replace mental health nursing
education in the curriculum (Kelly & Birks, 2017; Kitchener & Jorm,
2017), the general nursing students in this study were expected to
use mental health first-aid skills after their training to provide assistance and to refer people efficiently and effectively to mental health
professionals and local resources. The valuable and relevant information provided in the MHFA programme not only enhanced the
mental health knowledge and practical skills of the general nursing students but also helped to consolidate what they had learned
during their general nursing training (Hung et al., 2019).
0.04**
0.08**
0.15**
7.53
5.92
31.69
Repeated-measures analysis of variance with Greenhouse-Geisser and post hoc Bonferroni correction were performed.
SD, standard deviation.
*
p < 0.05.
**
p < 0.01.
3.20 (0.76)
3.87 (0.95)
2.21 (0.47)
3.11 (0.68)
4.34 (2.20)
1.82 (0.43)
2.94 (0.80)
2.17 (2.67)
2.09 (0.49)
3.17 (0.75)
1.18 (2.04)
1.72 (0.36)
2.89 (0.85)
1.33 (2.35)
2.04 (0.47)
Confidence in ability to help
Mental health first aid action
Chinese Health Questionnaire
2.94 (0.87)
2.23 (2.70)
2.10 (0.42)
0.16**
0.03**
0.04**
0.31**
33.94
6.28
7.50
81.35
2.34 (0.58)
4.25 (1.93)
5.31 (2.37)
2.88 (0.63)
2.87 (0.33)
4.89 (1.82)
6.08 (2.33)
2.21 (0.56)
2.32 (0.46)
4.69 (1.50)
5.49 (1.87)
2.85 (0.59)
2.91 (0.28)
5.23 (1.69)
6.33 (2.25)
2.07 (0.59)
2.35 (0.47)
4.84 (1.30)
5.89 (1.79)
2.71 (0.52)
Schizophrenia scenario
Recognition of schizophrenia
Professional consensus
Beliefs about treatment
Social distance
2.24 (0.46)
4.89 (1.23)
5.93 (1.52)
2.72 (0.56)
36.56
3.82
4.28
47.01
2.29 (0.57)
4.75 (1.68)
5.43 (1.99)
2.13 (0.54)
2.82 (0.41)
5.08 (1.75)
5.97 (2.17)
1.74 (0.54)
2.25 (0.55)
5.04 (1.13)
5.71 (1.40)
2.25 (0.54)
2.71 (0.48)
5.30 (1.65)
6.35 (2.12)
1.70 (0.51)
2.15 (0.45)
5.12 (1.04)
6.03 (1.20)
2.15 (0.47)
2.21 (0.42)
4.93 (1.13)
6.06 (1.13)
2.16 (0.53)
Depression scenario
Recognition of depression
Professional consensus
Beliefs about treatment
Social distance
Mean (SD)
Mean (SD)
Mean (SD)
Mean (SD)
Mean (SD)
Experimental
Experimental
Mean (SD)
Experimental
Post-test
Control
Pre-test
Table 2
Comparison of various measurements at pre-test, post-test and follow-up
Control
Follow-up
Control
F (time*group
effect)
0.17**
0.02*
0.02*
0.20**
Partial Eta Sq.
4.2. Social distance
Several studies have reported that insufficient mental health
related knowledge among healthcare professionals and students
might lead to higher levels of stigma, prejudice and social distance
when these individuals face mentally unwell patients (Bingham
& O’Brien, 2018; Chernomas & Shapiro, 2013; Fokuo et al., 2017;
Henderson et al., 2014; Giandinoto et al., 2018; Knaak et al., 2017;
Mak et al., 2015). An Australian survey that compared the attitudes of healthcare professionals, such as psychiatrists, clinical
psychologists, and general practitioners, with those of the general community, found that women and general practitioners were
prone to have more personal stigmatising attitudes about mentally ill patients, and maintain greater social distance from these
individuals, than men and psychologists, respectively (Reavley,
Mackinnon, Morgan, & Jorm, 2014). A local study of healthcare
professionals and students in Hong Kong found that doctors maintained the greatest social distance and social workers maintained
the smallest social distance from individuals with mental illness,
whereas nursing students maintained less social distance from
individuals with bipolar disorder or schizophrenia than social
work students (Mak et al., 2015). Despite the differences in the
magnitudes of these attitudes, all likely hinder the professional’s
motivation and ability to provide quality holistic care. It is therefore the development and implementation of relevant strategies
to reduce stigma and social distance in the healthcare system are
significant (Knaak et al., 2017).
In line with other results, this study demonstrated that MHFA
training could decrease the perceived social distance among nursing student participants (Bond et al., 2015; Burns et al., 2017; Hung
et al., 2019). In a recent qualitative study that explored nursing
students’ experiences three months after MHFA training, the participants reported that the MHFA programme had improved their
beliefs and values (Hung et al., 2019). Specifically, they had acquired
the belief that nurses should present a neutral, therapeutic attitude towards and become more accepting of people with mental
illness. This demonstrates that improvements in mental health literacy can help these students to normalise mental health problems
and dispel the myths that surround these problems, thus decreasing stigmatisation and social distancing from people with mental
illness.
However, the stigma of mental illness, including self-stigma
as well as family and public stigma, is deeply rooted in Chinese
communities, and those with mental disorders are perceived as
dangerous, impulsive and uncontrollable (Lam et al., 2010). Given
this cultural background, the individuals and their families are often
criticised and blamed because they are believed to be responsible
for the illness. This moralising view further frustrates the individuals and their families, who often already experience feelings of
shame or guilt when seeking help and treatment, leading to greater
social distancing from others (Lam et al., 2010). Thus, local government and healthcare organiszations are optimally positioned to
support these families and to promote compassionate and altruistic attitudes and behaviours towards those in society suffering from
mental illness.
Please cite this article in press as: Hung, M. S. Y., et al. Effectiveness of the Mental Health First Aid programme for general nursing
students in Hong Kong: A randomised controlled trial. Collegian (2020), https://doi.org/10.1016/j.colegn.2020.04.006
G Model
COLEGN-700; No. of Pages 8
ARTICLE IN PRESS
M.S.Y. Hung et al. / Collegian xxx (2020) xxx–xxx
6
4.3. Confidence in the ability to help and mental health first aid
action
Consistent with the findings of previous studies, the MHFA intervention was effective in increasing medical and nursing students’
confidence in providing help and support to people with mental
problems (Bond et al., 2015; Burns et al., 2017; Kitchener & Jorm,
2002; Hadlaczky, Hökby, Mkrtchian, Carli, & Wasserman, 2014).
The student participants acquired a greater understanding of mental disorders, including of the associated signs, symptoms, and
crises, which may have reduced the stigma and social distance and
increased their confidence in communicating and assisting individuals with mental disorders.
It was clear that the participants in the experimental group
were more willing than those in the control group to identify and
offer help to those in need, which is in agreement with the finding of another study that the participants became more competent
in identifying ways to help (Hung et al., 2019). The participants
learned basic ALGEE engagement skills during training via interactive teaching and learning activities, such as role-playing. A possible
explanation for this observation may be that the participants were
future ‘helping professionals’ and were therefore more inclined to
feel compassion and care for others. In addition, nearly half of the
study’s participants (173; 48.3%) reported that a friend or relative had encountered mental health problems. The skills they had
learned from these experiences had likely bolstered their readiness to encounter and interact helpfully with individuals in society
or patients in their future workplace suffering from mental health
problems.
Furthermore, it was notable that an enhanced sense of achievement and satisfaction was reported by the nursing students after
practising their skills and providing help for their peers or relatives
(Hung et al., 2019). They found that performing MHFA for family
members and friends was a positive and valuable experience. The
members of other helping professions, such as teachers, have also
reported gaining confidence in their ability to assist students and
colleagues after completing the MHFA course (Jorm et al., 2010).
4.4. Students’ mental health
Evidence suggests that healthcare students both locally and
globally often experience high levels of mental distress (Cheung
et al., 2016; He et al., 2018; Sun et al., 2016) and may have suicidal ideation due to the high academic and social demands of
their professional training (Mospan et al., 2017; Rotenstein et al.,
2016). In Hong Kong, 20.0%, 24.3%, and 39.9% of nursing students
reported modest to exceptionally severe levels of stress, depression, and anxiety, respectively (Cheung et al., 2016), and the most
common stressor was a lack of professional knowledge and skills
(Chan, So, & Fong, 2009). Similarly, more than one-third of participants in this study (127; 35.5%) had experienced problems with
their own mental health.
The initial evidence from this study, gleaned from repeated
use of the CHQ, demonstrated that the MHFA training improved
the student participants’ mental health as a secondary effect that
was maintained for six months. These results are consistent with
international evaluations of MHFA training, which have reported
its positive effects on the mental health of teenagers and medical and nursing students (Bond et al., 2015; Davies et al., 2018)
and its potential to intensify nursing students’ self-awareness of
their personal mental health (Hung et al., 2019). The student participants mentioned that they encountered various kinds of stressors
each day, including study workload and peer relationships. After
the MHFA training, they were more conscious of and attentive to
their own mental status when handling such life challenges.
In Hong Kong, general nursing and psychiatric nursing are
the two predominant registration streams in the field. This study
mainly targeted general nursing students, who receive limited theoretical mental health education (a minimum of 40 hours) and
clinical practicum (a minimum of 60 hours) during their nursing
training (The Nursing Council of Hong Kong, 2017). Although recent
studies have aroused concerns and discussion regarding the role of
MHFA training in Australian undergraduate mental health nursing
education (Happell & McAllister, 2015; Kitchener & Jorm, 2017),
the advantages of MHFA for local general nursing students cannot be underestimated. MHFA training is not intended to replace
mental health education in the pre-registration nursing curriculum
(Happell & McAllister, 2015; Kitchener & Jorm, 2017), but it clearly
can enrich the mental-health knowledge of future general nurses,
and thereby help them provide holistic care for patients with physical and mental comorbidities in Hong Kong. In addition, we suggest
that MHFA could be promoted as a compulsory subject for junior
healthcare students, who lack sophistication in caring for people
with mental health issues.
In recent years, the WHO has globally promoted the importance of mental health in response to an increase in the
prevalence of mental disorders, especially in youngsters. Local
governments and non-governmental organisations have organised various health promotion and disease prevention programmes
that target mental health awareness, stigma, suicide prevention, and school-based mental health promotion (World Health
Organization, 2017). Thus, MHFA training could be introduced as an
elective subject or extracurricular activity for all students at local
colleges/universities, to improve their mental wellbeing and ability to cope with the stressors of academic life. Furthermore, tertiary
institutions should review their existing curricula and provide adequate rest and leisure time with social and extracurricular activities
for students, to further enhance their students’ physical and mental
health (Legislative Council, HKSAR, 2017).
According to a local government report, adolescents with poor
mental health may achieve a low level of academic success, leading
to poor employment and social adaptation (Food & Health Bureau,
HKSAR, 2017). It is therefore in the interests of educational institutions to organise informative and beneficial activities, including
mental health seminars/workshops and counselling sessions, to
nurture an encouraging atmosphere and emphasise the significance of an environment that supports mental health, and enhances
their students’ academic achievement (Legislative Council, HKSAR,
2017).
5. Conclusions
In this study, we aimed to determine the effectiveness of MHFA
training and explore the mental wellbeing of university-level general nursing students. Similar to other recent studies, this study
confirmed the benefits of the MHFA programme for healthcare students in Hong Kong and other countries. The students’ awareness
of their own mental wellbeing was also enhanced after training.
We therefore suggest that the local promotion of MHFA training
will empower adolescents to develop good mental health status
and mental wellbeing, to optimise their potential and strengthen
their ability to overcome later hardships in life.
6. Implications
The evidence from this study shows that MHFA training
should be promoted and integrated as compulsory training and
as part of extracurricular activities for healthcare and general
college/university students locally and globally. It would also be
Please cite this article in press as: Hung, M. S. Y., et al. Effectiveness of the Mental Health First Aid programme for general nursing
students in Hong Kong: A randomised controlled trial. Collegian (2020), https://doi.org/10.1016/j.colegn.2020.04.006
G Model
COLEGN-700; No. of Pages 8
ARTICLE IN PRESS
M.S.Y. Hung et al. / Collegian xxx (2020) xxx–xxx
interesting and important to evaluate the effects of MHFA training
on the mental wellbeing of general university/college students.
Further qualitative research should be conducted to determine
in greater detail how the experience and application of MHFA provide an in-depth understanding of the best ways to assist people
with mental health problems. In addition, MHFA training workshops are suggested for existing healthcare professionals who have
not received mental health related training.
7. Limitations
Finally, several important limitations must be considered. First,
the study population is not representative of the general population, as general nursing students usually possess better healthcare
knowledge than other students at university do. Second, some
may have had prior theoretical and practical mental health nursing
inputs, because the students were recruited from different years of
undergraduate general nursing programmes. It may thus be wise
to invite junior year students without prior training to take part in
a future study. Third, the study had a higher sex ratio (63 [17.6%]
male, 295 [82.4%] female) than the active registered nurses practising locally (1505 [12.2%] male, 10,812 [87.8%] female) (Department
of Health, HKSAR, 2015). However, in terms of the effects of sex
and previous training on mental health, no significant differences
were found between the experimental and control groups. All the
nursing students were recruited from a single tertiary institution,
so study contamination may have occurred, although the students
were also instructed to avoid disclosing any study information to
their peers.
Since 2005, the Mental Health Association of Hong Kong was
authorised to develop the local version of MHFA Manual and to
translate the manual based on the Australia MHFA training programme (The Mental Health Association of Hong Kong, 2015).
However, a minor limitation is that the validity and reliability of
the translated training manual had not been conducted.
Funding
The study was fully supported by the Competitive Research
Funding Schemes for Local Self-financing Degree Sector, Research
Grants Council of the Hong Kong Special Administrative Region,
China (<GN1>UGC/FDS17/H01/16<GN1>).
Ethical statement
The study involved human research and obtained ethical
approval from the Tung Wah College’s Research Ethics SubCommittee.
Conflict of interest
This manuscript is the authors’ original work. It has not been
published and is not under consideration for publication elsewhere.
The authors have accepted the copyright terms and conditions of
Elsevier and the Australian College of Nursing. We had no initial
contacts with Collegian regarding the manuscript. We have no conflicts of interest to disclose. We have no particular preferences
regarding the selection of academic editors and reviewers to handle
this manuscript.
Author contribution
Conceptualization and design of the study, M.S.Y.H., M.C.M.C.,
W.T.C., P.Y.K.W.; data curation, M.S.Y.H.; literature review and
manuscript draft writing, M.S.Y.H.; manuscript review and editing,
7
M.C.M.C., W.T.C., P.Y.K.W.; funding acquisition, M.S.Y.H., M.C.M.C.,
W.T.C., P.Y.K.W. All authors have read and agreed to the published
version of the manuscript.
Acknowledgements
The authors would like to thank 1) all of the nursing students
who participated in this study; 2) the Mental Health Association of
Hong Kong for approving the use of the Mental Health Literacy Scale
(Chinese version); and 3) Dr Cheng, Andrew Tai Ann for approving
the use of the CHQ.
References
Bingham, H., & O’Brien, A. J. (2018). Educational intervention to decrease
stigmatizing attitudes of undergraduate nurses towards people with mental
illness. International Journal of Mental Health Nursing, 27(1), 311–319.
Bond, K. S., Jorm, A. F., Kitchener, B. A., & Reavley, N. J. (2015). Mental health first
aid training for Australian medical and nursing students: An evaluation study.
BMC Psychology, 3(1), 11.
Burns, S., Crawford, G., Hallett, J., Hunt, K., Chih, H. J., & Tilley, M. P. J. (2017).
What’s wrong with John? A randomised controlled trial of Mental Health First
Aid (MHFA) training with nursing students. BMC Psychiatry, http://dx.doi.org/
10.1186/s12888-017-1278-2, 171-12
Chan, C. K., So, W. K., & Fong, D. Y. (2009). Hong Kong baccalaureate nursing
students’ stress and their coping strategies in clinical practice. Journal of
Professional Nursing, 25(5), 307–313.
Cheng, T. A., Wu, J. T., Chong, M. Y., & Williams, P. (1990). Internal consistency and
factor structure of the Chinese Health questionnaire. Acta Psychiatrica
Scandinavica, 82(4) http://dx.doi.org/10.1111/j.1600-0447.1990.tb01389.x
Chernomas, W. M., & Shapiro, C. (2013). Stress, depression, and anxiety among
undergraduate nursing students. International Journal of Nursing Education
Scholarship, 10(1), 255–266.
Cheung, T., Wong, S., Wong, K., Law, L., Ng, K., Tong, M., . . . & Yip, P. (2016).
Depression, anxiety and symptoms of stress among baccalaureate nursing
students in Hong Kong: A cross-sectional study. International Journal of
Environmental Research and Public Health, 13(8), 779.
Cunningham, J. B., & McCrum-Gardner, E. (2007). Power, effect and sample size
using GPower: Practical issues for researchers and members of research ethics
committees. Evidence-Based Midwifery, 5(4), 132–137.
Davies, E. B., Beever, E., & Glazebrook, C. (2018). A pilot randomised controlled
study of the mental health first aid eLearning course with UK medical students.
BMC Medical Education, 18(1), 45.
Department of Health, HKSAR. (2015). 2013 health manpower survey. Summary of
the characteristics of registered nurse enumerated 27 January. https://www.dh.
gov.hk/english/statistics/statistics hms/sumrn13.html
Eisenberg, D., Hunt, J., & Speer, N. (2012). Help seeking for mental health on college
campuses: Review of evidence and next steps for research and practice.
Harvard Review of Psychiatry, 20(4), 222–232.
Fokuo, J. K., Goldrick, V., Rossetti, J., Wahlstrom, C., Kocurek, C., Larson, J., &
Corrigan, P. (2017). Decreasing the stigma of mental illness through a
student-nurse mentoring program: A qualitative study. Community Mental
Health Journal, 53(3), 257–265.
Food and Health Bureau, HKSAR. (2017). Mental health review report. https://www.
fhb.gov.hk/download/press and publications/otherinfo/180500 mhr/e mhr
full report.pdf
Giandinoto, J. A., Stephenson, J., & Edward, K. L. (2018). General hospital health
professionals’ attitudes and perceived dangerousness towards patients with
comorbid mental and physical health conditions: Systematic review and
meta-analysis. International Journal of Mental Health Nursing, 27(3), 942–955.
Hadlaczky, G., Hökby, S., Mkrtchian, A., Carli, V., & Wasserman, D. (2014). Mental
Health First Aid is an effective public health intervention for improving
knowledge, attitudes, and behaviour: A meta-analysis. International Review of
Psychiatry, 26(4), 467–475.
Happell, B., & McAllister, M. (2015). The challenges of undergraduate mental
health nursing education from the perspectives of heads of schools of nursing
in Queensland, Australia. Collegian, 22(3), 267–274.
He, F. X., Turnbull, B., Kirshbaum, M. N., Phillips, B., & Klainin-Yobas, P. (2018).
Assessing stress, protective factors and psychological well-being among
undergraduate nursing students. Nurse Education Today, 68, 4–12.
Henderson, C., Noblett, J., Parke, H., Clement, S., Caffrey, A., Gale-Grant, O., . . . &
Thornicroft, G. (2014). Mental health-related stigma in health care and mental
health-care settings. The Lancet Psychiatry, 1(6), 467–482.
Hung, M. S., Lam, S. K., & Chow, M. C. (2019). Nursing students’ experiences of
mental health first aid training: A qualitative descriptive study. Collegian,
26(5), 534–540. http://dx.doi.org/10.1016/j.colegn.2019.02.006
IsHak, W., Nikravesh, R., Lederer, S., Perry, R., Ogunyemi, D., & Bernstein, C. (2013).
Burnout in medical students: A systematic review. The Clinical Teacher, 10(4),
242–245.
Jorm, A. F., Kitchener, B. A., Sawyer, M. G., Scales, H., & Cvetkovski, S. (2010). Mental
health first aid training for high school teachers: A cluster randomized trial.
BMC Psychiatry, 10(1), 51.
Please cite this article in press as: Hung, M. S. Y., et al. Effectiveness of the Mental Health First Aid programme for general nursing
students in Hong Kong: A randomised controlled trial. Collegian (2020), https://doi.org/10.1016/j.colegn.2020.04.006
G Model
COLEGN-700; No. of Pages 8
8
ARTICLE IN PRESS
M.S.Y. Hung et al. / Collegian xxx (2020) xxx–xxx
Kelly, C. M., Mithen, J. M., Fischer, J. A., Kitchener, B. A., Jorm, A. F., Lowe, A., &
Scanlan, C. (2011). Youth mental health first aid: A description of the
programme and an initial evaluation. International Journal of Mental Health
Systems, 5(4), 1–9.
Kelly, J., & Birks, M. (2017). ‘It’s the simple things you do first that start the process
of help’: Undergraduate nursing and midwifery students’ experiences of the
Mental Health First Aid course. Collegian, 24(3), 275–280.
Knaak, S., Mantler, E., & Szeto, A. (2017). Mental illness-related stigma in healthcare:
Barriers to access and care and evidence-based solutions. pp. 111–116. Healthcare
Management Forum (Vol. 30) Sage CA: Los Angeles, CA: SAGE Publications. No.
2.
Kitchener, B. A., & Jorm, A. F. (2002). Mental health first aid training for the public:
Evaluation of effects on knowledge, attitudes and helping behavior. BMC
Psychiatry, 2(1), 10.
Kitchener, B. A., & Jorm, A. F. (2017). The role of Mental Health First Aid training in
nursing education: A response to Happell, Wilson & McNamara (2015).
Collegian, 24(3), 313–315.
Kitchener, B. A., Jorm, A. F., & Kelly, C. (2017). Mental health first aid manual (4th
edn). Parkville, Victoria: Mental Health First Aid Australia.
Lam, C. S., Tsang, H. W., Corrigan, P. W., Lee, Y. T., Angell, B., Shi, K., . . . & Larson, J. E.
(2010). Chinese lay theory and mental illness stigma: Implications for research
and practices. Journal of Rehabilitation, 76(1), 35.
Lee, S. J., Lee, J. E., & Lee, M. K. (2018). The influence of clinical practicum-related
stress, and awareness, and self-efficacy on strength on health promotion
behaviors among nursing students. Journal of Korean Academic Society of
Nursing Education, 24(2), 160–167.
Legislative Council, HKSAR. (2017). Legislative council panel on health services.
Panel on Education and Panel on Welfare Services (Joint Meeting). Mental Health of
Adolescents. LC Paper No. CB(2)512/17-18(01), 20 December. https://www.legco.
gov.hk/yr17-18/english/panels/ws/papers/edhsws20171220cb2-512-1-e.pdf
Liu, Q., Peng, W., Zhang, F., Hu, R., Li, Y., & Yan, W. (2016). The effectiveness of
blended learning in health professions: Systematic review and meta-analysis.
Journal of Medical Internet Research, 18(1), e2.
Mak, W. W. S., Cheung, F. M. C., Wong, S. Y. S., Tang, W. K., Lau, J. T. F., Woo, J., & Lee,
D. T. F. (2015). Stigma towards people with psychiatric disorders. Hong Kong
Medical Journal, 21(2 Supplement 2).
Morgan, A. J., Ross, A., & Reavley, N. J. (2018). Systematic review and meta-analysis
of Mental Health First Aid training: Effects on knowledge, stigma, and helping
behaviour. PloS One, 13(5), Article e0197102.
Mospan, C. M., Hess, R., Blackwelder, R., Grover, S., & Dula, C. (2017). A two-year
review of suicide ideation assessments among medical, nursing, and pharmacy
students. Journal of Interprofessional Care, 31(4), 537–539.
Reavley, N. J., Mackinnon, A. J., Morgan, A. J., & Jorm, A. F. (2014). Stigmatising
attitudes towards people with mental disorders: A comparison of Australian
health professionals with the general community. Australian & New Zealand
Journal of Psychiatry, 48(5), 433–441.
Rotenstein, L. S., Ramos, M. A., Torre, M., Segal, J. B., Peluso, M. J., Guille, C., . . . &
Mata, D. A. (2016). Prevalence of depression, depressive symptoms, and
suicidal ideation among medical students: A systematic review and
meta-analysis. JAMA, 316(21), 2214–2236.
Sun, F. K., Long, A., Tseng, Y. S., Huang, H. M., You, J. H., & Chiang, C. Y. (2016).
Undergraduate student nurses’ lived experiences of anxiety during their first
clinical practicum: A phenomenological study. Nurse Education Today, 37,
21–26.
Svensson, B., & Hansson, L. (2014). Effectiveness of mental health first aid training
in Sweden. A randomized controlled trial with a six-month and two-year
follow-up. PLoS One, 9(6), Article e100911.
The Mental Health Association of Hong Kong. (2015). Mental Health First Aid
Manual (Hong Kong) (3th edn). The Mental Health Association of Hong Kong.
The Nursing Council of Hong Kong. (2017). Handbook for Accreditation of Training
Institutions for Pre-Enrolment/Pre-Registration Nursing Education. https://www.
nchk.org.hk/filemanager/en/pdf/Accreditation Handbook.pdf
Vasiliadis, H. M., Lamoureux-Lamarche, C., & Guerra, S. G. (2017). Gender and age
group differences in suicide risk associated with co-morbid physical and
psychiatric disorders in older adults. International Psychogeriatrics, 29(2),
249–257.
Wong, F. K. D., Lam, Y. K. A., & Poon, A. (2010). Depression literacy among
Australians of Chinese-speaking background in Melbourne, Australia. BMC
Psychiatry, 10, 7. http://dx.doi.org/10.1186/1471-244X-10-7
Wong, D. K., Lau, Y., Kwok, S., Wong, P., & Tori, C. (2017). Evaluating the
effectiveness of Mental Health First Aid program for Chinese people in Hong
Kong. Research on Social Work Practice, 27(1), 59–67. http://dx.doi.org/10.1177/
1049731515585149
World Health Organization. (2017). Mental Atlas 2017 Retrieved from The World
Health Organization, Web site:. http://apps.who.int/iris/bitstream/handle/
10665/272735/9789241514019-eng.pdf?ua=1
Maria S.Y. Hung, RN, DHSc, is an associate professor in the School of Nursing at Tung
Wah College, Kowloon, Hong Kong.
Meyrick C.M. Chow, RN, PhD, is a professor in the School of Nursing at Tung Wah
College, Kowloon, Hong Kong.
Wai Tong Chien, RN(Psy), PhD, is a professor in the Nethersole School of Nursing at
The Chinese University of Hong Kong.
Prudence Y.K. Wong, RSW, BSW, is the co-ordinator of Mental Health First Aid
programme of the Mental Health Association of Hong Kong.
Please cite this article in press as: Hung, M. S. Y., et al. Effectiveness of the Mental Health First Aid programme for general nursing
students in Hong Kong: A randomised controlled trial. Collegian (2020), https://doi.org/10.1016/j.colegn.2020.04.006