Federal Democratic Republic of Ethiopia

Ministry of Health

Health Professionals’ Competency

Assessment and Licensure Directorate

Exam Administration Manual

March 2019
i
Message from the Director
As compared to the WHO standard of health professionals to population ratio for developing
countries, Ethiopia has wide gap yet and the government has been working to increase the
number of health professionals by increasing the intake of health science colleges and by
opening new institutions.

While this helped to reduce the gap we have compared to the standard, the quality of the teaching
and learning process hence, the service provided by the professionals became an issue for
different stakeholders and the government as well. For this reason and since licensure exams are
widely practiced internationally in different countries to insure the quality of health service, the
federal ministry of health took the initiative to launch licensure exam initiative within human
resource directorate since July 2015. Since then two round pilot tests given and reports discussed
with different stakeholders.

The initiative currently led by a directorate established for this purpose, which is composed of
three case teams and has been doing different activities to strengthen the system and capacitate
the directorate to implement the exam in wider range with a much better organization and
quality. As part of this, different documents were prepared to help facilitate the exam process
and create transparency and sustainability of the program.

In the preparation of this document and other manuals, different examining institutions and
countries experiences were reviewed, highly valued experts from different higher teaching
institutions and Jhpiego- Ethiopia were involved for whom my deep appreciation goes for and
for the high commitment and hardworking of the staffs at the federal ministry of health.

i
To this end, I would like to call for a collaborative work of all stakeholders in the health sector to
this initiative, which indeed has high contribution to quality assurance of the health service
delivery and yet cannot be effective unless all relevant bodies put their hands together for its
implementation and continuity.

Dr. Ruth Nigatu

Health Professionals’ Competency
Assessment and Licensure Directorate,
Director

ii
Acknowledgement
The Health Professionals’ Competency Assessment and Licensing Directorate (HPCALD)
would like to acknowledge Ministry of Health and JHPIEGO Ethiopia for providing
technical and fiscal support for the preparation of this guideline. Finally, the directorate
needs to appreciate the experts participated on the preparation of this guideline.

List of Contributors:

Mr. Alebachew Demelash Debre Birhan University

Ms. Aregash Molla Ministry of Health
Mr. Belayneh Abie Ministry of Health
Mr. Eden Workneh Ministry of Health
Mr. Fikadie Dagnaw Ministry of Health
Mr. Ibrahim Yimer Ethiopian Midwives Association
Mr. Kiflemariam Tsegaye Ministry of Health
Mr. Shewatatek Gedamu Jimma University
Mr. Tesfamariam Shimekit Education Strategy Center (ESC)
Ms. Betelhem Bizuayehu Ministry of Health
Ms. Tangut Dagnew Ministry of Health
Ms. Yeshiwork Eshetu Ministry of Health

iii
Table of contents

Acknowledgement ......................................................................................................................................... i

List of Contributors: ..................................................................................................................................... iii

Table of contents ......................................................................................................................................... iv

1. Introduction .......................................................................................................................................... 1

2. Purpose of the Manual ......................................................................................................................... 3

3. Principles of Exam Administration ........................................................................................................ 3

4. Flow chart ............................................................................................................................................. 4

5. Pre-exam Administration Activities ...................................................................................................... 5

5.1 Printing Examination Materials .......................................................................................................... 5

5.2 Forms used for exam administration process ...................................................................................... 6

5.3 Computer based Testing (CBT) .......................................................................................................... 7

5.4 Special Accommodations Policy and Process..................................................................................... 8

5.5 Safety Measures/security issues .......................................................................................................... 8

5.6 Breach of Security............................................................................................................................. 10

5.7 Recruitment and deployment of supervisors and examiners............................................................. 10

5.8 Orientation of supervisors and examiners ......................................................................................... 11

5.9 Orientation of supervisors and examiner about dressing code.......................................................... 12

5.10. OSCE exam station organizer preparation ..................................................................................... 12

5.10.1 Recruiting station organizers ................................................................................................... 13

5.10.2 Training of station organizer .................................................................................................... 13

5.10.3 Examination day briefings ....................................................................................................... 13

5.10.4 Standardized patient briefing ................................................................................................... 14

iv
 5.10.5 Candidates briefing .................................................................................................................. 14

5.10.6 Examiners briefing ................................................................................................................... 14

5.10.7 Before the day of the Exam...................................................................................................... 14

6. During exam administration ................................................................................................................... 15

6.1 MCQ exam administration ................................................................................................................ 15

6.1.1 Assessing the Seating Arrangement ........................................................................................... 16

6.1.2 Meeting Candidates.................................................................................................................... 16

6.1.3 Distributing exam materials and filling the answer sheet .......................................................... 17

6.1.4 Managing cheating ..................................................................................................................... 18

6.1.5 Handling the unexpected............................................................................................................ 20

6.1.6 Managing the answer sheet ........................................................................................................ 20

6.2 Computer based exam administration ............................................................................................... 20

6.2.1 Before exam starting time, the following should be done: ........................................................ 20

6.2.2 During running of the computer based exam ............................................................................. 20

6.3 During OSCE Administration ........................................................................................................... 21

6.3.1 Setting up the OSCE circuit and Station preparation on the day of the Exam ........................... 21

6.3.2 Running the circuit ..................................................................................................................... 22

6.3.3 Arrange refreshment facilities ................................................................................................. 22

6.3.4 Quarantine ........................................................................................................................... 23

6.3.5 Candidate misbehavior during OSCE ................................................................................. 23

6.3.6 Appealing Process during the Examination (MCQ and OSCE) ......................................... 23

7. Post-exam administration Procedures................................................................................................ 24

7.1. Handling Exam Materials ........................................................................................................... 24

7.2. Scoring Process ........................................................................................................................... 25

7.3. Exam Analysis ............................................................................................................................ 25

v
 7.3.1. Post-Exam Analysis for Written Exam ............................................................................... 26

7.3.2. Post Exam Analysis for OSCE............................................................................................ 27

8. HPCALD examinee result ratification .................................................................................................. 28

9. Complaints and appeals ...................................................................................................................... 28

10. Feedback and reporting .................................................................................................................. 29

10.1. Feedback Mechanism .............................................................................................................. 29

11. Discarding exam materials .............................................................................................................. 30

12. Quality Assurance ........................................................................................................................... 30

12.1. Pre- exam quality Assurance................................................................................................... 31

12.2. During - exam quality Assurance............................................................................................ 32

12.3. Post- exam quality Assurance ................................................................................................. 32

13. Roles and responsibilities of Exam Team ........................................................................................ 33

Annexes ....................................................................................................................................................... 37

Annex I. Candidate Comment Form ....................................................................................................... 37

Annex II: Incident report form ................................................................................................................ 38

Annex III. Test Security Agreement ....................................................................................................... 40

Annex IV: Candidate’s complaints and/appeals writing format ............................................................. 41

Annex V: Single station OSCE Quality assurance checklist (To be filled by QA team).......................... 1

Annex VI: Pre-OSCE Quality assurance checklist (To be filled by QA team) ........................................ 2

Annex VII: Pre-Written Examination Quality assurance checklist (To be filled by QA team) ................ 5

Annex VIII: During OSCE Quality assurance checklist (To be filled by QA team) ................................ 7

Annex IX: During-Written Examination Quality assurance checklist (To be filled by QA team) ........... 8

Annex X: Post-OSCE Quality assurance checklist (To be filled by QA team) ........................................ 9

Annex XI: Post-Written Examination Quality assurance checklist (To be filled by QA team) ............. 10

vi
Acronyms

CBT ----------------------Computer Based Testing

HERQA------------------Higher Education Relevance and Quality Agency
MOH---------------------Ministry of Health
MOU---------------------Memorandum of Understanding
MCQ ---------------------Multiple Choice Questions
MOE----------------------Ministry of Education
HPCALD ------------- Health Professionals’ Competency Assessment and Licensure Directorate
OSCE --------------------Objectively Structured Clinical Examination
QA ------------------------Quality assurance
SP -------------------------Standardized patient
WHO----------------------World Health Organization

vii
1. Introduction

The World Health Organization (WHO) asserted that no matter how many individuals are
educated and deployed; health professionals cannot transform population health unless they
have the necessary competencies. Hence, the WHO recommended changes in regulations
including certification and licensing of graduates. It is a fact that many countries around the
world including those in North America, Europe and Africa verify competence and fitness to
practice of health professionals by administering standard qualification or licensure
examinations.

The Government of Ethiopia recognizes that healthcare is one of the crucial components of
basic social services with direct linkage to growth and development of the country as well as
to the welfare of the society. The quality of health care delivered in a country has an
immediate and long-term impact on the quality of life, livelihood, morbidity and mortality of
its citizens and on the nation’s economy and its national security. Accordingly, the Ministry
of Health (MOH) is committed to reduce morbidity, mortality and disability and to improve
the health status of Ethiopian people by providing and regulating health services. Health
professionals’ competency assessment is among the approaches of regulating health service;
hence MOH designed health professional Competency assessment and licensure
examination.

The sole purpose of competency assessment is to identify health professionals who possess
the minimum basic knowledge and experience necessary to perform tasks on the job safely
and competently, but not to select the “top" candidates. It is also to validate safe and effective
medical practice, job-related skills and knowledge to provide an independent assessment and
documentation of competency. Therefore, competency assessment is very different from
academic or employment examination.

The health professionals’ licensing exam administration has two parts. The first part of the
examination deals with the administration of written exam followed by the second part, the
administration of skill assessment. Among the different knowledge and skills assessment

1
methods, Multiple Choice Questions (MCQ) and Objectively Structured Clinical
Examination (OSCE) are used in Ethiopian health professionals’ licensure examination.

Exam administration has different team members who function based upon their roles and
responsibilities vested to them. Members involving in the exam administration are central
exam overseer, supervisors, invigilators, coordinators and other support staff. The exam
administration can be viewed from the very preparation to actual administration and post-
exam activities.

It is imperative that licensing the professional competencies of health professionals is crucial

to have quality health education and service provision in the country. In addition, the
examination process must adhere with the principles of fairness and equity. Therefore, it is
necessary to have exam administration manual used to direct the health professionals’
licensure examination. Concerning to this, MOH developed this exam administration manual
to direct the exam administration process and lead the exam process by similar national
implementation standards i.e. to standardize Health Professionals’ Licensure Exam
Administration process.

2
2. Purpose of the Manual
• To guide the implementation of health professionals’ licensure exam administration
effectively.

3. Principles of Exam Administration

• Fairness
• Accountability
• Security
• Efficiency

3
 4. Flow chart
Over all exam administration process will be carried out based on the following flowchart.

•Recruitment, training and certification of examiners and supervisors

•Orientation of supervisors and examiners
•Signing contractual agreement between exam development and exam administration case team for
Capcity building of receiving the exam items
exam team

•Storage and coding of exam items at administration case team

•Printing and duplication of exam materials and forms
•Signing contractual agreement between exam administration case team and supervisors after counting and
Preparation of exam packing in front of them and receive exam materials and stamping the seal of NLED on its edge.
admin team at NLED

•Transportation of exam materials and forms to exam centers and signing contractual agreement between
supervisors and site coordinators (stake holders) and storage
•Ensuring the readiness of exam center
•Signing contractual agreement between site coordinators and supervisors and receive the exam materials
Transportation and and forms
pre eaxm admin •Signing contractual agreement between the supervisor and examiners

•Administering the exam

•Collecting the exam materials
•Signing contractual agreement between examiners and supervisor and receive exam materials and forms
Exam admin,post •Transportation of the exam materials and forms to the NLED, signing contractual agreement between
admin and supervisors and NLED administration case team
transportation

•Scanning the exam answer sheets to the computer

•Scoring and analysis
•Standard setting
Scoring and •Compiling and submitting to the ratification committee
publication •Publication of results

•Managing compliant and appeals in all steps and appeal committee in the directorate
•Reporting and feedback
•Quality assurance in all steps
Feed back and •Evaluation of the exam administration process
discarding exam
•Discarding the exam materials environmental friendly and securely
materials

4
5. Pre-exam Administration Activities
The pre-examination activities are crucial for subsequent accomplishment of the entire exam
administration. The preparatory activities could take months for some activities (like
scheduling exam) or lesser for some other activities based on the nature of the task. Activities
that should be done before actual administration of the exam are listed below.

• Receive list of exam candidates from the registrar

• Select exam site

• Schedule the exam date

• Select supervisor and invigilator /assessor

• Train selected supervisor and invigilator /assessor

5.1 Printing Examination Materials

▪ The central exam administration team shall oversee the print/photocopy process of the
exam booklet; answer sheets and other documents. The designated person should print
or photocopy sufficient copies of the exam booklet to administer the exam to the
expected number of candidates that will be sitting for the scheduled exam. At least 10%
additional exam booklets should be printed in case a defective booklet or missing page is
discovered on the day of the administration.
▪ Print audit shall be made to make certain that all pages are printed and that all of the
exam questions and answers sheets are included and legible. Attendance sheets, contact
address and essentials documents should be printed. The exam has two parts, make sure
that both booklets are identified so that the right booklet is used for right time. If any
page is found to be defective during printing, it should be destroyed immediately using a
suitable paper shredder.
▪ After proper shredding using a mechanical device designed for such purpose, the
shredded paper should be placed in a recycling container.

5
▪ All exam booklets must be securely stapled three places at the left edge of the paper.
▪ Water proof and strong packing material should be used to pack examination materials
and locked with secured key during transportation.

5.2 Forms used for exam administration process

1. Candidate Eligibility Form – This form includes the information that will need to be
submitted to HPCALD about each examinee that wishes to sit for the HPCALD Entry
Level Exam.
2. Answer Sheets – Only original answer sheets shall be used. Answer sheets are scored
using an automated scoring machine (for instance OMR software) and only official,
original HPCALD answer sheets will be accepted. The answer sheet should be handled
carefully because the machine rejects unclean or torn answer sheets during scanning and
scoring.
3. Exam Roster – The Exam Roster must be used to indicate who sat for the Exam. Answer
Sheets submitted without a corresponding signature on the exam roster will not be scored.
Examines must sign in on the exam roster when they first enter the room. Before the
Exam is administered, the invigilator utilizes the exam roster to acknowledge that they
have confirmed the candidate’s eligibility and identification.
4. Candidate Comment Form – This form included at the end of the exam booklet and
allows candidates to communicate questions or concerns about the exam administration
or the contents of the exam to HPCALD. If the MCQ exam has two parts, the candidate
comment form should only be attached at the end page of the second booklet. (Annex I)
5. Incident Report Form – This form provides the exam invigilator with a way to let
HPCALD know of any unusual occurrences that could have an effect on the candidate’s
score(s). Annex II: Incident report form
6. Test Security Agreement – This agreement must be signed by the exam team member
before start of the examination.
7. Annex III. Test Security Agreement
8. Packing List – The packing list must be filled out and included when receiving and
returning exam materials from and to HPCALD.

6
5.3 Computer based Testing (CBT)
The computer Based Testing (CBT) has enormous advantages if administered with prior
rigorous preparation. It avoids waste of time and money on duplication and transportation of
exam materials. Moreover, the time required for scanning of answer sheets is overlooked
leaving the result publication period as short as a week or two. Dedicated exam forms and
reserve versions should be uploaded and made inaccessible until the exam start time. IT
personnel’s role at center and exam center is much needed in CBT. They should proactively
act on instances that could possibly lead to terrible failures. The role of supervisors,
invigilators, exam center coordinators and officials’ commitment on CBT should not be
belittled. The functionality and integration of national data center and local data center
requires meticulous work for the betterment of the CBT.

Preparation

▪ Institutional readiness for CBT should be assessed using checklists.

▪ It is a must to avail additional 10% reserve back-up computers.
▪ Software compatibility issue should be assessed and resolved for all computers including
back-up ones.
▪ IT experts should upload examines for online computer based examination.
▪ IT experts should upload items such as written exam, videos, lab result, imaging, and
audio that can be used to examine candidate.
▪ Back-up power generator should be available and standby.
▪ Back-up hard copy including 10% contingency should be printed and distributed along
with CBT.
▪ Orientation for exam team including IT personnel includes but not limited to: ensuring
software compatibility of computers, creating user name, online/computer testing is
administered, security cautions, quick fix for glitches).

7
5.4 Special Accommodations Policy and Process
Candidates who may apply for special accommodation includes those with:

▪ Documented /Diagnosed disabilities that would prevent them from taking the
examination
▪ Transitory conditions which are generally not disabilities like pregnancy, breast
feeding mothers, fractures, sprain
▪ Chronic medical illness (DM, Epilepsy, heart disease, COPD…...) May apply for
special accommodation

The institution should notify for candidate’s preparation of special accommodation within
two weeks. The special accommodation includes:
• The exam hall should be as per the national standards for disabilities
• Providing available waiting area for baby caregivers
• Protect the pregnant candidates from unfavorable condition
• Wheelchair, elevator and others
• Candy, water and others for DM patients

5.5 Safety Measures/security issues

General measures

▪ Testing personnel are not permitted to view the contents of the exam booklet at any
time before, during, and after the exam administration.
▪ No one is allowed to duplicate or retain any portion of the exam booklet.
▪ Keep all exam booklets in secure storage before and after the exam administration.

8
 Before the exam administration
✓ All the HPCALD staffs should sign contractual agreement not to pass the exam material to
anyone else other than permitted.
✓ Storing the soft copy of exam with password formed and known only by the
administration case team
✓ Signing contractual agreement in all steps of transferring of exam materials using legal
forms prepared by HPCALD
✓ Packing exam materials should be hard enough using water proof materials
✓ Exam materials should be transported by security police and supervisors to the exam
center
✓ Storing the exam materials in empty room whose key is kept only in Supervisor, central
team and exam center. The gate should be locked and board up
✓ The exam storing computers and storing rooms should be locked and not used for other
purposes
✓ Examiner should check the candidate to identify passing of hidden forbidden materials to
the exam room.

During the exam administration

✓ Arranging the exam booklet according to the proper coding
✓ Prohibition of using material and equipment other than the exam materials during
examination
✓ Invigilators and supervisors should be vigilant enough that candidates could use
sophisticated technology to breach the security of the exams.
After exam administration
✓ Signing an agreement in all steps of transferring of exam materials using legal forms
prepared by HPCALD
✓ Packing materials should be hard enough and water proof
✓ Exam materials should be transported by security police and supervisors to the HPCALD
✓ Exam materials should be stored securely three to six months before discarding

9
5.6 Breach of Security
• For any suspects of exam security breaches, report shall be made using the incident
report form to the HPCALD and then the directorate may request a legal action
accordingly.

5.7 Recruitment and deployment of supervisors and examiners

▪ Supervisors: must have Master’s Degree in any field of health science with four-
years of teaching/working experience. They should be trained/ certified with
appropriate training to administer MCQ/OSCE exams. Staffs from the HPCALD
can be assigned as an invigilator.
▪ Invigilators: must have at least first Degree in any field of health science with
two-years of teaching experience and are trained/certified with appropriate
training to invigilate MCQ. Staffs from the HPCALD can be assigned as an
invigilator.
▪ Assessor: must have Master’s Degree in any field of health science with four-
years of teaching/working experience. They should be trained/ certified with
appropriate training to administer OSCE exam.
▪ The recruitment and deployment of supervisors and examiners is the
responsibility of HPCALD.
▪ Supervisors and examiners should have recommendation letter from their
institution stating that they are capable of performing exam administration
properly.
▪ Those who brought recommendation letter and selected by the directorate will be
provided training on exam administration by HPCALD.
▪ The number of supervisors and examiners assigned depends on the number of
candidates who sit for an exam.
▪ The ratio of invigilators to candidates is 1:30 in one classroom.
▪ Regarding the reserve invigilators; there should be one for 1-5 class rooms, and 2
for 6 and above class rooms.

10
 ▪ Recruitment of supervisors and examiners shall not be from the same institution
where they work.
▪ The supervisor checks the availability of exam administration team on time.
▪ Sample schedule is shown below. However, one can use a different schedule as
appropriate.

Days Activity Responsible body

Day one Orientation to supervisors Central team
Day two Packaging and receiving exam materials Supervisors & Central team
Day three Start trip to exam centers** Supervisors & Central team
Day four Arrival to exam center Supervisors
Day five Orient invigilators and candidates, meet coordinators Supervisors
& get updates, check overall exam center status
Day six Administer the exam Supervisor & invigilators
Day Return exam materials to the directorate Supervisors
seven
▪ Note: **starting a trip depends on how far the exam centers are from Addis
Ababa

5.8 Orientation of supervisors and examiners

They should be oriented prior to the exam date. The HPCALD will decide the exact
orientation date. The orientation will be given on these topics:

▪ Delivering relevant knowledge and skills that enables to manipulate the computer
based exam

▪ Maintaining the security chain and sign contractual agreement of the exam
administration process

▪ Orient issues related to the exam materials and forms

▪ Orient issues related to code of conduct of exam team and candidates

11
 ▪ Instructions regarding the time and venue required for that specific station should be
included

5.9 Orientation of supervisors and examiner about dressing code

The candidate needs to respect the following Dressing Codes, so that He/she should:

▪ Wear a clean cloth which is free of rips and holes and not overly restrictive or
revealing.
▪ Wear Shirts with long sleeves and must have collars and also tucked in,
▪ Head Veil/Turban: Can be worn on religious grounds.
▪ Candidates’ hair, moustaches and beards must be neatly groomed. Long hair
(below the shoulders) must be tied back or up. Hair decorations should not be
worn.
▪ Not use perfume or cologne or any other scented products.
▪ Not wear cap or hat in exam rooms.
▪ Be kept Nails short and unvarnished with no extensions or decoration.
▪ Use make up in a moderate way.
▪ Only wear wedding or commitment ring. No other jeweler is allowed.
▪ Wear plain flat and rubber soled shoes, enclosing the whole foot and not made of
porous or absorbent material in order to protect the foot from bodily fluids and
equipment; do not wear shorts or sandals.

N.B: In addition to the above dressing code, Candidates should wear the appropriate
professional dressing (White gown (Uniforms), OR attires and aprons, etc.) during
OSCE examination.

5.10. OSCE exam station organizer preparation

▪ In addition to exam supervisor and examiner, recruitment of station organizer is
equally important for a successful of OSCE. This should be a local staff working in
liaising with logistic and support services to resolve the physical issues.

12
5.10.1 Recruiting station organizers
▪ They should have good interpersonal skills
▪ They should be a team player

Specific criteria for being an OCSE Station organizer are

▪ They should be local clinical lab/OSCE organizer/worker or Health professional

licensing examination directorate staff with relevant experience

▪ Completion of national workshop on being OSCE station organizer

▪ Station organizer should brief attendance one day prior to implementations.

5.10.2 Training of station organizer

▪ To help them with their responsibilities, station organizer should be trained by
preparing a workshop in which, ideally, all station organizers would participate prior
to being used in the examination.
▪ The workshop might take the following form:
✓ General information about the OSCE, including a definition, what it can assess,
the process, the evidence supporting its use, etc.
✓ Information specific to the OSCE for which the examiners are being trained
including the purpose, the number and type of stations being used, the length of
each station, the nature of the cases etc.
✓ The role of station organizer including knowing what to look for, how to
complete the forms, minimizing intrusiveness and interference, etc.
✓ Orientation and Debriefing (Examiner, SP trainers, SPs, candidates)

5.10.3 Examination day briefings

▪ Candidates, Examiners and Standardized patients are expected to arrive at the
examination site at least an hour before the examination start time to familiarize
themselves with the details of the station.

13
 ▪ Even though examiners and standardized patients had received prior training, separate
short information must be given before the exam.

5.10.4 Standardized patient briefing

• Emphasis should be given to inform necessity of maintaining consistent performance
for all candidates. The SPs should be to the point and follow the script.

• Inform standardized patient that they will be substituted at appropriate intervals in

case of lengthy examinations or if they need a break with similar patients.

5.10.5 Candidates briefing

• Candidates should address the description of the circuit, start stations, rest stations
and reminders of rules and regulations including details of any quarantine
arrangements.

5.10.6 Examiners briefing

• Check students’ identity at the start of the station, reviewing the scoring checklists,
how to complete the mark sheets and the procedures for reporting concerns about
candidates. Setting-up the OSCE circuit (marking station-sign posts)

5.10.7 before the day of the Exam

• Ideally the examination should be held in a clinical simulation Center or Skills
development lab that have realistic clinical environment picture. The examination
center should have enough space to accommodate standardized patients, candidates,
examiners, supervisors and all support staffs. If the center has no enough space, the
station can be prepared in open space using local materials. Regarding OSCE station,
there should be a minimum of 12 stations in all exam centers.

• When planning an OSCE, the exam center should be booked one month in advance.
Complete and precise information to standardized patients regarding the details of the
case scenario, where the encounter takes place, how they should be dressed, behave,
positioned, and one -two weeks before the OSCE.

14
 • All listed equipment should be collected, and checked for their functionality at least a
day before the exam. It is always important to reserve spare equipment, batteries in
case of run out and examination materials failed functioning and additional stations in
case of problems with the planned stations.

• The availability of all examination materials such as candidate instructions sheet, a

skill checklist for each station, must be rechecked.

6. During exam administration

6.1 MCQ exam administration

The exam center coordinator, supervisors, and invigilators should arrive at least one hour
prior to the scheduled time. Before admitting the candidates, the following preparations
should be completed:

1. Ensure all exam classrooms are ready for examination.

2. Ensure all exam team members are available in the exam center on time; if there is
absent member, substitute immediately by reserve members.
3. Ensure required numbers of exam materials are ready at the exam center.
4. Ensure examinee eligibility forms are on hand and are arranged alphabetically
5. Exam materials must be kept out of reach of the examinees until distributed by
invigilators.
6. The following information are written on the black/white board:
✓ Invigilator(s) name(s)
✓ Date
✓ “The Examination is ______ hours long”
✓ “The Examination began at _________” (enter time when the Exam starts)
✓ “The Examination will end at _________” (enter time 2 hours after start time)
✓ Time notice will be given three times during the exam; i.e. on half of the
examination, at 30 minute before the end of the exam, five minutes before the
end of the exam.

15
 6.1.1 Assessing the Seating Arrangement
▪ Ensuring that the examinees are seated in sequence, row-by-row, beginning in the
front of the room.

▪ Invigilators must ensure that candidates are seated at least 1.25 meters apart in all
directions

6.1.2 Meeting Candidates

▪ Exam Invigilators should welcome candidates as they enter, check candidate
eligibility, confirm identification, and have them sign on the exam roster.

▪ Once a candidate has been checked, they should be directed to sit in the appropriate
location.

▪ During the exam administration, candidates must not come with personal items other
than allowed materials for examination; such as laptops, ipads, cameras, any sharp and
war weapon materials, eye glass (unless ordered by physician).

▪ If in case candidates bring personal items other than the allowed materials, these
materials can be put in front of the exam room throughout the session near an exit. So,
candidates can quickly gather their belongings as they exit the room. The exam
team/exam center will not take any responsibility for any loss or damage of
candidate’s properties.

▪ Only exam team and others who have completed and filled an exam Administration
agreement with HPCALD are permitted in the examination area.

▪ The examiner or supervisor can check identification and confirm eligibility any time
during the examination.

▪ Candidates must have signed consent form.

NB: Under no circumstances an individual shall be allowed to take the Exam if it

cannot be verified that they have completed the Candidate Eligibility Form or their
identity cannot be confirmed.

16
6.1.3 Distributing exam materials and filling the answer sheet
▪ Once examinees are seated, examination packet should be opened in front of the
examinees by assuring the sealing of the packet patent.

▪ The examiner must count the number of exam booklets and answer sheets before
administering the exam. If there is any loss, he should report timely and take
appropriate measure before starting the exam.

▪ Exam booklet and answer sheets should be given directly to each examinee.

▪ Examinees are not allowed to mark on the exam booklet and answer sheet until
invigilators give the permission to start.

▪ Once the examinations are started, the doors better be closed.(special considerations
should be given accordingly)

▪ The candidates should be told to check for completeness of exam booklets before
starting the exam. If an incomplete or misprinted exam booklet is found, it should be
replaced with the right one.

▪ Candidates shall be allowed to enter to examination room until 30 minutes of the

starting time, and candidates are not permitted to leave the exam room before one hour
from the starting time.

▪ Absent candidates and candidates who come later than 30 minutes during exam should
not be allowed to enter the exam room and reported to HPCALD.

▪ Absent candidates with sound reason, will be re-schedule for the next round without
any registration fee and the attempt will not be counted.

▪ Absent candidates without sound reason, will be re-schedule for the next round with a
registration fee and additional 50% penalty, but the attempt will not be counted.

▪ A candidate has the right to take the exam five times including the first attempt.

▪ The room should never be left unattended while the exam is in progress.

17
6.1.4 Managing cheating
▪ If an invigilator notices or suspects an examinee are looking at another's paper: a
general announcement should be made for candidates to keep their eyes focused on
their own papers. If the action continues, the examinee should be asked to move to
another seat considering the exam booklet code. If this continues, the examiner will
disqualify the result of the candidate and the incident must be documented using the
incident report form.

▪ If an examinee is found using written materials other than the exam materials, the
document will be attached with the answer sheet and sent to the HPCALD using the
incident report form but the result will be disqualified.

▪ Other misconducts like: being noisy, disruptive, causing any kind of altercation,
removing examination materials, any forceful act in exam rooms and other attempts
believed by exam team as misconduct, the following measures will be taken using
relevant evidences attached with the incident report form. For the first misconduct the
invigilator inform the candidate. If the Misconduct continues for the 2nd time, last
warning will be given. If it continues, the candidate will be banned from the exam room
and the result will be disqualified.

The following rules shall be followed during the exam administration process;

▪ If a Candidate has an emergency or need to use the restroom, only one candidate at a
time is permitted to leave the room together with the exam team member ensuring the
rest room is free from activities that breaks the security issues with close follow up.
▪ Smoking, eating and drinking are strictly forbidden in the exam room for exam team
and candidates.
▪ Exam team members should circulate quietly throughout the room during the
examination period
▪ If a candidate feels there is an error within an item, they should note this on the
candidate comment form

18
 ▪ Exam team members are not allowed to answer questions concerning the content of
the exam; however, any procedural questions that arise on exam administration may
be answered to the best of their ability.
▪ Examinees should remain in their seats starting from the time announcement of
“15minute is left” until end of the examination to prevent cheating and disturbance to
examinees.
▪ The invigilator should take their exam materials before allowing the candidate to
leave the room

Following exam time completion

▪ The invigilator should pick up all of the examinee’s exam materials and the examinee
should be informed that they are not allowed to do after the end of exam time.

▪ If the examinee insists on finishing the exam, they can be allowed to finish; but should
be told that the exam will not be scored.

▪ The incident must be documented on the Incident form and the evidences should be
attached to the form.

Reporting Flow of Cheating

Examiner Local Supervisor NHPCALD

coordinator

NB: The NHPCALD may subject examinees suspected of cheating to penalties including
(but not limited to) the followings: canceling their examination score (with no refund or
credit for any future examination), denying scoring of their examination, revoking their
certification or legal proceedings to recover costs associated with lost examination
questions.

19
 6.1.5 Handling the unexpected
▪ In case of a fire or weather-related evacuation, the exam team should never endanger
themselves or any examinee. The examinee should keep their exam materials with
them and exit quickly.

▪ If an Examinee becomes ill or injured, a second invigilator/supervisor should be asked

to call for assistance. An invigilator should remain with the ill or injured examinee
until help arrives. If an examinee decides not to finish the examination, exam materials
should be collected. The incident shall be documented on the incident report form and
examinee’s answer sheet should be attached to the form.

6.1.6 Managing the answer sheet

▪ The supervisors should pack the answer sheets and there should be a signature of
exam team and stamping the seal of exam center on its edge.

6.2 Computer based exam administration

6.2.1 Before exam starting time, the following should be done:

▪ Functionality of computers and other accessories should be checked.
▪ Availability and functionality of generator (alternative electric power) should be
checked.
▪ Availability and functionality of back up computers and other accessories should be
assured
▪ All exam computers must be checked that they are free from other source of internet
connectivity than the exam system.
▪ The computers used for the exam purpose should be disabled receiving other data
transferring devices like USB device, hard disk, CD and others.

6.2.2 During running of the computer based exam

▪ Clear instructions of log in should be given for candidates and candidates should be
assisted to log in by invigilators.
▪ IT specialists should be ready during the exam and problem encountered.

20
 ▪ Candidates who faced a technical problem should be shifted to other computers and the
time wasted during the transfer should be considered.
▪ If large numbers of computers failed, the exam center should be prepared for paper
exam.
▪ There should be one-meter distance between two computers in the exam room.
▪ Candidates who deliberately perform harmful activities and causes damage on
computers will be considered as serious security issue and the result should be
disqualified. The candidate should also pay estimated price of the computer.

6.3 During OSCE Administration

6.3.1 Setting up the OSCE circuit and Station preparation on the day of the Exam
▪ It is important to maintain uniformity among examination sites if an examination is
taking place at many sites. Arrangement of the position of the chairs, tables and
couches will be based on the script writer specifications. Care must be taken to
allocate space appropriate to the tasks, equipment and the number of personnel
expected to stay in the room taking into account providing an appropriate
environment for the candidates to perform the procedures.

▪ Rest station must be located in area where candidate at this station cannot over hear
what is being said at the other stations. Only two candidates will be in rest stations
during the OSCE exam. So all candidates will stay in rest station turn by turn starting
from the beginning of the exam to the end for duration of time equivalent to the
length of one station.

▪ A candidate should not move to the next station before the ringing of the “move bell”
even though he finishes earlier than the given time.

▪ Each station lasts for 5-10 minutes. It takes 1 minute to read instruction, 8 minutes to
perform the task and 1 minute to move to the next station. The direction of the Circuit
is clockwise/counter clockwise(follow station number labeled on each spot in
ascending order)
21
 ▪ According to the designed circuit flow stations, sign and arrows have to be labeled
with bold A4 size paper.

▪ Standardized patients would be selected and assigned for exam from the pool of
trained standardized patient. Always preparing an extra number, at least two
Standardized patients for each station.

6.3.2 Running the circuit

• Each candidate will be allocated a start station and move from station to station in the
direction of the circuit until all stations completed.

• The movement of the candidates from one station to another can either be managed
by ringing a bell manually or by using automatic set up with voice commands clearly
instructing the candidates and the examiners.

• The OSCE can be started with the command ‘Start Preparation, during which time the
candidates read the question, followed one minute later with instructions to ‘enter the
station’. The next instruction could be ‘one-minute left’ and the station would end a
minute later with the command ‘move on’.

• Once the examination is started, there should be enough personnel available to ensure
that the candidates move in the right direction, clean and supply stations with all the
required exam supplies prepare stations between candidate changes and assess if any
SPs need a break.

• Examiners and support staff should be vigilant for candidates leaving stations with
equipment removal, or mark sheets.

6.3.3 Arrange refreshment facilities

• Providing refreshments for examiners and standardized patients may be necessary to
accommodate the large number of candidates in morning and afternoon session.

22
6.3.4 Quarantine
• If the same set of OSCE exam is used for the entire candidate, those candidates
waiting for the exam should be placed in a separate room until all of the earlier
candidates who have completed the exam.

6.3.5 Candidate misbehavior during OSCE

• Candidates should have respectful relationship with standardized patients and
assessors but if a candidate shows disrespect for assessors and mistreat (harmful act)
standardized patients, first the assessor should give correction. If the candidate insists
on doing these misbehave, it should be reported for the center coordinator and
HPCALD.

• Candidates should respect the instruction given by time keepers. If a candidate

disobeys and insist to flow the instructions given by time keeper, first the assessor
should give correction. If the candidate insists on doing, it should be reported for the
center coordinator and HPCALD.

• If candidate purposefully destruct any material in the circuit, it should be reported to

the center coordinator and HPCALD.

6.3.6 Appealing Process during the Examination (MCQ and OSCE)

• Candidates can appeal any compliant related to the exam both verbally and in written
form.
• Exam related complaints can be solved by discussion between candidate and
invigilators.
• Those complaints beyond the invigilator shall be managed by supervisors and exam
coordinator respectively.
• The candidate has the right to complain for Health professional competency
assessment and licensure directorate if he/she is not satisfied by solution given at exam
center level.

23
7. Post-exam administration Procedures

7.1. Handling Exam Materials

• Following the examination, the mark sheets and all forms used during exam should be
collected and counted

• All completed forms and all used and unused exam materials shall be placed in the
package in a neat and orderly fashion:

✓ Packing List
✓ Exam invigilator agreement form(s)
✓ Exam assessors’ agreement form(s)
✓ Exam rosters
✓ Completed Answer Sheet
✓ Exam booklets
✓ Station profile booklets
✓ Incident report Form

Note: Do not fold, bend, crease, staple, tape, paper-clip, bind with rubber band or
otherwise damage the answer sheets. Make sure materials are packed in strong packaging
material to prevent damage in transit. Answer sheets are processed through an automatic
scoring machine and must be delivered flat and unbound for proper handling.

After the exam materials are collected:

▪ The Exam Supervisor should collect all exam materials and shipped, within two
business days after the completion of the exam to HPCALD.
▪ The central team at HPCALD should collect all the exam materials according to
the checklist and should be signed and stamped

24
7.2. Scoring Process
• The licensure examination can be scored manually or electronically.
• The answer key format should be standardized and readable by the software
adjusted computer.
• Because the HPCALD uses a criterion-referenced examination, each examination
has a passing score that is independent of the passing scores of previous or
subsequent examinations.
• The central team shall communicate the exam development team regarding the
passing score
• Pass score must be computed and compared independently (for MCQ and OSCE)
against a conditioning threshold specific for each category set by the exam
development team.
• The central team shall records examinees results (please use score submission form
developed by the registrar case team) and submits to the registrar case team after
its approved by the HPCALD directorate
• The security and confidentiality of the exam should be maintained.

NB: If an examinee achieves above or equal to the standard set score for the two exam
categories, the overall result is “Pass.” If an examinee achieves below the standard
set score for one or two exam categories, the overall result is “Fail.”

7.3. Exam Analysis

• Both psychometric Analysis and expert judgment will be performed for MCQ and
OSCE exams to determine item or station properties. It includes but not limited to item
difficulty index, item discrimination index, inter-rater reliability tests, and expertise
judgment value.
• The exam analysis process shall involve psychometric experts and exam development
team.
25
 • The exam analysis should be conducted in collaboration with the exam development
team and results should be documented for future use.

7.3.1. Post-Exam Analysis for Written Exam

• Statistics should be calculated for individual exam items (item analysis) and for the
whole examination.
• The examinations and individual items are reviewed in a post-test analysis. The
purpose of the post-exam analysis is to review exam items that do not, on the item
analysis, perform statistically as expected. Items are evaluated individually based on
three criteria: the percentage of examinees chose the correct response, how well the
item discriminated between examinees who had high scores and those who had low
scores, and the spread of responses (i.e., the percentage of examinees who chose each
response).
• Items that few examinees answered correctly, items that were poor “discriminators”,
and items for which equal numbers of examinees chose three or four responses are
flagged for review. A list of the flagged items is sent to the Exam development team
for review.

The following exam analysis should be considered:

• A Cronbach’s alpha of > 0.7should be taken as an acceptable level of reliability for

national licensure exam.
• Item difficulty or item mean: the item difficulty index ranges from 0 to 100; the higher
the value, the easier the question. The HPCALD MCQ exam is four responses multiple
choice, thus 70 is the ideally difficulty.
• Item discrimination: determines the extent to which the given item discriminates
among examinees in the function or ability measured by the item. Value of the
discrimination index can range from -1.00 to +1.00. Items having negative
discrimination are rejected. Items having discrimination index above .20 are ordinarily
regarded satisfactory for HPCALD.

26
7.3.2. Post Exam Analysis for OSCE

Psychometric Analysis
The administration and exam development case team should collaborate for post-exam
analysis. The following analysis should be performed for each OSCE by the data
management team and should be available at the exam board (For Additional analysis
psychometrician should be consulted):
1. Cronbach’s alpha of > 0.7 should be taken as an acceptable level of reliability for
national licensure exam. The examination board should apply Cronbach’s alpha that
allows the detection of the OSCE stations which are main sources of error, by removing
one station at a time from the analysis and looking at the reliability of the remainder.
An analysis of the difference removing each station would make to the Cronbach’s alpha
should be undertaken. If there is a substantial difference (>.05 increase in alpha), this
may be an indication of a problem with the OSCE station. In such circumstances, quality
improvement should be undertaken by revisiting the performance of the station and
reviewing checklist and station design.

2. G-theory use is recommended to investigate the sources of error and the number of
observations required for a given level of reliability. Multi-facet ANOVA analysis should
be conducted to obtain the estimates of the components of variance for each of the
following facets: stations, sites, test versions, and all their interactions. G coefficient of
0.7 to 0.9 should be taken as an acceptable level of reliability for national licensure exam.
3. Coefficient of R2 analysis should be performed to measure of the correlation between the
checklist score and. the global rating for individual stations. Good correlation (R2>0.5)
will indicate a reasonable relationship between checklist scores and global grades. When
low R2 reported, quality improvement should be undertaken by re-writing of the station
and checklist with plans for re-use of this station and subsequent analysis of performance
within a future OSCE.
4. Between-group variation: this metric should normally be <30%, and anything higher
should prompt investigation of possible systematic biases by time, examination site or

27
 examiner factors. A line representing 2 standard deviations will be added to the examiner
feedback graphs by the data management team to indicate outliers.
5. Number of fails: An unusually small or large number of student fails for a single station
should prompt investigation
6. Difficulty index (Item Analysis): refers to the “easiness” of the station, Stations with
difficulty levels of about .50 have been found to be most useful.
7. Discrimination index (Item Analysis): The purpose of this index is to help determine
how well each station separates more knowledgeable students from less knowledgeable
students. Additional Psychometric analysis could be done as deemed as necessary.

8. HPCALD examinee result ratification

• The results should be ratified by the directorate
• The directorate should ratify the results and signs them within 2 months after submission
of the compiled exam materials to HPCALD. But in case of any doubts, the directorate
has the right to request extension of deadline and verify the results again.
• The exam ratification team also decides disciplinary issues of both the candidates and
examiners reported across the exam process on this meeting.
• In addition to this the central team at HPCALD evaluates the feedback reported on the
performance of examiners and makes a decision.

9. Complaints and appeals

• Examinees have the right to present their complaints through the annexed appeal
writing format.
•
• Annex IV: Candidate’s complaints and/appeals writing format
• Examinee must submit their appeal to HPCALD in writing (should provide proof of
payment) within 10days after the exam, addressed to the Administration case team.
Appeal will be presented to the HPCALD.
• The directorate should make every effort to respond to an examinee with a decision
within 10 days of receipt of their appeal based on rule and regulations of HPCALD.

28
10. Feedback and reporting

10.1. Feedback Mechanism

• The national licensing examination procedures should employ various methods of
feedback solicitation to inform and improve the licensing examination process by
collecting opinions from students, assessors, institutions and other stakeholders.
• In addition, the national licensing examination shall provide feedback based on the
analysis of results of examinees to exam center, training institutions and other
stakeholders.
The following methods can be used to provide and get feedbacks:

• Post exam evaluation using questionnaires for examiners, examinees and exam
centers/institutions
• Feedback will be collected using the annexed checklists throughout the exam
process accordingly.
• Get feedback using suggestion box or via the internet.
• Using supportive supervisions at exam centers and various sections of the
departments at exam centers.
• Periodic review meetings with stakeholders

General
• After each batch of examination, a comprehensive report should be made within two
months.
• Based on approval, consent and obtaining ethical clearance from authorized body, the
health professionals’ competency assessment and licensure examination results and/or
information’s can be used for generating scholarly materials that can be reported and/or
published for the scientific community.

29
Institution
▪ After publication of results and handling of any related complaints and appeals, with
aim for improvement of institution, specific comprehensive written feedback; including
total number of student who passed and fail the examination by academic discipline,
sex and domain should be sent for each institution within 2 months
• Based on the feedback sent to each institution, the HPCALD directorate may call for
national consultative meeting with the responsible personnel’s and stakeholders.

Each report includes but not limited to:

• Examiner’s adherence to all rules and regulations
• Examiners’ completeness and accuracy of all checklists and/or rating forms

11.Discarding exam materials

▪ Exam booklet should be mashed/ burned properly in front of the exam committee
within three days after it returned back to the center.
▪ Whereas answer sheet should stay up to 3- 6 months of result publication.
▪ One copy of all items of the exam should be kept in both soft and hard copies in
licensing exam directorate office to compute further analysis.

12. Quality Assurance

Quality Assurance is vital components in designing, implementing and evaluating national
licensure examination. Quality Assurance team should be established. Monitoring and
evaluation could be conducted based on the annexed checklist. Although many quality
assurance procedures take place following the licensure examination, this guideline directs
quality assurance from three perspectives. These are Pre-examination, during examination
and Post- examination quality assurance mechanisms.

The quality assurance system should be integrated in all phase of activities and it should
be internal and external. The internal quality assurance system may be assessed by the
local coordinators, examination supervisors and invigilators or assessors. The external
quality assurance will be assessed by externally teamed stakeholders (professional

30
 associations, HERQA, FMOH, MOE, National Examination Agency, oversight
committee).

All quality assurance team should follow the annexed quality assurance assessment
checklist.

The role and responsibilities of the Quality Assurance team may include:

• Conduct Pre-examination evaluation and timely report to the exam coordinator

• Evaluate the Quality of each item and station and take immediate remedial actions
• Ensure the Pilot testing of the exam and each station before the commencement of the
examination.
• Evaluate during the exam process and provide timely report to the exam coordinator
• Evaluate the Post- exam process and provide timely report to the exam coordinator
• Compile Psychometric analysis from the data management team and propose appropriate
remedial actions based on the analysis.
• Ensure the shipping and disposing mechanisms of the pilot test and original exam
according to stated procedures, including using different representatives, security bodies,
and local and health professionals’ competency assessment and licensure examination
personnel.
• Compile report from External assessors and propose appropriate remedial actions
• Compile Evaluation reports from students, examiners, support staff and report timely to
the exam coordinator

12.1. Pre- exam quality Assurance

Before the implementation of both written and practical examination (OSCE) strict quality
assurance procedures should be followed to maintain the quality of national licensure
examination. For the Pre-examination quality assurance process, pre-written exam quality
assurance checklist (

31
 Annex VII: Pre-Written Examination Quality assurance checklist (To be filled by QA team) ) and
Pre-OSCE quality assurance checklist (Annex VI: Pre-OSCE Quality assurance checklist (To
be filled by QA team)) could be used to monitor and evaluate the Pre-examination
procedures.

12.2. During - exam quality Assurance

All examination procedures should be checked for their quality using during examination
quality assurance checklist for written (

Annex IX: During-Written Examination Quality assurance checklist (To be filled by QA team))
and OSCE examination

Annex VIII: During OSCE Quality assurance checklist (To be filled by QA team). During
examination, quality assurance mainly focuses on out sourcing of all the listed exam
materials, checked to ensure that it is in good working order and smooth running of the
examination).

12.3. Post- exam quality Assurance

Following administration of an examination, post exam quality assurance procedures
should be followed to assess the key activities considered after the conduct of the written
and OSCE examination using post examination quality assurance checklist (

Annex XI: Post-Written Examination Quality assurance checklist (To be filled by QA team), Annex
X: Post-OSCE Quality assurance checklist (To be filled by QA team) respectively). Thus
includes, handling result, exam team and ratification, publication of result, compliant and
appeals, and reporting and feedback. In addition, the health professionals’ competency
32
 assessment and licensure examination directorate should arrange for content experts to
review the examination results and verify that the content of the test items is accurate. For
multiple choice tests, a standard statistical analysis, or item analysis should be performed.
The item analysis can identify items that are mis-keyed, too easy, or too difficult.
Questionable items should be edited, revised, or removed. Examination results should not
be released until content experts have reviewed the results of the item analysis. For OSCE
examinations, a statistical analysis of the results typically addresses the distribution of
candidate scores, reliability of examiner ratings, percentage of agreement between
examiners, and correlations amongst different subject matter areas of the examination.

13.Roles and responsibilities of Exam Team

Responsibilities of the central coordinator

✓ Facilitate signing of Memorandum of Understanding (MOU) with training institutions

✓ Prepares examination schedules and timetables.
✓ Coordinates the printing, packaging, storage and delivery of examination materials.
✓ Recruit and deploy supervisors, invigilators and assessors
✓ Give orientation for supervisors.
✓ Distribute supervisors and invigilators to exam centers
✓ Supervise exam security at all levels
✓ Ensure timely administration and return of exam materials
✓ Ensuring if exam is ready for administration
✓ Manage candidate appeals
✓ Resolves problems as well as makes recommendations to improve examination process
✓ Receive complete exam materials
✓ Scan and score answer sheets
✓ Perform psychometric analysis
✓ Prepare technical report for officials and institutions
✓ Notify result and provide feedback to candidates and training institutions
✓ Store good performing items in the Q-bank
✓ Monitor and evaluate exam administration process

33
 Responsibilities of the supervisors

✓ Receive orientation and get certified

✓ Have complete materials for the exam center (exam booklet, answer sheet, incident
form, roster (attendance sheet), candidate comment form, invigilator agreement,
candidate eligibility form)
✓ Have list of candidates and a photo album showing the candidates photo, name,
admission ID and type of program.
✓ Orient invigilators, candidates, exam center coordinator and support staff.
✓ Manage and report exam irregularities in a written form.
✓ Supervise the entire exam administration at the exam site.
✓ Ensure venue is arranged and equipped with materials ahead the exam date
✓ Maintain exam security at all levels
✓ Communicate with central team overseer and local coordinators
✓ Collect exam materials from invigilators and return to the center using checklist
✓ Report the overall OSCE result to the center

Responsibilities of the Exam Center coordinator

✓ Communicate with central coordinators and supervisor

✓ Recruit local support staff and assign roles
✓ Arrange exam venue with standard requirement
✓ Post exam schedules at convenient place
✓ Facilitate exam administration process at the center
✓ Setting up individual stations
✓ Setting up the bell system for OSCE
✓
Responsibilities of invigilators/assessors

✓ Arrive to the exam center on time

34
 ✓ Take orientation
✓ Cross-check personal identity of examinees
✓ Examine candidates
✓ Report any irregularities to supervisor
✓ Record and report the overall OSCE result to the supervisor
✓ Take corrective actions during examination
✓ Manage time properly
✓ Collect and return answer sheets and exam booklets to supervisor
Responsibility of IT expert

✓ Perform institutional readiness for assessment of CBT

✓ Check the functionality and compatibility of computers
✓ Assist orientation for examinees on CBT
✓ Provide IT support for computer based testing
✓ Arrange the computer lab in a row-by-row at 1 meter between examinees
✓ Fix in case of glitches
✓ Contribute in appeals management pertaining to CBT
✓ Make sure the back-up generator is ready and supply power to the data center
Security police

✓ Provide security service at the center

✓ Manage conflict
✓ Contribute for appeals management (act as a witness if possible)
All exam team members

✓ Arrange the venue, computer lab

✓ Arrange and check functionality of back-up generator
✓ Arrange exam hall
✓ Photocopying
✓ Arrange arm chairs
✓ Support setting up OSCE stations
✓ Provide refreshment for examiners
35
✓ Arrangement of required material and equipment
✓ Maintaining a central store for necessary material, equipment and their replacements for
skill exams
✓ Quarantine arrangements for examinees waiting for OSCE exam

Role and responsibilities of station organizer

✓ Setting up individual stations

✓ Preparing station materials.
✓ Distributing materials to each stations
✓ Attending the needs of examinees, examiners, standardized patients and supervisors
✓ Developing the OSCE address and post it on accessible place
✓ Setting up the bell system
✓ Developing and placing the number and arrow signage at appropriate places
✓ Arrangement of required material and equipment
✓ Maintaining a central store for necessary material, equipment and their replacements
✓ Quarantine arrangements for examinees waiting for their exam

36
Annexes
Annex I. Candidate Comment Form
Federal Ministry of Health
Health Professionals’ Competency assessment and Licensure Examination
Directorate
Candidate Comment Form
Name of candidate: Date:
Sex:
Professional category:
Exam center:
General comment on construction of the exam, difficulty level, examiners, etc

Specific comment of questions:

37
Annex II: Incident report form
Federal Ministry of Health
Health Professionals’ Competency assessment and Licensure Examination
Directorate
Incident report form

Type of incident:

Course ID: Health Professional

Category:______________
Paper title:
Exam Start Time: Date:
Time of Incident: Time of Incident:
Candidate ID: Academic Year________
Candidate Name:

Details of Incident & Action Taken :

Suspected Academic Dishonesty Notice issued

YES NO
Copy of Suspected Academic Dishonesty Notice attached
YES NO
Confiscated Items (list):
Name of Supervisor: Signature:

38
Additional action taken :

Attachments:
Name of Supervisor: Name of Supervisor:

Also witnessed by Supervisor: Also witnessed by Supervisor:

Additional Comments:

Examinations Office use:

Signature: Date:

39
Annex III. Test Security Agreement
Federal Ministry of Health
Health Professionals’ Competency assessment and Licensure Examination
Directorate
Test Security Agreement

Date: _______________________
As a member of the team for the development, administration, and scoring of the
health professionals’ competency assessment and licensure examination for Health
Professionals’ in Ethiopia, I, the undersigned, accept the responsibility for
maintaining strict confidentiality of items, materials and information related to the
examination as detailed below:
I declare that no one from my family members, including my spouse, children,
brothers and sisters, is a candidate for the licensure examination in the year
___________.
I will not share any information about the examination to anybody including my
families and friends except duly authorized persons from the national licensing
examination though any means or media including telephone, text message,
pictures, internet, print materials, face to face discussion, etc.
I am aware that examination items under my control including, but not limited to
scratch papers, draft copies, are to be kept in a secure location.
I am aware that I may be sued in accordance with the country’s law, if I violate
these provisions.
I have read and understood the provisions of this security agreement. My signature
below signifies that I do accept the terms and conditions of this security agreement.

On behalf of Ministry of Health Exam team Member

Name: ______________________ Name: ______________________
40
Signature: __________________ Signature: __________________
Date: ______________________ Date: ______________________
Annex IV: Candidate’s complaints and/appeals writing format
Federal Ministry of Health
Health Professionals’ Competency assessment and Licensure Examination
Directorate

Date: ______________________
Name of exam center: ___________
Candidate’s full name: ________________________________
Candidate’s ID: ____________________
Candidate’s discipline (professional category):_____________________
Examiner’s full name: ______________________
Reason for the complaints and/appeals:
__________________________________________________________________
__________________________________________________________________
Description for the complaints and/appeals:
__________________________________________________________________
__________________________________________________________________
__________________________________________________________________
_______________________________________________________________.
Receipt number for results appeal____________________

41
 Annex V: Single station OSCE Quality assurance checklist (To be filled by
QA team)
Federal Ministry of Health
Health Professionals’ Competency assessment and Licensure Examination
Directorate
Exam center:_______________

Station Author/s: ______________________ Station Title: _____________________

No STATION COMPONENT EXAMPLES Yes No

1. Exam Consider purpose of the exam (For the National Licensure Exam)

2. Objective To assess a candidate’s ability to take a history from a patient … (Clearly

stated)

3. Assessed Discipline Medicine, Midwifery, Public Health, Anesthesia etc. (Clearly Indicated)

4. Assessed skills/ roles Communication, History-Taking, Physical Examination, etc. (Clearly

Indicated)

5. Time allotted for the task An appropriate and realistic time allocation for tasks at individual stations
(Clearly Indicated) 5 minutes, 10 minutes, 11 minutes etc.

6. Candidate Instructions Clinical stem, task required, time allotted (Clearly stated)

7. Scoring Instrument Checklist, rating scales (Clearly stated)

8. Case-Related Questions Questions asked by examiner or SP; Patient Encounter Probe (PEP)
questions (Clearly stated)

9. SP Instructions Demographics, affect, physical findings, opening statements and/or

questions, HPI, etc. (Clearly stated)

10. Room Set-up and Props Skill Development Labs, Hospital bed, radiographs, IV poles, reference
material, etc. (Clearly designed and stated)

11. Review, Revise and Pilot Input from multiple content experts (performed)

QA team members:
Name Signature Date

__________ ____________ ______________

___________ _____________ ______________
___________ _____________ ________________
 Annex VI: Pre-OSCE Quality assurance checklist (To be filled by QA team)
Federal Ministry of Health
Health Professionals’ Competency assessment and Licensure Examination
Directorate
Date: ________________________________
Exam center: ___________________________
Professional category____________________
N Areas Detail Means of Y N
Verification e o
o s
.
1 Administratio OSCE Coordinators assigned Observation
ns
OSCE committee established Observation

Personnel appointed responsible for the time keeping Observation

2 Design and Academic Program The spread of skills within an Observation and
Development specific Station examination is reasonable
blue print is Document
developed Academic Specific Blue prints
were prepared based on the metrics review
indicated in the guideline

The test content represents what

the curriculum needs to assess, the
tasks are realistic and the right
domains are being assessed

Stations roadmap is done Document review

An OSCE station bank is developed and maintained Document review

3 Examiners, Examiner recruitment , Specific selection Document review

External selection and Training criteria is available
assessors,
SPs and Role and responsibility Document review
Candidates communicated

Contractual (Security) Document review

agreement was made

Examiner training was Document review

conducted
 Pool of trained Document review
examiners exist

Briefing orientation was Document review

done

External Assessors Specific selection Document review

criteria is available

Role and responsibility Document review

communicated

Contractual (Security) Document review

agreement was made

Assessor pool Document review

External Assessors Document review

training was conducted

Briefing orientation was Document review

made

SPs recruitment , Criteria for selecting Document review

selection and Training SPs is available

Role and responsibility Document review

communicated

Contractual agreement Document review

was made

SPs were trained Document review

pool of trained SPs Document review

Candidates / students Orientation is given Document review

Hand book is developed Document review

4 OSCE Is comparable to the stations designed for the Observation

venue number of stations and seamless flow of
candidates candidates through the
examination/ The
stations are clearly
signposted

Briefing rooms are Observation

available
 Administrative offices Observation
are available

waiting rooms for Observation

patients and examiners
are available
Quarantine facilities are Observation
available

Refreshment areas are Observation

available

5 Assessment The assessment tools (OSCE) were developed and Observation

tools validated by a committee of experts

QA team members:
Name Signature Date

_____________ _____________ _____________

_____________ ______________ _____________
______________ _____________ ______________
Annex VII: Pre-Written Examination Quality assurance checklist (To be filled
by QA team)
Federal Ministry of Health
Health Professionals’ Competency assessment and Licensure Examination
Directorate
Date: __________________
Professional category__________
N Areas Detail Means of Yes No
o Verification
.

1. Design There is a sufficient sampling of targeted standards

and
Developm Time is considered for both educators and students.
ent of
Blue Print The cognitive demands reflect those articulated in the targeted
standards.

The blueprint lists or references the targeted content standards.

The blueprint designates item counts for each standard.

The blueprint reflects a range of domain of knowledge levels.

The blueprint item/task distribution reflects that in the

specification tables.

2. Design Presents a definite, explicit, and singular question.

and
Developm Worded positively (when possible).
ent of
MCQ Only one (1) correct answer.

All distractors are plausible and capture common

misconceptions or errors.

Answer option and distractors are the same length, structure,

and format.

All choices are grammatically consistent with the item stem.

Contains no clues to the correct answer or to any other answer.

Charts, tables, graphs, and images are clearly placed within

the item/task.
 “All of the above” and “None of the above” have been
avoided.

Directions state what to do, where and how to respond, and

point value.

3. Administr Orientation given to supervisors and invigilators

ative
Guidelines Orientation given to candidates

Orientation given to support staff

Exam seat prepared per candidates number

Exam booklet and sheet ready per candidates number

QA team members:
Name Signature Date

_____________ _____________ _____________

_____________ ______________ _____________
______________ _____________ ______________
Annex VIII: During OSCE Quality assurance checklist (To be filled by QA
team)
Federal Ministry of Health
Health Professionals’ Competency assessment and Licensure Examination
Directorate

sr. Standards Verification criteria Yes No

No

1. Conduct of Students arrived at the examination site well before the commencement of
OSCEs the examination

Invigilators placed mobile phone signage in examination area prior to

commencement of exam

Identification (student ID) requested

Examination day Examination day Candidates briefing

briefing
Examination day Examiners briefing

Examination day SPs briefing

The responses to the test item are accurately recorded, handled, stored, and
analyzed.

Standardized scoring rubrics were used/ The raters evaluated the students
with a station-specific checklist

Trained examiners were used

Trained/Standardized patients were used

2. Running The movement of the candidates from one station to another is well
managed
the
circuit personnel available to ensure that the candidates move in the right direction
(Monitor was assigned)

The marking sheets are collected and the stations are reset for the following
run of the circuit as needed

The signal to change the station is clearly heard while at the same time not
be too noisy (Time keeper)
Annex IX: During-Written Examination Quality assurance checklist (To be
filled by QA team)
Federal Ministry of Health
Health Professionals’ Competency assessment and Licensure Examination
Directorate
Date:_____________________________
Exam center:________________________
Professional category:_________________

No. Standards Verification criteria Yes No

1. Exam Students arrived at the examination site well before the commencement
administration of the examination

Invigilators arrived at the examination site well before the

commencement of the examination

Invigilators placed mobile phone signage in examination area prior to

commencement of exam

Identification (student ID) requested and checked

Examination day Candidates briefing

Examination day Examiners briefing

All candidates get exam booklet and answer sheet on time

Trained Invigilators were used

The answer sheets are collected and the stations are reset for the
following run of the circuit as needed

The time for exam completion is clearly communicated (Time keeper)

QA team members:
Name Signature Date
_____________ _____________ _____________
_____________ ______________ _____________
______________ _____________ ______________
Annex X: Post-OSCE Quality assurance checklist (To be filled by QA team)
Federal Ministry of Health
Health Professionals’ Competency assessment and Licensure Examination
Directorate
Date:______________________________
Professional category:________________
No. Items Yes No
1. Mark sheets are collected and cross-checked for accuracy and any missing
scores

2. Examination team ratifies and signs the results based on stated period of time

3. The results are made available based on the stated period of time

4. The checklists and the questions used carefully stored for possible use in
future examinations

5. Mechanisms are in place for mitigating and dealt complaints and appeals

6. Analysis of the sources of error Cronbach’s alpha analysis was conducted

to measure internal consistency

The G-coefficient was calculated by

considering all facets

Coefficient of R2 analysis was performed

to measure the correlation between the
checklist score and the global rating for
individual stations

Between-group variation was computed

Difficulty and Discriminability index were

computed

7. Psychometrics Analysis finding were utilized to improve the quality OSCE

examination for subsequent uses

QA team members:
Name Signature Date
_____________ _____________ _____________
_____________ ______________ _____________
Annex XI: Post-Written Examination Quality assurance checklist (To be filled
by QA team)
Federal Ministry of Health
Health Professionals’ Competency assessment and Licensure Examination
Directorate
Professional category:______________
No Items Yes No
.
1.T The responses to the test item are accurately recorded, handled, stored, and analyzed.

2. Mark sheets are collected and cross-checked for accuracy and any missing scores

3. Examination team ratifies and signs the results based on stated period of time

4. The results are made available based on the stated period of time

5. The checklists and the questions used carefully stored for possible use in future
examinations

6. Mechanisms are in place for mitigating and dealt complaints and appeals
7. Analysis of the Cronbach’s alpha analysis was conducted to measure internal
sources of error consistency

The G-coefficient was calculated by considering all facets

Coefficient of R2 analysis was performed to measure the

correlation between the checklist score and the global rating
for individual stations

Between-group variation was computed

Difficulty and Discriminability index were computed

8. Psychometrics Analysis finding were utilized to improve the quality examination

for subsequent uses

QA team members:
Name Signature Date
_____________ _____________ _____________
_____________ ______________ _____________
______________ _____________ ______________
References
• MOH (2015). Implementation Guideline for National Licensing Examination of Health
Professionals. Addis Ababa. Ethiopia.
• Jhpiego (2015).Task analysis as a tool for health systems strengthening: an
implementation
guideline.USA.
• Kamran Z. K., Kathryn,G, Sankaranarayanan R.(2013). The Objective Structured Clinical
Examination (OSCE): AMEE Guide No. 81 Part 2: Organization & Administration.
Medical Teacher, 35:1447–e1463.
• James W., Abdelmoniem, E. M, Hamza, A., Imran, S. (2014). Objective Structured
Clinical Examination. Saudi Commission for Health Specialties