How To Read A Paper - Critical Appraisal
How To Read A Paper - Critical Appraisal
How To Read A Paper - Critical Appraisal
Mark Kerr
Clinical Librarian, EKHUFT
Learning objectives
By the end of this session you will be able to:
• critically appraise research papers
• identify internal and external validity and consider usefulness
• identify different study designs and know which type of
research question each relates to
• understand and identify sources of bias, confounding and other
weaknesses which may affect the validity or applicability of
their results
• interpret the key statistics in a research paper
• consider teaching critical appraisal with greater confidence
Critical appraisal definitions
“EBP requires that decisions about health and social care are
based on the best available, current, valid and relevant
evidence. These decisions should be made by those receiving
care, informed by the tacit and explicit knowledge of those
providing care, within the context of the available resources”
Dawes et al. 2005 p.7
What is critical appraisal?
Critical > Not “criticise” but “critique”
CA is not just: CA is:
Negative dismissal Balanced assessment
Assessment of results Test of processes
Statistical Analysis Look at all aspects
Task for experts Task for all ‘users’
1.
1. SCREEN
SCREEN 2.
2. APPRAISE
APPRAISE 3.
3. ASSESS
ASSESS 4.
4. APPLY
APPLY
IRRELEVANCE PICO
BIAS RAMMBO
REPRODUCABILITY
Similar studies,
meta-analysis
Evidenced by:
Essential first questions to ask of a
paper
• Who was the research carried out by
(organisation/individuals)
• Which journal was the research published by?
• When was the research carried out and when was it
published?
• Where is the research based?
- relevant questions, but are they critical appraisal?
Study Design
In a Nutshell . . .
1. PICO - if relevant, then read it
2. Difference due to chance? Due to bias?
- If not then likely to be due to intervention
3. Is it different enough to be clinically useful?
- Is it ‘real’ (patient-oriented) or surrogate outcome?
4. Does it change/inform knowledge/practice?
- Can I apply this to my patient?
1. Similar population
2. Outcomes important and all
Will they help me care for my patient? considered
3. Patient preferences and values
Are the chosen outcomes valid?
• A biomarker is a characteristic that is objectively
measured and evaluated as an indicator of normal biological
processes, pathogenic processes, or pharmacologic
responses to a therapeutic intervention.
• A clinical endpoint is a characteristic or variable that
reflects how a patient feels, functions, or survives.
• A surrogate endpoint is a biomarker that is intended to
substitute for a clinical endpoint:
• Cholesterol down = risk down =mortality down?
• Skin thickness = skin condition?
• Exercise tolerance = risk of MI?
• Patient-oriented vs research-oriented outcomes
Hierarchy of Evidence
Randomised Controlled Trial
Experimental Causation RR, OR, ARR, NNT
Sample
Analysis
Population
Selection
& screening Control Arm Outcomes
Good Outcome
Treatment/Exposure
Bad Outcome
Identify Classify
Study Treatment
Subjects Status
Good Outcome
Control
Bad Outcome
Recruitment Time Data Collection
Prospective Cohort Study
Data Collection Recruitment
Retrospective Cohort Study
Key biases to identify/avoid: selection (of controls), recall (if retrospective), information
Case Control Study
Observational Association + Time OR
Treatment or Exposure A
Cases
Treatment or Exposure B
Study
Population
Treatment or Exposure A
Controls
Treatment or Exposure B
• Matched
• Each case is matched for some relevant characteristics (eg age, gender,
ethnicity, comorbidities), to minimise confounding
• Nested
• Where the study population is drawn from within an arm of a cohort study
Cross-Sectional Study
Observational Prevalence OR
Study Population
Survey Sample
Preliminary Protocol/
Search Proposal
Key biases to identify/avoid: author, publication, outcome selection, data extraction, also check
for heterogeneity
Choose/Identify Design
• A study that aims to establish the normal height of 4yr old children by
measuring height at school entry Cross-sectional
• A study that compares a group of children whose heights are below the
tenth centile with a group of matched controls of normal height aiming
to identify possible causative factors Case-control
• A study that compares the height of a group of 4yr olds living near a
nuclear plant with the height of a group of 4yr olds who live elsewhere
Cohort
• A study that looks at all children born at one hospital in one Longitudinal
year and
measures their height at intervals up to four years of age
Bias, Confounding, Limitations
Assessing the risk of
bias in studies
What makes a study
questionable?
• Confounding Factors
– Underlying factors that can affect the results, which
are outside the control of the research team. For
example, age, comorbidities, etc.
• Systematic Bias
– Mistakes made by the investigators (not necessarily
intentional) that result in a false conclusion
Selection bias at Recruitment
– is it a representative sample?
• Undercoverage
• convenience sampling (clinic, registered lists) misses the inconvenient!
• Non-response
• respondents differ from non-respondents, higher health literacy, social visibility
• Hospital Admission rate
• patients more likely to have other conditions
Selection/Sampling Bias
The survey sample does not accurately represent the population
• Exclusion
• criteria used to reduce confounding not applied equally to control group
• Publicity / Awareness
• The Jade Goody, Michael J Fox, Terry Pratchett effect – rise in ‘incidence’ or
reporting due to greater visibility of condition
• Voluntary Response /Self-selection
• favours those with strong opinions, more informed
Information Biases
Accuracy of information about the exposure differs between cases and controls
• Interviewing
• Interviewers more thorough with cases than controls
• Abstracting
• Same as interviewing, but done when retrieving case data from records – limited
data for controls
• Ascertainment/Surveillance
• Caused by patient or clinician knowing which group they are in, by more intense
investigation in cases compared to controls
Information Biases
Accuracy of information about the exposure differs between cases and controls
• Recall
• past exposures and events better recalled than non-events; memory fallibility,
especially in consumption of alcohol, caffeine, tobacco...
• Reporting
• When a case emphasises exposure/symptoms that they believe to be important
• Treatment bias
• Difference in any element of treatment between cases & controls – sham surgery,
transport etc
Fixing Biases
• Bias is especially important in observational studies, as they lack the
random sampling and random allocation of RCTs, leading to greater
potential for error.
• Increasing sample size cannot compensate for selection/survey biases
• Blinding helps with interviewing, reporting and treatment biases,
cannot ‘fix’ sampling biases
Designing-out confounders
• Restriction
• Inclusion/exclusion criteria to prevent confounders entering
sample population
• Matching
• Allocating subjects with confounding factors equally across
both/all arms of the study
• Randomisation
• Appropriate randomisation should distribute those with
confounding factors (known AND unknown) among the study
groups (cluster randomisation might concentrate them!)
It’s all about evidence