ARTICLE

Repeatability of measurements
with a double-pass system
Alain Saad, MD, Marc Saab, MD, Damien Gatinel, MD, PhD

PURPOSE: To evaluate the repeatability of measurements with a double-pass system.

SETTING: Rothschild Foundation, Paris, France.
METHODS: Eyes were separated into 2 control groups (<30 years old and >40 years), a post-refrac-
tive surgery group, and a cataract group. Measurements were performed using the Optical Quality
Analysis System. The main outcome measures were the objective scattering index (OSI), the cutoff
frequency of the modulation transfer function (MTF), and the Strehl ratio. The repeatability limit was
obtained from the individual standard deviations.
RESULTS: Forty-two eyes were evaluated. The mean OSI value was 0.47 G 0.11 (SD) in the youn-
ger control group, 1.73 G 0.26 in the older control group, 1.34 G 0.16 in the post-refractive sur-
gery group, and 6.15 G 0.50 in the cataract group. The mean cutoff MTF value was 39.44 G 3.93
cycles per degree (cpd), 26.07 G 3.89 cpd, 28.34 G 2.84 cpd, and 13.3 G 1.69 cpd, respectively,
and the mean Strehl ratio, 0.234 G 0.023, 0.146 G 0.021, 0.169 G 0.023, and 0.098 G 0.010,
respectively. The repeatability limit for the whole population was 0.841 (33.5%) for the OSI,
8.499 (31.1%) for the cutoff MTF, and 0.051 (31%) for the Strehl ratio.
CONCLUSIONS: The repeatability limit was good and equivalent for the OSI, the MTF, and the Strehl
ratio values. There was a wide interval between the normal and pathologic threshold for OSI mea-
surements, indicating that the reliability of the double-pass device complies with the requirements
for quantitative assessment of scattering.
Financial Disclosure: No author has a financial or proprietary interest in any material or method
mentioned.
J Cataract Refract Surg 2010; 36:28–33 Q 2010 ASCRS and ESCRS

The 2 leading causes of reduced optical quality of the only valid if ocular transparency is not altered.
human eye in clinical practice are uncorrected refrac- Another limitation of these systems is their low interre-
tive abnormalities and increased media opacities that producibility.2–5 The Optical Quality Analysis System
cause increased light diffusion.1 Aberrometers mea- (Visiometrics)1,6,7 is the only currently available device
sure optical aberrations of low and high degree but that allows direct objective measure of the effect of op-
not the loss of ocular transparency. Thus, the optical timal aberrations and the loss of ocular transparency on
quality estimation obtained with these aberrometers is the optical quality of the human eye. This double-pass
system performs these measurements by analyzing
the retinal image of a point source of light obtained after
focalization of an infrared beam. This retinal image cor-
Submitted: April 12, 2009.
responds to the point-spread function (PSF).
Final revision submitted: July 8, 2009.
The clinical applications of the double-pass system
Accepted: July 10, 2009.
are numerous. The system can be used in all clinical sit-
From the Rothschild Foundation (Saad, Saab, Gatinel), AP-HP Bi- uations in which it is important to quantify the reduc-
chat Claude Bernard Hospital (Saad, Saab, Gatinel), University Paris tion in the optical quality of the eye caused by an
VII, and the Center for Expertise and Research in Optics for Clini- increase in higher-order aberrations and a reduction
cians (Gatinel), Paris, France. in the transparency of the ocular media. The direct
Corresponding author: Damien Gatinel, MD, PhD, Fondation Oph- measurement of ocular light diffusion is potentially
talmologique Adolphe de Rothschild, 25, Rue Manin, 75019, Paris, relevant because it shows the objective effect of ocular
France. E-mail: gatinel@aol.com. media opacities on the light incident on the retina.

28 Q 2010 ASCRS and ESCRS 0886-3350/10/$dsee front matter

Published by Elsevier Inc. doi:10.1016/j.jcrs.2009.07.033
 REPEATABILITY OF DOUBLE-PASS MEASUREMENTS 29

It may be possible to confirm that mild opacities of the

lens are responsible for visual symptoms in a phakic
patient. Similarly, the role of posterior capsule opacifi-
cation in the loss of vision in a pseudophakic eye can
be confirmed by deterioration in the retinal PSF. More-
over, many studies1,8–11 have concluded that Hart-
mann-Shack aberrometers may overestimate image
quality in eyes affected by scattering (cataract, diffrac-
tive multifocal intraocular lenses).
Before relying on measurements by any device to
diagnose a measured abnormality, it is necessary to
ensure that repeated scans give consistent results.
Repeatability, as adopted by the International Organi-
zation for Standardization,12 is defined as a condition
in which independent test results are obtained with
the same method and equipment in the same subject
by the same operator with the shortest possible time Figure 1. Schematic of the double-pass aberrometry system (AP1 Z
artificial pupil 1; AP2 Z artificial pupil 2; BS Z beam splitter; C1 Z
between successive readings. camera 1; C2 Z camera 2; CCD Z charge-coupled device; L1 Z lens
To our knowledge, no comprehensive data on 1; L2 Z lens 2; L3 Z lens 3; L4 Z lens 4; PC Z personal computer).
the repeatability of measurements obtained with the
Optical Quality Analysis System have been published.
Thus, we performed a study to assess repeatability
it. Spherical refraction in the subject’s eye is performed at
of measurements obtained using the double-pass the focus corrector by modifying the optical paths between
system. lens 2 and lens 3. The eye forms the image of the point
source on the retina. The optical pathway from the laser
SUBJECTS AND METHODS source to the retina constitutes the single pass of the system.
The double pass is determined by the light on its way from
This study was performed according to the tenets of the Dec- the retina to a charge-coupled device (CCD) camera. The re-
laration of Helsinki. All subjects provided informed consent flected light passes through the 2 doublet lenses and through
after receiving an explanation of the nature and intent of the the beam splitter, where 50% of light is lost. Light that passes
study. To be included, the subject had to have 10 successive through the beam splitter encounters the second artificial
measurements in 1 or both eyes. pupil. The effective exit pupil is the second artificial pupil
The eyes were divided into 2 control groups, a post-re- or the natural pupil if it is smaller than the artificial pupil.
fractive surgery group, and a cataract group. The first con- An objective focuses the image on a CCD camera; a personal
trol group comprised eyes of subjects younger than 30 computer is used to grab and process the retinal images.
years who were emmetropic or ametropic with spectacle The double-pass system provides several measurements.
correction and with normal distance visual acuity (20/20 The first is the modulation transfer function (MTF), which
or better). Corneal regularity and the absence of topo- evaluates the ratio between the contrast in the retinal image
graphic abnormalities were confirmed by Placido-based of a sinusoidal grating and its original contrast as a function
corneal topography. The second control group comprised of the spatial frequency of the grating. The MTF value is the
eyes of subjects older than 40 years with no lens opacities highest when the contrast in the image is the same as the con-
detectable at the slitlamp, a corrected distance visual acuity trast in the object. The value considered is the cutoff point of
(CDVA) of 20/20 or better, regular corneas, and no topo- the MTF curve (cutoff MTF) on the x-axis; the results are
graphic abnormalities. The post-refractive surgery group given in cycles per degree (cpd), representing the highest
comprised eyes that had uneventful laser in situ keratomi- spatial frequency at lower contrast. The classic definition of
leusis and 20/20 uncorrected distance visual acuity. To the cutoff frequency is an MTF value of zero. The MTF cutoff
be included in 1 of the first 3 groups, the eye had to be in the double-pass system is the frequency at which the MTF
free of ocular anomalies. The cataract group comprised reaches a value of 0.01. Because the PSF images recorded by
eyes with impaired visual acuity caused by moderate to the double-pass instrument can be affected by high-fre-
severe cataract. quency noise, which is inherent in the use of cameras, the fre-
The same experienced investigator performed all proce- quency for very small MTF values may become unstable,
dures. No medication was given to dilate the pupils, and potentially leading to artifacts. To avoid this problem, the de-
no patient received topical drops or ointment. vice uses an MTF threshold value of 0.01, which corresponds
to 1% contrast. Thus, the cutoff MTF in this paper refers to
the frequency up to which the eye can image an object in
Double-Pass System the retina with a significant 1% contrast. Under optimum
The light source of the double-pass system is a 780 nm la- conditions (low level of optical aberration and diffraction),
ser diode, which acts as a point object (Figure 1). After re- the maximum spatial frequency the human eye can detect
flecting through a beam splitter, the light passes through 2 is close to 60 cpd (the limit imposed by the retinal sampling).
achromatic doublet lenses (lens 2 and lens 3) and through The second measurement is the objective scattering index
a mobile focus corrector, which has 2 mirrors attached to (OSI), which is an objective evaluation of intraocular

J CATARACT REFRACT SURG - VOL 36, JANUARY 2010

30 REPEATABILITY OF DOUBLE-PASS MEASUREMENTS

Table 1. Demographic characteristics by group.

Group

Characteristic Control !30 Y Control O40 Y Post Refractive Surgery Cataract

Subjects (n) 8 6 4 6
Eyes (n) 15 9 7 11
Mean age (y) G SD 27.5 G 2.8 53.1 G 6.9 27.6 G 4.1 67.3 G 13.7
Male sex, n (%) 3 (37.5) 2 (33.3) 3 (75.0) 3 (50.0)
CDVA
Value 1.0 1.0 1.0 0.46 G 0.17*
Range d d d 0.05 to 0.60

CDVA Z corrected distance visual acuity

*Mean G SD

scattered light. The index is calculated by evaluating the repeatability standard deviation. This value is useful in de-
amount of light outside the double-pass retinal intensity termining a repeatability limit; a value less than or equal to
PSF image in relation to the amount of light on the center. the absolute difference between 2 test results obtained under
The higher the OSI value, the higher the level of intraocular repeatability conditions can be expected to lie within a prob-
scattering. According to the user’s manual, the OSI value is ability of 95%.
lower than 0.5 in eyes with a normal degree of scattering The repeatability limit was calculated from the individual
(young eyes), between 1.5 and 4.0 in eyes that are developing standard deviations as follows:
cataract, and higher than 4 in eyes with mature cataract.
The third measurement is the Strehl ratio, which is an ex- RZSDt0:05;n
pression of the ratio of the central maximum of the illumi- where R is the repeatability limit, SD is standard deviation,
nance of the PSF in the aberrated eye to the central and t is the critical value of the Student t distribution at the
maximum that would be found in the corresponding aberra- 95% confidence level (t Z 2.262 for 10 measurements).13
tion-free system. It is the measure of the fractional drop in the The mean repeatability limit in the population was calcu-
peak of the PSF as a function of the wavefront error. A Strehl lated by adding the square of individual repeatability of each
ratio of 1 indicates perfection. individual eye and calculating the root mean square of the
mean value as follows:
Measurement Technique

RZO R2 þR2 þ..þR2 þR2 N
The head of the subject was positioned on the chin rest and
fixated on the center of a figure. The operator manually where N is the number of subjects in the study population.
aligned the subject’s pupil center with the optical axis of Repeatability is given with 95% confidence. In the remaining
the device. Spherical correction within 11.00 and C5.00 di- sections of the paper, the term repeatability is used as equiv-
opters (D) of ametropia was automatically performed by the alent to the confidence interval of repeatability correspond-
machine. Cylindrical refractive errors were corrected with ing to the range of random errors determined at the 95%
a cylindrical trial lens. Twenty consecutive measurements confidence level. Standard deviation and repeatability limit
were taken (10 OSI; 10 MTF and Strehl ratio); the pupil center are expressed in absolute values and in percentage of the
was realigned between each measurement. Subjects were mean values of each tested parameter in the study
asked to blink before the measurement. population.
For each parameter (OSI, MTF, and Strehl ratio), the de-
vice took 6 measurements. It then calculated the mean of
the measurements to provide the final results for each pa- RESULTS
rameter. The operator can ignore 1 or more of the 6 measure-
ments judged to be atypical to allow the machine to calculate
Forty-two eyes of 24 subjects were enrolled in the
a more accurate final result from the remaining measure- study. Table 1 shows the demographic data of the sub-
ments. However, this option was not used during our study jects and the number of subjects in each group. The
(ie, all 6 measurements were always kept). mean age of the subjects was 44.4 years G 18.9 (SD).
The mean spherical equivalent of the attempted refrac-
Repeatability Calculation tive correction in the post-refractive surgery group
Repeatability is the closeness of agreement between the was 3.1 G 1.4 D.
results of successive measurements of an identical test mate- Table 2 shows the mean values for OSI, cutoff MTF,
rial performed under defined conditions. Conditions include and Strehl ratio and the repeatability limit of each. The
the same operator, same apparatus, and a short time be- OSI was higher in the older control group (O40 years)
tween analyses. The conditions under which these measure-
ments were performed are known as the repeatability than in the younger control group (!30 years). The
conditions. The results of the repeatability experiments can cutoff MTF and the Strehl ratio decreased with age
be used to calculate a standard deviation, called the (Figure 2). There was a statistically significant

J CATARACT REFRACT SURG - VOL 36, JANUARY 2010

 REPEATABILITY OF DOUBLE-PASS MEASUREMENTS 31

Table 2. Repeatability OSI, cutoff MTF, and Strehl ratio values.

Group

Parameter Control !30 Y Control O40 Y Post Refractive Surgery Cataract All

Mean values G SD
OSI 0.47 G 0.11 1.73 G 0.26 1.34 G 0.16 6.15 G 0.50 2.51 G 0.28
Cutoff MTF (cpd) 39.44 G 3.93 26.07 G 3.89 28.34 G 2.84 13.3 G 1.69 27.29 G 3.23
Strehl ratio 0.234 G 0.023 0.146 G 0.021 0.169 G 0.023 0.098 G 0.010 0.165 G 0.019
Repeatability limit (%)
OSI 0.26 (56.1) 0.63 (36.5) 0.4 (29.7) 1.28 (20.9) 0.84 (33.5)
Cutoff MTF 9.54 (24.2) 9.62 (36.9) 6.79 (24.0) 5.11 (38.4) 8.49 (31.1)
Strehl ratio 0.053 (22.6) 0.053 (36.6) 0.059 (34.8) 0.030 (30.6) 0.051 (31.0)

cpd Z cycles per degree; MTF Z modulation transfer function; OSI Z objective scattering index

difference in OSI values between the 2 control groups The precision (repeatability, reliability) indicates the in-
and between the younger control group and the post- strument’s ability to repeat its own results.14 Assessing
refractive surgery group (P!.001, Kruskal-Wallis). the accuracy of the Optical Quality Analysis System dou-
There was no significant difference in OSI values be- ble-pass device, defined as the trueness of the results,
tween the older control group and the post-refractive was not the primary goal of this study. However, accu-
surgery group (P Z .138, Kruskal-Wallis) The repeat- racy testing is dependent on the repeatability of the de-
ability limit (percentage of mean value) ranged be- vice, which we did study.
tween 20.9% and 56.1% for the OSI, between 20.4% The calibration of an instrument against known stan-
and 38.4% for the cutoff MTF, and between 22.6% dards eliminates systematic errors. The errors associ-
and 32.6% for the Strehl ratio. Thus, the limit was al- ated with routine use of an instrument are random;
most constant for the whole population, varying be- these can be minimized by a detailed routine procedure
tween 33.5% for the OSI, 31.1% for cutoff MTF, and and the use of repeated independent measurements.
31% for the Strehl ratio. The determination of random errors leads to the iden-
tification of instrument measurement repeatability.
DISCUSSION Measurements of the OSI, cutoff MTF, and Strehl ra-
tio by the double-pass system had a good repeatability
Measurement validity or accuracy is dependent on 2
limit. For comparison, the repeatability limit of the
types of measurement uncertainties: systematic errors
Zernike coefficient computed for corneal wavefront re-
and random errors. The accuracy (validity) of an
construction (pupil diameter of 4.5mm) varied in one
instrument indicates the closeness between the mean
study between 24% and 231%4; a repeatability limit
measured value and the true value of each measurement.
of 50% is the highest acceptable value in biological
metrics.4 Because the standard values indicated by
manufacturer of the double-pass system (OSI !0.5
for normal eyes, between 1.5 and 4.0 for mild to mod-
erate cataract) specify a free interval between the nor-
mality and pathology thresholds higher than the
repeatability limit, reliability of the double-pass mea-
surements should not be affected. Therefore, the OSI
results in normal young eyes did not reach a pathologic
level, suggesting measurement consistency. In the nor-
mal group, the lowest cutoff MTF value was 14.5 cpd.
This value is much lower than expected for a 46-year-
old patient with a CDVA of 20/20. Although the MTF
cutoff frequency with the system is defined by the fre-
quency at which the MTF reaches the value of 0.01 (not
Figure 2. Objective scattering index (OSI) and cutoff MTF (cf MTF) as
0 as in the classic definition), high-frequency noise
a function of age in normal eyes (2 control groups) (MTF Z modu- from the instrument’s camera may limit the precision
lation transfer function). of the calculation of the cutoff frequency value.

J CATARACT REFRACT SURG - VOL 36, JANUARY 2010

32 REPEATABILITY OF DOUBLE-PASS MEASUREMENTS

Repeatability error can be caused by operator- or pa- the double-pass system. Because the system we used
tient-dependent factors. We thought that it would be is based on a double pass of light through the eye’s op-
interesting to eliminate aberrant measurements from tics, the image is twice degraded. Even in a ‘‘perfect’’
the 6 captures performed by the machine during the eye, the PSF measurement would not be an Airy
initial acquisition to evaluate the effects on the instru- diffraction pattern; in addition, for the same pupil di-
ment’s repeatability. We applied this method in the ameter on the 2 passes, the derived MTF would corre-
young control group and found no difference in the fi- spond to the square of the true ocular MTF. Thus,
nal result; 5 of the 6 OSI measurements were always caution should be used when comparing our MTF
identical. The different measurement (present in 4 of measurements with those obtained with other devices.
8 eyes) did not differ by more than 0.1 from the other Based on its numerous clinical applications, we be-
5 measurements. Thus, the mean value of the 6 mea- lieve the double-pass system will play an important
surements remained unchanged. A second source of role in daily clinical practice. Future studies should as-
error is noncycloplegic fixation, which can lead to ac- sess the normal range of values in eyes with various
commodative spasm in the last measurements. Fi- pathologies and evaluate the accuracy of the double-
nally, any variation in the refractive formula used pass system in these cases.
during the acquisition (patient’s glasses, trial frame
lenses) could affect the final results. In this study, the
10 measurements were performed using the same REFERENCES
method of correction (trial lenses); thus, this variation 1. Dı́az-Doutón F, Benito A, Pujol J, Arjona M, Güell JL, Artal P.
did not affect the repeatability but may have affected Comparison of the retinal image quality with a Hartmann-Shack
the accuracy of the results. The possible variation in wavefront sensor and a double-pass instrument Available at.
pupil diameter with time, individual, or age is known Invest Ophthalmol Vis Sci 2006; 47:1710–1716. http://www.
iovs.org/cgi/reprint/47/4/1710. Accessed September 23, 2009
to affect MTF measurements. However, the effective 2. Davies N, Diaz-Santana L, Lara-Saucedo D. Repeatability of
exit pupil was fixed at 4.0 mm in our study and no ocular wavefront measurement. Optom Vis Sci 2003; 80:142–
eye had a pupil smaller than 4.0 mm during the mea- 150
surements. Thus, pupil variation did not affect the re- 3. Ginis HS, Plainis S, Pallikaris A. Variability of wavefront aberra-
peatability or reliability of the results. tion measurements in small pupil sizes using a clinical Shack-
Hartmann aberrometer Available at. BMC Ophthalmol 2004;
With almost every measuring device, the smaller the 4:1. Available at: http://www.pubmedcentral.nih.gov/picrender.
absolute value of the measured parameter, the more fcgi?artidZ362876&blobtypeZpdf. Accessed September 23,
considerable the possible relative repeatability error. 2009
Because most OSI values in our young control group 4. Gobbe M, Guillon M, Maissa C. Measurement repeatability of
population were close to zero, the high repeatability corneal aberrations. J Refract Surg 2002; 18:S567–S571
5. Lewis CD, Krueger RR. Reproducibility of wavefront measure-
error (56.1%) we observed may be the logical conse- ments using the LADARWave aberrometer. J Refract Surg
quence of the low absolute values. Despite a low abso- 2006; 22:S973–S979
lute value, the repeatability limit of the Strehl ratio was 6. Ortiz D, Alió JL, Ruiz-Colechá J, Oser U. Grading nuclear cata-
very good. Because this parameter is correlated only ract opacity by densitometry and objective optical analysis. J
with the relative maximum height of the PSF intensity, Cataract Refract Surg 2008; 34:1345–1352
7. Güell JL, Pujol J, Arjona M, Diaz-Douton F, Artal P. Optical Quality
it may be less affected by instrument variability than Analysis System: instrument for objective clinical evaluation of oc-
the OSI and cutoff MTF. The repeatability limit was ular optical quality. J Cataract Refract Surg 2004; 30:1598–1599
not affected by age or other specific conditions (ie, pre- 8. Gatinel D. Limited accuracy of Hartmann-Shack wavefront sens-
vious refractive surgery or presence of cataract). Ex- ing in eyes with diffractive multifocal IOLs [letter]. J Cataract Re-
cept in the young control group, the OSI varied fract Surg 2008; 34:528; reply by Toto L, Falconio G, Vecchiarino
L, Scorcia V, Di Nicola M, Ballone E, Mastropasqua L, 528–529
between 20.9% and 38.4%. We found that the OSI 9. Charman WN, Montés-Micó R, Radhakrishnan H. Problems in
and the cutoff MTF were correlated with age. Scatter- the measurement of wavefront aberration for eyes implanted
ing increased with age, whereas the cutoff MTF de- with diffractive bifocal and multifocal intraocular lenses. J Refract
creased with age. This can be caused by an increase Surg 2008; 24:280–286
in corneal higher-order wavefront aberrations in older 10. Charman WN, Montés-Micó R, Radhakrishnan H. Can we mea-
sure wave aberration in patients with diffractive IOLs? J Cataract
age15 or by decreased ocular transparence that is unde- Refract Surg 2007; 33:1997
tectable at the slitlamp. Methods to estimate scattering 11. Campbell CE. Wavefront measurements of diffractive and
in Hartmann-Shack raw images have been proposed, refractive multifocal intraocular lenses in an artificial eye. J Re-
one of which is to calculate the brightness of pixels fract Surg 2008; 24:308–311
within an area containing each lenslet’s PSF tail.16 It 12. International Organization for Standardization. Accuracy (True-
ness and Precision) of Measurement Methods and Results. Part
would be interesting to compare the accuracy of mea- 2. Basic Methods for the Determination of Repeatability and Re-
surements of the scattering level obtained with these producibility of a Standard Measurement Method. Geneva, Swit-
methods with the accuracy of those obtained with zerland, ISO, 1994; (ISO 5725-2)

J CATARACT REFRACT SURG - VOL 36, JANUARY 2010

 REPEATABILITY OF DOUBLE-PASS MEASUREMENTS 33

13. Zar JH. Biostatistical Analysis 2nd ed. Englewood, NJ, Prentice- to evaluate nuclear cataract. J Refract Surg 2004; 20:S515–
Hall, 1984; 4847 S522
14. International Organization for Standardization (ISO). Statistics
Vocabulary and Symbols. Geneva, Switzerland, ISO, 1977;
ISO 3534 First author:
15. Amano S, Amano Y, Yamagami S, Miyai T, Miyata K, Alain Saad, MD
Samejima T, Oshika T. Age-related changes in corneal and Rothschild Foundation, Paris, France
ocular higher-order wavefront aberrations. Am J Ophthalmol
2004; 137:988–992
16. Donnelly WJ III, Pesudovs K, Marsack JD, Sarver EJ,
Applegate RA. Quantifying scatter in Shack-Hartmann images

J CATARACT REFRACT SURG - VOL 36, JANUARY 2010