Abstract
This experiment investigated whether there might be an effect on the visual search strategy of radiologists during image interpretation of the same adult chest radiographs when given different clinical information. Each of 17 experienced radiologists was asked to interpret a set of 57 (10 abnormal) posteroanterior chest images to identify the presence of pulmonary lesions using differing clinical information (leading to unknown, low and high expectations of prevalence). Eye position metrics (search time, dwell time and time to first fixation) were compared for normal and abnormal images, as well as between conditions. For all images, there was a significantly longer search time at high prevalence expectation compared to low prevalence expectation (W = 75.19, P = <0.0001). Mann–Whitney analysis of the abnormal images demonstrated that the dwell time on correctly identified lesions was significantly shorter at low prevalence expectation compared to both unknown (U = 364.5, P = 0.02) and high prevalence expectation (U = 397.0, P = 0.0002). Visual search patterns of radiologists appear to be affected by changing a priori information where such information fosters an expectation of abnormality.
Keywords: Nodule, Lung, Eye-tracking, Search
Introduction
The detection of pulmonary nodules is essential for the early diagnosis of lung cancer. A false-negative (FN) diagnosis due to a perception or cognitive error is a recognised problem [1]. Evidence from earlier eye-tracking experiments using chest images reveals that most radiologists do not follow a systematic search strategy [2, 3], and interpretation of these images can be influenced significantly by the clinical information provided by referrers [4–7]. Clinical information is provided as a guide for patient diagnosis and can be disease specific, symptom specific, location specific or a combination of these. Although clinical history can assist in diagnostic decision-making [8], it may possibly modify visual search by altering a radiologist’s expectations [9]. Expectations are mind-sets that reflect a priori information about what is possible or probable in a future sensory environment [10]. Whilst a reader’s attention may be directed to the area of interest by the given information, thus increasing the accuracy of the diagnosis, such information may also cause the reader to have diagnostic expectations about the image leading to possible error. In addition, once an area of suspicion has been observed, some readers may use the clinical information to confirm the findings whilst others might use it to question the findings. In summary, a lesion might be missed because it was not expected and conversely, the reader might identify a finding where none exists based on an expectation that a lesion should be present.
Unlike other experiments, which have contrasted experts and naïve observers [11–13], this is the first, to the authors’ knowledge, to compare the effect of varying clinical information on visual search behaviour in radiology. In this experiment, one of the clinical conditions suggested a low expectation of abnormality (visa application), one suggested a high expectation (previous cancer) and one gave no clinical information at all suggesting an unknown expectation of abnormality. We deliberately employed a general non-specific clinical history in two of the three conditions to encourage the radiologists to formulate their own expectations. Swensson et al. [14] hypothesised that errors may be attributed to faulty search and Berbaum et al. [15] suggested that clinical prompts can influence search patterns, which can lead to a positive effect in the perception of certain abnormalities but a negative effect in others. Recent research has demonstrated changes in performance with a significant decrease in specificity at the higher abnormality expectation condition although sensitivity was unaffected [16]. The purpose of this study was to determine the effect expectation of disease prevalence has on the visual search strategies of radiologists when viewing adult chest images.
Materials and Methods
Institutional ethics approval for this study was granted (_2012/981_)
Subjects
Seventeen experienced radiologists, with a minimum of 11 years and a maximum of 27 years post-certification with the American Board of Radiology (ABR), were involved in the study. Seven of these were thoracic radiologists and deemed experts.
Image Bank
The same dataset of 57 adult posteroanterior (PA) digital chest images (high resolution (2048 × 2048 matrix size, 0.175 mm pixel size) was used for all three conditions described below. The images were selected from a dataset created by the Japanese Society of Radiological Technology (JSRT) in cooperation with the Japanese Radiological Society (JRS) [17]. The lung nodules were categorised according to the degree of subtlety from 5 (obvious) to 1 (extremely subtle), and nodule presence or absence was validated by 20 radiologists (not involved in this study) using computed tomography. The test set consisted of 47 normal images and 10 abnormal images, where each of the latter contained a single pulmonary nodule. Nine of these single nodules had a subtlety categorisation of three and one image contained a category four nodule. Six nodules were located in the left lung and four in the right lung (Table 1). Only one condition was read in each reading session.
Table 1.
Location and size of nodules on the 10 abnormal images
| Case | Conspicuity | Size (mm) | Size (pixels) | Location |
|---|---|---|---|---|
| 1 | 4 | 10 | 35.70 | Lt lower lobe |
| 2 | 3 | 26 | 92.82 | Lt lower lobe |
| 3 | 3 | 14 | 49.98 | Lingula |
| 4 | 3 | 15 | 53.55 | Lt lower lobe |
| 5 | 3 | 23 | 82.11 | Rt lower lobe |
| 6 | 3 | 8 | 28.56 | Rt upper lobe |
| 7 | 3 | 13 | 46.41 | Lingula |
| 8 | 3 | 26 | 92.82 | Rt upper lobe |
| 9 | 3 | 25 | 89.25 | Rt middle lobe |
| 10 | 3 | 12 | 42.84 | Lt upper lobe |
Viewing
Images were displayed on a Viewsonic VG810b monitor (ViewSonic, Walnut, Calif) with a screen resolution of 1280 × 1024 pixels using a graphics card (NVIDIA Quadro FX 560; Nvidia, Santa Clara, California) that exceeded the minimum recommendation by the American Association of Physicists in Medicine [18]. On each day of the study, the monitor was calibrated to the Digital Imaging and Communications in Medicine grey-scale display function standard using Verilum software and luminance pod (Verilum; Image Smiths, Bethesda, MD). Ambient light remained within 35–40 lux, as measured with a calibrated photometer (model 07–631; Nuclear Associates, Everett, WA).
The visual search behaviour of each participant was monitored using a remote eye tracker Tobii ×50. The eye tracking system measures the relative position of the pupil and the corneal reflection in order to identify the gaze direction. The accuracy of this system is 0.5° of visual angle, the minimum dwell time for recording a fixation is 20 ms and the system sample rate is 50 Hz. The minimum fixation duration was set to 100 ms with a fixation radius of 50 pixels using Tobii Studio 2.2.8 software (Tobii AB, Danderyd, Stockholm, Sweden). This software allows the creation of gaze plots, screen video recordings and demarcations of areas of interest on the test images. The Tobii eye tracker (Fig. 1) is a standalone remote eye tracking device and does not require the user to wear any special head or eye equipment. Instead, the user’s eye positions are tracked via infrared sensors located in the bottom of the Tobii PC monitor.
Fig. 1.
The distance from the user to the eye tracker is 55–60 cm; the eye tracker is positioned below the stimuli. The angles from the eye tracker to the user to the stimulus must be within ±35°
Readers and Image Conditions
There were three conditions that used different clinical information. Individual radiologists were not told in advance which condition they would read. Radiologists were randomly assigned to one of the three workstations for the test.
Six readers (two experts) undertook condition 1 and were given the following information: “All these patients have had a previous primary malignancy” (high prevalence expectation).
Six readers (two experts) undertook condition 2 and were offered no clinical information (unknown prevalence expectation).
Ten readers (three experts) undertook condition 3 and were told that “All these patients had a routine chest x-ray for a visa application” (low prevalence expectation).
All images were presented in an individual reading session in a single condition.
Reader Instructions
Each radiologist was asked to identify the location(s) of suspected lung nodules on 57 PA adult chest images under one of the previously 3 stated conditions. After identification, each nodule was scored using a five-point scale. A higher score indicated increased confidence that a lesion was present and a score of one was automatically assigned by the system when the radiologist reported no lesion (normal case). The radiologists were not told the number of lesions that might be present on each image nor were they informed of the ratio of normal to abnormal images or the age or gender of each patient. The chest images were presented in random order for each reading session, and the radiologists were allowed to zoom, pan and window the images. No time restriction was imposed. The subjects were unaware of the purpose of the experiment and unaware that the same image dataset would be used for each condition. The radiologists were not given any information regarding the pathologies that were present.
Analysis
A true positive (TP) was identified as any mouse click within a 50-pixel acceptance radius from the centre of the nodule as identified in the database; otherwise, the mark was scored as a false positive (FP). All nodules were smaller than the 100-pixel diameter designated as the region of interest. Haygood et al. [19] concluded that a 50-pixel radius was an optimal acceptance radius for nodule perception experiments where the largest nodule size was 100 pixels in diameter. Any nodule that was not marked was deemed a false negative (FN). The results for each condition were compared with each of the other conditions. For abnormal images (n = 10), visual search was evaluated based on the time to first fixation and the dwell time on the location of the lesion, regardless of whether it was reported (TP) or not (FN). The dwell time was defined as the time in seconds the reader scrutinised a region of interest. Time to first fixation on FP locations was also analysed on abnormal (n = 10) and normal images (n = 47) and the total time spent viewing all cases (n = 57) over all three conditions. The search time was defined as the time in seconds from when the radiograph was presented until the readers completed their examination of the image. The time to first fixation was defined as the time in seconds from when the image was presented until the start of the first fixation within an area of suspicion (TP, FP or FN). Kruskal-Wallis and ANOVA post hoc analysis were conducted to look for any significant differences between the three conditions. Kruskal-Wallis and ANOVA post hoc analysis were also employed to compare the performance between each of the three groups of readers.
Comparison between the time to first fixation and the search time on abnormal and normal images and comparison of the dwell time and time to first fixation between TPs and FNs employed the non-parametric Mann-Whitney test using GraphPadPrism version 5.00 for Windows (GraphPad Software San Diego California USA) (www.graphpad.com).
Results
Table 2 demonstrates the performance of each of the three reading groups. Analysis demonstrates no significant difference between the ability of each group (P = 0.66) Table 3 demonstrates the total time spent reading all of the cases (n = 57) and also the time to first fixation on a suspected lesion under each of the three search conditions. There was significant increase in the time (W = 75.19, P = <0.0001) taken to view the high expected abnormality (cancer) and unknown expected abnormality conditions when each were compared with the low expected abnormality (visa) condition. No significant difference in search time was seen when comparing unknown expected abnormality with high expected abnormality. The time to first fixation was examined. This was not significantly different among the three conditions (W = 5.311, P = 0.07) although the data suggested that first fixation was borderline significantly faster at low prevalence expectation.
Table 2.
JAFROC performance comparison between each group of readers
| Cancer group (n = 6) | No history group (n = 6) | Visa group (n = 10) | |
|---|---|---|---|
| Median | 0.6919 | 0.6122 | 0.6419 |
| Interquartile range | 0.2034 | 0.1952 | 0.2865 |
| Kruskal-Wallis statistics | 0.8371 | ||
| P value | 0.66 | ||
The 25 and 75 % percentiles are shown as is the Kruskal-Wallis test statistic and P value for the differences between each of the three conditions
Table 3.
The median search duration on all images and the median time to first fixation on any suspicious lesion seen on all images (N = 57)
| Parameter | Low abnormality expectation | Unknown abnormality expectation | High abnormality expectation | P value | Kruskal-Wallis test statistics |
|---|---|---|---|---|---|
| Search time on all cases | 17.88 | 27.78 | 28.49 | P = <0.0001 | W = 75.19 |
| Interquartile range | 19.70 | 25.41 | 19.53 | ||
| Time to first fixation | 3.53 | 4.51 | 4.24 | P = 0.07 | W = 5.311 |
| Interquartile range | 6.51 | 9.79 | 8.85 |
The 25 and 75 % percentiles are shown as is the Kruskal-Wallis test statistics and P value for the differences between each of the three conditions. All times are in seconds
Table 4 focuses upon the abnormal images (n = 10) with the Kruskal-Wallis analysis of the dwell times on the lesion and the time to first fixation on the lesion. There was no significant difference in the time to first fixation for any of the three conditions on FPs, TPs and FNs.
Table 4.
The median dwell time on all abnormal images (N = 10) and the median time to first fixation on any suspicious lesion
| Parameters (time is in seconds) | Low abnormality expectation | Unknown abnormality expectation | High abnormality expectation | P value | Kruskal-Wallis test statistics |
|---|---|---|---|---|---|
| Dwell time on all lesions | 0.32 | 1.16 | 2.64 | P = <0.0001 | W = 21.55 |
| Interquartile range | 1.29 | 3.71 | 4.78 | ||
| Dwell time on false negatives | 0.22 | 0.48 | 0.30 | P = 0.03 | W = 6.87 |
| Interquartile range | 0.54 | 1.29 | 0.87 | ||
| Dwell time on true positives | 0.58 | 1.77 | 3.47 | P = 0.0007 | W = 14.42 |
| Interquartile range | 2.10 | 4.68 | 4.42 | ||
| Time to first fixation on lesion | 3.42 | 5.29 | 5.04 | P = 0.39 | W = 1.88 |
| Interquartile range | 5.92 | 8.82 | 7.96 | ||
| Time to first fixation on a false positive | 2.96 | 4.53 | 3.54 | P = 0.08 | W = 5.04 |
| Interquartile range | 5.01 | 8.06 | 8.08 | ||
| Time to first fixation on a true positive | 3.28 | 5.32 | 3.90 | P = 0.19 | W = 3.32 |
| Interquartile range | 4.50 | 6.95 | 8.66 | ||
| Time to first fixation on a false negative | 5.20 | 5.18 | 5.80 | P = 0.97 | W = 0.05 |
| Interquartile range | 7.17 | 11.19 | 7.19 |
The 25 and 75 % percentiles are shown as is the Kruskal-Wallis test statistics and P value for the differences between each of the three conditions. All times are in seconds
Conversely, the dwell time statistics on all lesions using the same analysis were significant (W = 21.55, P = <0.0001).
Table 5 demonstrates a comparison of the dwell time on FNs between each of the three conditions. There was a significant difference in dwell time between the low prevalence expectation and the unknown prevalence expectation (U = 247.5, P = 0.008) with a longer dwell time in the unknown condition.
Table 5.
Mann-Whitney test comparisons of dwell time on lesions incorrectly ignored (FNs) between each condition
| Medians | P value | Mann-Whitney test statistics | |
|---|---|---|---|
| Low abnormality expectation (n = 35) vs unknown abnormality expectation (n = 24) | 0.22 vs 0.48 | P = 0.008 | U = 247.5 |
| Low abnormality expectation (n = 35) vs high abnormality expectation (n = 13) | 0.22 vs 0.30 | P = 0.57 | U = 202.5 |
| Unknown abnormality expectation (n = 24) vs high abnormality expectation (n = 13) | 0.48 vs 0.30 | P = 0.21 | U = 117.0 |
All times are in seconds. Number of FNs are in brackets
Table 6 demonstrates a comparison of the dwell time on FPs between each of the three conditions. There was a significant difference in dwell time between low expected prevalence condition and both the unknown (U = 364.5, P = 0.02) and high expected prevalence conditions (U = 364.5, P = 0.0002) with a significantly shorter dwell time in the low expectation condition.
Table 6.
Mann-Whitney test comparisons of dwell time on lesions correctly identified (TP)
| Medians | P value | Mann-Whitney test statistics | |
|---|---|---|---|
| Low abnormality expectation (n = 41) vs unknown abnormality expectation (n = 27) | 0.58 vs 1.77 | P = 0.02 | U = 364.5 |
| Low abnormality expectation (n = 41) vs high abnormality expectation (n = 38) | 0.58 vs 3.47 | P = 0.0002 | U = 397.0 |
| Unknown abnormality expectation (n = 27) vs high abnormality expectation (n = 38) | 1.77 vs 3.47 | P = 0.48 | U = 459.0 |
All times are in seconds. Number of TPs are in brackets
Table 7 shows the comparison between normal (n = 47) and abnormal images (n = 10). There was no significant difference in the time to first fixation, although there was a significant difference in search time demonstrated with a longer search time on the abnormal images.
Table 7.
Median search time and time to first fixation between normal (n = 47) and abnormal (n = 10) images
| Normal | Abnormal | P value | Mann-Whitney test statistics | |
|---|---|---|---|---|
| Time to first fixation | 4.09 | 4.06 | P = 0.91 | U = 65501 |
| Interquartile range | 1.41–9.92 | 1.64–9.55 | ||
| Search time | 21.50 | 32.06 | P = <0.0001 | U = 62093 |
| Interquartile range | 20.90 | 27.49 |
The Mann-Whitney test statistics and P value for the differences between abnormal and normal images are shown. All times are in seconds
Table 8 demonstrates the dwell time and the time to first fixation on the lesion subject to the decision made by the radiologist (i.e. whether the decision was a TP or FN) on abnormal images (n = 10). No significant difference was seen in the time to first fixation on areas of suspicion, but there was a significant difference in dwell times with a much shorter dwell time seen on FNs regardless of the a priori information.
Table 8.
The median dwell and the median time to first fixation between true positive (TP) decisions and false negative (FN) decisions
| TP (n = 106) | FN (n = 72) | P value | Mann-Whitney test statistics | |
|---|---|---|---|---|
| Dwell time all | 1.750 | 0.31 | P = <0.0001 | U = 1959 |
| Interquartile range | 0.41–4.51 | 0.10–0.77 | ||
| Dwell time low expected abnormality | 0.58 (n = 41) | 0.22 (n = 35) | P = 0.004 | U = 441 |
| Interquartile range | 2.10 | 0.54 | ||
| Dwell time unknown expected abnormality | 1.77 (n = 27) | 0.48 (n = 24) | P = 0.01 | U = 193.5 |
| Interquartile range | 4.68 | 1.29 | ||
| Dwell time high expected abnormality | 3.47 (n = 38) | 0.30 (n = 13) | P = 0.001 | U = 93 |
| Interquartile range | 4.42 | 0.87 | ||
| Time to first fixation | 3.84 | 5.37 | P = 0.46 | U = 3462 |
| 25–75 % percentile | 7.65 | 7.95 | ||
| Time to first fixation low expected abnormality | 3.31 | 4.94 | P = 0.22 | U = 675.5 |
| Interquartile range | 4.04 | 7.16 | ||
| Time to first fixation unknown expected abnormality | 5.85 | 5.18 | P = 0.47 | U = 209 |
| Interquartile range | 7.39 | 11.19 | ||
| Time to first fixation high expected abnormality | 3.90 | 6.01 | P = 0.45 | U = 206 |
| Interquartile range | 8.66 | 6.47 |
The 25 and 75 % percentiles are shown as are the Mann-Whitney test statistics and P value for the differences between each of the three conditions. All times are in seconds
Discussion
This experiment demonstrated that the most significant increases in dwell and search time occurred during the cancer (high prevalence expectation) readings.
Perceptions are greatly influenced by expectations [20]. Previous research has demonstrated how prior expectation might bias perceptual decisions [21, 22]. We investigated whether an expectation of abnormality affects visual search by altering clinical information when looking for pulmonary nodules in adult chest radiographs.
Eye position recordings can be used to study the features that attract a radiologist’s attention during the diagnostic process to investigate the perceptual and cognitive processes used during visual search [23–27]. Experiments have shown that radiologists examining radiographs have longer dwell times at locations where tumours are present, even when they fail to identify and report them [28]. Methods of analysing the search pathways in medical images have been produced by Nodine and Kundel and Nodine at al. [28, 29]. Nodine reported that the minimal dwell time for abnormality recognition of lung nodules is 900 ms and, with Kundel, provided a method to reduce the recognition failure of abnormal features [28]. Recognition errors are associated with gaze durations of less than 900 ms whilst decision errors are associated with durations greater than 900 ms [25]. This method identifies regions in the image that the eye has remained on for a certain amount of time. These regions are the areas that attention has been drawn to during search.
There was a significantly shorter total search time (W = 75.19, P = <0.0001) during low abnormality expectation suggesting that abnormality expectation does affect search (Table 3).
The authors speculate that during the low expectation condition, the radiologists assumed there would be a low probability of pathology and consequently did not spend as long scrutinising the image. This corresponds with previous research by Wolfe et al. (2013) that the prior probability of target presence affects search duration [30]. Interestingly, there was no significant difference in search times between the unknown and high abnormality expectations implying that absence of clinical information encourages the radiologist to be equally more cautious in his approach to diagnosis as when viewing images containing a possible malignancy. The increase in search time on images with a high abnormality expectation supports the findings of Reed et al. [31]. Time to first fixation on a TP or FP “lesion” was not significantly different. Search times between abnormal and normal images was also affected with a significantly shorter search time on normal images (U = 62,093, P = <0.0001). However, this may be related to the disparity in the small number of abnormal images (n = 10) compared to normal images (n = 47).
There was also a highly significant increase in the dwell time on TP lesions in the high expected abnormality (U = 397.0, P = <0.0002) and unknown expected abnormality conditions (U = 364.5, P = <0.02) when compared to the low expected abnormality condition (Table 6).
Comparison between the high expected abnormality and unknown expected abnormality conditions demonstrated no significant change. The shorter dwell time in the low abnormality condition draws a parallel with the overall shorter search time in the lower abnormality condition. Analysis of the dwell times on the region of interest for FN lesions was also undertaken (Table 5). There was a significant difference between both the low expected abnormality and unknown expected abnormality conditions (U = 247.5 P = 0.008) with a longer dwell time under the unknown condition when the radiologist misdiagnosed the image (FN). All other metrics were not significant although it was thought that the lack of FNs (n = 13) in the high expected prevalence condition may have contributed to this finding. An analysis on the dwell time of TPs vs FNs (Table 8) indicated a significantly longer dwell time on lesions with a true positive decision regardless of the clinical information provided.
This finding supports the findings of Manning et al. [24] and Nodine et al. [32]. It suggests that this is an indication of more extensive information processing. However, part of this increase in dwell may have been because the radiologists had to spend some time clicking on the image and designating a confidence level to their decision. The availability of a priori information had no significant effect on how quickly participants focused on an area of suspicion.
Limitations of the study include the small number of abnormal images and an assumption that each reader would formulate a low, unknown and high prevalence expectation appropriate to the information provided. Each radiologist was randomly assigned to one of the three workstations using each condition. The availability (or lack) of a workstation caused the unequal number of readers for each condition. There is also the question of whether radiologists expect a higher prevalence of abnormalities in laboratory experiments than that experienced in the clinical arena.
Conclusions
The presence or the complete lack of a priori information influenced the radiologists by increasing search times when they either expected to observe an abnormality or were unsure of what to expect. Consequently, such an influence should be taken into account when deciding how clinical information should be presented to the radiologist. Such information may direct search patterns to expected areas of pathology. If no pathology is seen, there might be an inclination for radiologists to terminate their search too early [33]. Alternatively, if an abnormality is present, further examination might not be undertaken leading to the “satisfaction of search” phenomenon [34]. Future studies should also include the best way of providing clinical information. Further research might be to evaluate radiologists’ search patterns in a clinical environment to observe whether eye fixations and search paths can be used to highlight and possibly eliminate misdiagnoses.
References
- 1.Shah PK, Austin JH, White CS, Patel P, Haramati LB, Pearson GDN, Shiau MC, Berkmen YM. Missed non–small cell lung cancer: radiographic findings of potentially resectable lesions evident only in retrospect. Radiology. 2003;226:235–241. doi: 10.1148/radiol.2261011924. [DOI] [PubMed] [Google Scholar]
- 2.Tuddenham WJ, Calvert WP. Visual search patterns in roentgen diagnosis. Radiology. 1961;76(2):255–6. doi: 10.1148/76.2.255. [DOI] [PubMed] [Google Scholar]
- 3.Carmody DP, Kundel HL, Toto LC. Comparison scans while reading chest images taught, but not practiced. Investig Radiol. 1984;19(5):462. doi: 10.1097/00004424-198409000-00023. [DOI] [PubMed] [Google Scholar]
- 4.Cruz MF, Edwards J, Dinh MM, Barnes EH. The effect of clinical history on accuracy of electrocardiograph interpretation among doctors working in emergency departments. Med J Aust. 2012;197(3):161–165. doi: 10.5694/mja11.11598. [DOI] [PubMed] [Google Scholar]
- 5.Doubilet P, Herman PG. Interpretation of radiographs: effect of clinical history. Am J Roentgenol. 1981;137:1055–1058. doi: 10.2214/ajr.137.5.1055. [DOI] [PubMed] [Google Scholar]
- 6.Song KS, Song HH, Park SH, et al. Impact of clinical history on film interpretation. Yonsei Med J. 1992;33:168–172. doi: 10.3349/ymj.1992.33.2.168. [DOI] [PubMed] [Google Scholar]
- 7.Houssami N, Irwig L, Simpson JM, McKessar M, Blome S, Noakes J. The influence of clinical information on the accuracy of diagnostic mammography. Breast Cancer Res Treat. 2004;85:223–228. doi: 10.1023/B:BREA.0000025416.66632.84. [DOI] [PubMed] [Google Scholar]
- 8.Berbaum KS, Franken EA, Jr Anderson KL, Dorfman DD, Erkonen WE, Farrar GP, Geraghty JJ, Gleason TJ, MacNaughton ME, Phillips ME, Renfrew DL, Walker CW, Whitten CG, Young DC. The influence of clinical history on visual search with single and multiple abnormalities. Investig Radiol. 1993;28:191–201. doi: 10.1097/00004424-199303000-00001. [DOI] [PubMed] [Google Scholar]
- 9.Swenson RG, Hessel SJ. The value of searching films without specific preconceptions. Investig Radiol. 1985;20(1):100–14. doi: 10.1097/00004424-198501000-00024. [DOI] [PubMed] [Google Scholar]
- 10.Summerfield C, Egner T. Expectation (and attention) in visual cognition. Trends Cogn Sci. 2009;13(9):403–9. doi: 10.1016/j.tics.2009.06.003. [DOI] [PubMed] [Google Scholar]
- 11.Balslev T, Jarodzka H, Holmqvist K, de Gravee W, Muijtjens AMM, et al. Visual expertise in paediatric neurology. Eur J Paediatr Neurol. 2012;16:161–6. doi: 10.1016/j.ejpn.2011.07.004. [DOI] [PubMed] [Google Scholar]
- 12.Mallett S, Phillips P, Fanshawe TR, Helbren E, Boone D, Gale A, et al. Tracking eye gaze during interpretation of endoluminal three-dimensional CT colonography: visual perception of experienced and inexperienced readers. Radiology. 2014;273(3):783–92. doi: 10.1148/radiol.14132896. [DOI] [PubMed] [Google Scholar]
- 13.Vaidyanathan P, Pelz J, Alm C, Pengcheng S, Haake A: Recurrence quantification analysis reveals eye-movement behavior differences between experts and novices. Proceedings of the Symposium on Eye Tracking Research and Applications. New York: ACM, 2014, pp 303–6
- 14.Swensson RG, Hessel SJ, Herman PG. Omissions in radiology: faulty search or stringent reporting criteria? Radiology. 1977;123(3):563–7. doi: 10.1148/123.3.563. [DOI] [PubMed] [Google Scholar]
- 15.Berbaum KS, Franken EA, Jr, Dorfman DD, Barloon TJ. Influence of clinical history upon detection of nodules and other lesions. Investig Radiol. 1988;23(1):48–55. doi: 10.1097/00004424-198801000-00011. [DOI] [PubMed] [Google Scholar]
- 16.Littlefair S, Mello-Thoms C, Reed W, Pietryzk M, Lewis S, McEntee M, Brennan P. Increasing abnormality expectation in thoracic radiology leads to overcall. Acad Radiol. 2016;23(3):284–9. doi: 10.1016/j.acra.2015.11.007. [DOI] [PubMed] [Google Scholar]
- 17.Shiraishi J, Katsuragawa S, Ikezoe J, Matsumoto T, Kobayashi T, Komatsu K, Matsui M, Fujita H, Kodera Y, Kunio D. Development of a digital image database for chest radiographs with and without a lung nodule: Receiver operating characteristic analysis of radiologists’ detection of pulmonary nodules. AJR. 2000;174(1):71–74. doi: 10.2214/ajr.174.1.1740071. [DOI] [PubMed] [Google Scholar]
- 18.Haygood TM, Ryan J, Brennan PC, Li S, Marom EM, McEntee MF, Itani M, Evanoff M, Chakraborty D. On the choice of acceptance radius in free-response observer performance studies. Br J Radiol. 2012;86(1021):42313554. doi: 10.1259/bjr/42313554. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 19.Samei E, Badano A, Chakraborty D, et al. Assessment of display performance for medical imaging systems. American Association of Physicists in Medicine Task Group 18. http://www.aapm.org/pubs/reports/OR_03.pdf. 2005
- 20.Seriès P, Seitz AR. Learning what to expect (in visual perception) Front Hum Neurosci. 2013;7:668. doi: 10.3389/fnhum.2013.00668. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 21.Rahnev D, Lau H, de Lange FP. Prior expectation modulates the interaction between sensory and prefrontal regions in the human brain. J Neurosci. 2011;31(29):10741–10748. doi: 10.1523/JNEUROSCI.1478-11.2011. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 22.Reed WM, Ryan JT, McEntee MF, Evanoff MG, Brennan PC. The effect of abnormality prevalence expectation on expert observer performance and visual search. Radiology. 2011;258(3):938–43. doi: 10.1148/radiol.10101090. [DOI] [PubMed] [Google Scholar]
- 23.Tourassi G, Voisin S, Paquit V, Krupinski E. Investigating the link between radiologists’ gaze, diagnostic decision, and image content. J Am Med Inform Assoc. 2013;20(6):1067–75. doi: 10.1136/amiajnl-2012-001503. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 24.Manning DJ, Ethell SC, Donovan T. Detection or decision errors? Missed lung cancer from the posteroanterior chest radiograph. Br J Radiol. 2004;77:231–5. doi: 10.1259/bjr/28883951. [DOI] [PubMed] [Google Scholar]
- 25.Krupinski EA. Visual scanning patterns of radiologists searching mammograms. Acad Radiol. 1996;3:137–44. doi: 10.1016/S1076-6332(05)80381-2. [DOI] [PubMed] [Google Scholar]
- 26.Hu CH, Kundel HL, Nodine CF, Krupinski EA, Toto LC. Searching for bone fractures: a comparison with pulmonary nodule search. Acad Radiol. 1994;1(1):25–32. doi: 10.1016/S1076-6332(05)80780-9. [DOI] [PubMed] [Google Scholar]
- 27.Mello-Thoms C, Nodine CF, Kundel HL: What attracts the eye to the location of missed and reported breast high expected abnormalities? In: Proceedings of the Eye Tracking Research and Applications (ETRA) Symposium 111–117, 2002
- 28.Nodine CF, Kundel HL: A Visual Dwell Algorithm Can Aid Search and Recognition of Missed Lung Nodules in Chest Radiographs. In: Brogan D Ed. Visual Search, Taylor and Francis, 1990, pp 399–406
- 29.Nodine CF, Kundel HL, Toto LC, Krupinski EA. Recording and analysing eye-position data using a microcomputer workstation. Behav Res Methods Instrum Comput. 1992;24(3):475–485. doi: 10.3758/BF03203584. [DOI] [Google Scholar]
- 30.Wolfe JM: When do I Quit? The Search Termination Problem in Visual Search. In: Dodd MD, Flowers JH Eds. The Influence of Attention, Learning, and Motivation on Visual Search, Nebraska Symposium on Motivation, 2013
- 31.Reed WM, Chow SL, Chew LE, Brennan PC. Can abnormality expectations drive radiologists’ behavior? Acad Radiol. 2014;21(4):450–6. doi: 10.1016/j.acra.2013.12.002. [DOI] [PubMed] [Google Scholar]
- 32.Nodine CF, Mello-Thoms C, Kundel HL, Weinstein SP. Time course of perception and decision making during mammographic interpretation. AJR Am J Roentgenol. 2002;179(4):917–23. doi: 10.2214/ajr.179.4.1790917. [DOI] [PubMed] [Google Scholar]
- 33.Wolfe JM. When do I quit? The search termination problem in visual search. Neb Symp Motiv. 2012;59:183–208. doi: 10.1007/978-1-4614-4794-8_8. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 34.Berbaum KS, Franken EA, Dorfman DD, Roolholamini SA, Kathol MH, Barloon TJ, Behlke FM, Sato Y, Lu CH, El-Khoury GY, Flickinger FW, Montgomery WJ. Satisfaction of search in diagnostic radiology. Investig Radiol. 1990;25(2):133–140. doi: 10.1097/00004424-199002000-00006. [DOI] [PubMed] [Google Scholar]