International Journal for Quality in Health Care 2012; Volume 24, Number 2: pp. 114– 120 Advance Access Publication: 24 January 2012 10.1093/intqhc/mzr088 Evaluation of a pilot surgical adverse event detection system for Italian hospitals CATERINA CAMINITI 1, FRANCESCA DIODATI 1, DONATELLA BACCHIERI 1, PAOLO CARBOGNANI 1, PAOLO DEL RIO 1, ELISA IEZZI 1, DANTE PALLI 1, ISABELLA RABOINI 1, ERICAVECCHIONE 1 AND LUCA CISBANI 2 Address reprint requests to: Caterina Caminiti, Research and Innovation Unit, Parma Teaching Hospital, Italy. Tel: þ39-0521-1702138; Fax: þ39-0521-1702277; E-mail: ccaminiti@ao.pr.it Accepted for publication 28 December 2011 Abstract Objective. To devise an adverse event (AE) detection system and assess its validity and utility. Design. Observational, retrospective study. Setting. Six public hospitals in Northern Italy including a Teaching Hospital. Participants. Eligible cases were all patients with at least one admission to a surgical ward, over a 3-month period. Interventions. Computerized screening of administrative data and review of flagged charts by an independent panel. Main Outcome Measures. Number of records needed to identify an AE using this detection system. Results. Out of the 3310 eligible cases, 436 (13%) were extracted by computerized screening. In addition, out of the 2874 unflagged cases, 77 randomly extracted records (3%) were added to the sample, to measure unidentified cases. Nursing staff judged 108 of 504 (21%) charts positive for one or more criteria; surgeons confirmed the occurrence of AEs in 80 of 108 (74%) of these. Compared with random chart review, the number of cases needed to detect an AE, with the computerized screening suggested by this study, was reduced by two-thirds, although sensitivity was low (41%). Conclusions. This approach has the potential to allow the timely identification of AEs, enabling to quickly devise interventions. This detection system could be of true benefit for hospitals that intend assessing their AEs. Keywords: adverse events, patient safety, quality improvement, quality management, safety indicators, surgery, teamwork, audit, external quality assessment, medical errors Introduction Adverse events (AEs) represent a serious problem; they interest nearly 1 out of 10 hospitalized patients and a substantial part of these is preventable [1]. It is increasingly accepted that most medical errors are not due to ignorance or negligence of a single individual, the so-called ‘person approach’, but that the causes of errors must be searched for within the system, towards which improvement interventions should be directed (the ‘systems approach’) [2]. This latter approach should be systematic, and should include the analysis of healthcare problems, selection of evidence-based interventions involving all stakeholders, context analysis for the identification of barriers and facilitators for change and program testing and performance measuring [3, 4]. Different detection strategies have been reported in the literature, aimed at identifying AEs in hospitals. Thomas and Petersen [5] examined as many as eight AE measurement methods, specifying that these were only the most commonly used out of a wide range of strategies applied in different settings. Each method had advantages and limits, suggesting that the choice of which method to use should be determined by various factors, such as aims, context and resource availability. Clinical documentation review seems to be the most widely used method for AE identification in hospitals. Despite its known limitations, chart review is very easy to implement, it allows planning of data collection and, thus, is generally favored by staff [6]. In most experiences, clinical records to be reviewed are randomly selected [7 – 12], or all records from a given period International Journal for Quality in Health Care vol. 24 no. 2 # The Author 2012. Published by Oxford University Press in association with the International Society for Quality in Health Care; all rights reserved 114 Downloaded from http://intqhc.oxfordjournals.org/ at Biblioteca della Facolta' di economia, Universita' di Pavia on February 28, 2014 1 Research and Innovation Unit, Parma Teaching Hospital, Parma, Italy, and 2Emilia Romagna Regional Health Trust, Regional Health Care and Social Agency, Clinical Governance, Bologna, Italy Detecting adverse events † are considered [13, 14]. In both strategies, without an initial screening of cases with potential AEs, the number of records to be examined in order to detect AEs must be very high, making investigations extremely time consuming and cumbersome. More recently, the use of indicators applied to administrative data, to screen cases with potential AEs for chart review, has been proposed [15]. This study intended to take advantage of both computerized screening methods and medical record review, with three objectives: The findings will determine whether this system could be introduced into routine practice for timely detection of AEs, allowing for the identification of corrective measures. Methods Definitions The following definitions are provided in order to enable a comparison of the results with other international research data. In this study, positively screened cases, or flagged cases, are cases selected by computerized screening because they exhibited one or more indicators. Unflagged cases are cases that were not selected by computerized screening. An AE was defined as an injury that was caused by medical management (rather than the underlying disease) and that prolonged hospitalization, produced a disability at the time of discharge, or both. Causation is the degree to which the reviewer is confident that the event was caused by healthcare management, i.e. that an AE had occurred. Preventable AEs were those that would not have occurred if the patient had received ordinary standards of care, appropriate for the time of the study [6, 9]. For each AE found in this study, the degree of confidence in preventability was assigned on a six-point scale, and events were judged preventable if they received a confidence score of 4 or higher, as indicated by the Harvard Study [9]. The reference standard indicates the clinical chart review method used to confirm the occurrence of an AE. Findings on the comparison between computerized screening and the reference standard are described in terms of: sensitivity, the ability to flag correctly the cases with an AE; specificity, the ability to not flag correctly the cases without an AE; positive predictive value (PPV), the proportion of flagged cases that were also confirmed as having an AE; negative predictive value (NPV), the proportion of unflagged cases that were also confirmed as not having an AE. System development The study was conducted in six public hospitals within the Northern Italian Provinces of Parma and Piacenza, which volunteered to participate: the Parma Teaching Hospital (i.e. a hospital with full-time core residency training programs in medicine and surgery, 1334 beds), two Community Hospitals in the Parma Province (293 and 118 beds) and three Community Hospitals in the Piacenza Province (130, 178 and 524 beds). The geographical proximity of the two Provinces facilitated the conduction of the study. Data collection took place at the coordinating center, at the Parma Teaching Hospital. Since a large proportion of AEs occurs in surgical wards—up to 60% according to different studies [1] the investigation was limited to general surgical units. The first phase of the study concerned the development of a computerized method for the screening of discharged summaries containing possible AEs. For this purpose, the working group, made up of three surgeons, four nurses and one epidemiologist, analyzed the literature to identify indicators which could be used to detect AEs in the hospital. Three sets of indicators were found: those used in the Harvard Study [9], the limited adverse occurrence screening by Wolff [13] and the Patient Safety Indicators (PSIs) developed by AHRQ [16, 17]. Although the first two sets of indicators were designed to be applied to clinical records, the latter is a screening tool for administrative data, the content of the three sets often overlapped. After eliminating redundancies, a list of 29 possible indicators was completed, and each item was reviewed and discussed by the working group, resulting in the selection of seven indicators relevant to surgical patients and judged most suitable according to the discharge summary coding rules used at the participating hospitals. The selected indicators are the following [13, 16]: (i) Death—in-hospital death of surgical patients aged ,75 years. (ii) Transfer to intensive care—surgical cases transferred from a general ward to the intensive care unit during the same hospitalization. (iii) Return to operating theater—surgical cases returning to the operating room within 7 days during the same hospitalization. (iv) Unplanned readmission—unplanned readmissions for surgical cases within 28 days of discharge, also to a different hospital. (v) Length of stay (LOS)—surgical cases with LOS . 21 days. (vi) Postoperative PE or DVT—cases of deep vein thrombosis or pulmonary embolism in surgical discharges. (vii) Postoperative respiratory failure—cases of acute respiratory failure in elective surgical discharges. The chosen indicators were used for the development of a computer program based on SAS software, by the Agenzia Sanitaria e Sociale Regionale of the Emilia Romagna Region, designed to screen discharge summaries to identify possible AEs using ICD-9-CM diagnosis codes. A hospital discharge case was flagged if at least one of the indicators scored positive based on the algorithms defined by their respective authors [13, 16]. A file containing all flagged cases, and a 115 Downloaded from http://intqhc.oxfordjournals.org/ at Biblioteca della Facolta' di economia, Universita' di Pavia on February 28, 2014 (1) devise an AE detection system; (2) assess its validity; (3) assess its utility (to what extent computerized screening improves efficiency). Patient Safety, Quality Measurement Caminiti et al. Table 1 k coefficient values for the measurement of agreement between reviewers K Classification 95% CI .................................................................................... Nursing interrater reliability Nursing intrarater reliability Medical interrater reliability Medical intrarater reliability 0.59 0.75 0.50 0.86 Moderate High Moderate Almost perfect 0.53 – 0.65 0.69 – 0.81 0.41 – 0.59 0.77 – 0.95 adopted in many subsequent studies of this kind, is appealing as the selection by qualified nurses allows to save more costly physician time, still enabling surgeons to look into the clinical details for reasons and prevention strategies. To avoid corporate bias, reviewers did not examine charts of patients hospitalized at the institution where they operated, and the names of the surgeons were deleted from the chart photocopies used during the review. Discharge summaries of all patients with at least one admission in a surgical ward of the participating institutions, during the index period (3 months), were included in the investigation (eligible cases); no exclusion criteria were applied, to ensure sample representativeness. The review of flagged charts required a total of 215 h/ nurse and 96 h/physician; during each meeting, abstracted records relative to patients discharged in the previous 2 weeks were reviewed. To test reliability, calculated for the presence of AEs, 10% of records was randomly selected and resubmitted 3 months after the first review, both to the same reviewer (intrarater reliability) and also to a different reviewer (interrater reliability). Reviewers were blinded to the outcome of the first assessment. Overall, k coefficient values for the measurement of agreement between reviewers was good (Tables 1), better than those recorded in studies using a similar methodology [10]. Statistical analysis System evaluation Findings of screening were validated using clinical records as the reference standard. Charts abstracted by the centralized software were reviewed in a two-stage process similar to the one used by the Harvard Study: charts relative to flagged cases were first reviewed by nurses of the Research Team, who identified those containing potential safety-related events. The selected records were then submitted to the physician reviewers of the Research Team, who were asked to confirm the presence of AEs, as well as to rate the degree of causation and preventability of each event. To prevent diagnostic review bias, nursing and medical reviewers were blinded to the output of computerized screening, i.e. they did not know whether the charts they had reviewed had been flagged as containing potential AEs or not. The two-stage process, initially used in the Harvard Study [9] and then 116 Sample size Because the paper is descriptive in nature, the sample size was calculated based on practical considerations, since the literature provides no indications on the efficiency of a system of this kind. The sample size was estimated considering an expected accuracy of 0.75 and a minimal acceptable lower confidence limit of 0.65 [18]; the necessary number of cases (with confirmed AE) was 262. Assuming a prevalence value for the study population between 5 and 10%, a total sample of ≏4000 records, corresponding to 3 months of collection, were necessary. In addition to flagged records, to measure the proportion of false negatives and estimate the total prevalence of AEs ( pretest probability), 3% of unflagged cases was abstracted at the same time, stratifying by hospital. Downloaded from http://intqhc.oxfordjournals.org/ at Biblioteca della Facolta' di economia, Universita' di Pavia on February 28, 2014 random sample of cases not identified by computerized screening (see the ‘Sample size’ section) was sent to the coordinating center for chart retrieval and review. The second phase of the study was the identification of a tool to be used for the two-stage (nursing and medical) manual chart review for the confirmation of AEs, following the methodology outlined by the Harvard Study [9]. The working group decided to employ the instruments developed by Michel et al. [6], which were kindly sent to us by the author, translated into Italian and adapted to the Italian healthcare system. The tool consisted in two questionnaires: the detection sheet, to be used by nurses to judge the presence of possible AEs, consisting in 17 criteria, and the confirmation sheet, used by clinicians to confirm AEs and rate the degree of causation and preventability of each event. After defining the instruments for the study, the project’s Research Team was created, consisting in the nurses and clinicians who would be responsible for the manual chart review. The Research Team was made up of 10 nurses and 7 surgeons, including the healthcare professionals of the working group, representative of all participating institutions. The number of professionals was determined based on an estimate of the number of charts to be reviewed, and by the estimate of the time required for the chart review. The nurses were employees of their institution’s administration offices, and had experience of clinical record coding. The physicians were highly motivated surgeons mainly operating in clinical practice. A 3-day training program was provided to all members of the Research Team, between December 2007 and January 2008, with the aim of ensuring uniformity in judgment and minimize interpretation biases. During the course, reviewers were introduced to the project and to the tools adapted from Michel et al. [6]. Training included simulations of the chart review process, both on an individual basis, with each person reviewing the same chart before and after receiving instructions, and in groups consisting in one nurse and one clinician, where each group examined the same 10 charts followed by discussion of results. For training, charts were selected relative to events difficult to evaluate. Detecting adverse events † Patient Safety, Quality Measurement Table 2 Characteristics of eligible cases compared with flagged, unflagged and confirmed cases Total of Flagged eligible cases cases 3310 100% 436 13% Unflagged cases 2874 87% Confirmed by nurses 108 3% Confirmed by physicians 80 2% ............................................................................................................................................................................. Local health area Parma Teaching Hospital Piacenza Community Hospitals Parma Community Hospitals Age P25 P50 (median) P75 LOS P25 P50 (median) P75 78% 271 62% 2299 80% 16 9% 65 15% 217 8% 5 14% 100 23% 358 12% 87 100% 100% 100% 1509 46% 250 57% 1584 55% 80 17% 6% 77% 100% 71% 27 7 14 16 30 7 2 5 25% 19 6% 5 12% 6 14% 9 28% 26 6% 3 2% 2 4% 10 100% 24% 6% 8% 12% 32% 4% 2% 12% 100% 703 449 413 380 356 179 165 665 21% 143 33% 14% 48 11% 12% 37 8% 11% 38 9% 11% 88 20% 5% 16 4% 5% 29 7% 20% 37 8% 100% 100% 578 397 370 343 277 162 137 610 57 71 81 56 70 78 57 72 82 61 71 78 57 70 78 2 5 11 4 9 18 2 5 10 8 13 24 8 15 23 Data analysis Continuous data were reported as median and 25 – 75th IQ range values. Dichotomous variables were compared by two-sided x 2 test, for the evaluation of patient characteristics. All performance parameters were determined inferentially, deriving from the unflagged sample (3%) the information relative to the total of unflagged cases. Point estimates and 95% confidence intervals for the measures of diagnostic performance were determined. SAS software release 8.2 was used for the statistical analysis. Results Figure 1 depicts the study’s flow diagram. Discharge summaries of all patients hospitalized in surgical wards between February and April 2008 were screened by the computer program every 2 weeks. Out of the 3310 eligible cases, 436 (13%) were abstracted. Out of 2874 unflagged cases, 77 randomly abstracted records (3%) were added to the sample to measure diagnostic performance. Missing data concerned 8 of 436 (2%) flagged charts and 1 of 77 (1%) unflagged charts; they could not be retrieved either because patients had been readmitted 20% 14% 13% 12% 10% 6% 5% 21% 100% 15% 14 4% 5 81% 61 100% 74% 57 and were currently hospitalized or because they still had not been signed and archived. Nursing staff identified 108 of 504 charts (21%) as positive for one or more criteria and submitted these to medical review. Surgeons confirmed the occurrence of AEs in 80 charts, corresponding to 74% (80 of 108) of nurse-flagged cases and 16% (80 of 504) of all reviewed records. Demographic and clinical features of all eligible cases, of flagged and unflagged cases, as well as of cases confirmed by nurse and physician review are reported in Table 2. Seventy-eight percent of patients under study were cases from the Parma Teaching Hospital; 46% were male, with a median age of 71 years, median LOS of 5 days. The most common major diagnostic category (MDC) was digestive system (MDC 6). Distribution of flagged cases suggested a higher prevalence in the community hospitals, with respect to the total number of eligible subjects in each institution. Notably, 80% of cases confirmed by nurse review and 77% of cases confirmed by physician review were concentrated in the community hospitals. The frequency of flagged cases for each individual indicator, the corresponding percentages of confirmed and preventable AEs after medical review, is given in Table 3. The screening indicator ‘Return to operating theater’ was the most frequent, but also the less predictive indicator (only 16% of flagged cases turned out to be confirmed AEs). 117 Downloaded from http://intqhc.oxfordjournals.org/ at Biblioteca della Facolta' di economia, Universita' di Pavia on February 28, 2014 Male sex Most frequent major diagnostic categories Digestive system (MDC 6) Kidney and urinary tract (MDC 11) Circulatory system (MDC 5) Respiratory system (MDC 4) Hepatobiliary system and pancreas (MDC 7) Male reproductive system (MDC 12) Skin, subcutaneous tissue and breast (MDC 9) OTHER 2570 282 458 Caminiti et al. Table 3 Distribution of flagged cases, confirmed and preventable AEs for each indicator Flagged cases Confirmed AEs Preventable AEs ............................................................................................................................................................................. Death Postoperative respiratory failure Transfer to intensive care LOS . 21 days Unplanned readmission Return to operating theater Postoperative PE or DVT 12 9 56 131 11 264 5 2% 2% 11% 27% 2% 54% 1% 7 6 28 39 2 42 3 58% 67% 50% 30% 18% 16% 60% 2 3 3 6 0 5 1 29% 50% 11% 15% 0% 12% 33% The same case may be reported more than once being eligible for more than one indicator; similarly, the same AE may be linked to more than one indicator. The data of chart review on AE frequency, preventability and outcome, relative to flagged and unflagged cases are shown in Table 4. The frequency of AEs was 18% (77 of 428) for flagged cases and 4% (3 of 76) for unflagged cases. Out of 88 AEs confirmed by surgeons in the flagged case-mix, 17% was judged preventable (15 of 88) vs. one-third in the unflagged case-mix (33%). Concerning the outcome in flagged cases, 22% of AEs caused disability at discharge, 49% prolonged LOS, 14% influenced survival, and 8% caused death. AEs found in unflagged cases consisted in hematoma, bleeding and intense pain. 118 Validity The diagnostic performance of computerized screening for the detection of AEs is detailed in Table 5. Sensitivity was 41%, specificity was 89%, PPV was 18% and NPV was 96%. Overall, the screening accuracy (true positives þ true negatives over the total of cases) was 86%. Efficiency Computerized screening reduced by two-thirds the number of charts to be reviewed to detect an AE compared with Downloaded from http://intqhc.oxfordjournals.org/ at Biblioteca della Facolta' di economia, Universita' di Pavia on February 28, 2014 Figure 1 Flow diagram. Numbers of medical records are shown. Detecting adverse events † Table 4 AEs, preventability and outcome Reviewed charts Flagged Unflagged random samplea .................................................................................... a 3% of unflagged cases was randomly abstracted. Table 5 Diagnostic performance—for unflagged cases inferential data were used estimates of Cases with AEs Cases without AEs .................................................................................... Flagged 77 Unflagged 112 (3) 189 (80) Sensitivity Specificity PPV NPV 41% 89% 18% 96% 351 2725 (73) 3076 (424) 95% CI 34% 87% 14% 95% 428 2837 (76) 3265 (504) 48% 90% 22% 97% The sample actually analyzed, equal to 3% of unflagged cases, is given in parentheses. random sampling. This value was derived comparing AE prevalence in the total of eligible cases (6%) and the frequency of AEs in flagged cases (18%). Specifically, without computerized screening, physicians would have had to review 17 medical records to detect one AE, while with the screening, physicians needed to review only 6 medical records to find an AE. Discussion The detection system used in this study improved efficiency in locating potential AE cases, enabling physicians to quickly converge on the true AE cases to find causes and potential strategies of correction. In fact, to detect an AE, the number of charts to be reviewed was reduced by two-thirds. Unfortunately, only 41% of AEs was captured ( probably the most serious ones) implying the risk of underestimating problems and the need to adopt interventions. It should be emphasized, however, that work aimed at the refinement of surgical AE screening indicators is ongoing [19], and the computerized method will be gradually updated, thus improving its performance. Although this study was not designed to measure validity and utility of each single indicator, our data suggested that the indicator ‘LOS . 21 days’ worked best, as it captured the highest number of confirmed and preventable AEs, requiring the lowest number of charts. Moreover, this indicator exhibited a highly significant association with the other indicators, i.e. frequently, cases flagged for this indicator were also flagged for other indicators. A curious and unexpected finding was the difference in the rate of confirmed cases among the hospitals; specifically, the highest concentration of AEs was found in one community hospital. This finding suggests a potential use of the system as a tool for the prioritization of corrective interventions when applied to multiple hospitals, for example, in a given geographical area. To our knowledge, this is the first research investigating the importance of AEs in an Italian setting. With respect to other international research published in this field, our study is mainly characterized by two aspects. It is the product of the cooperation between representatives from the management, medical and nursing staff, who worked together from the initial stages of the protocol development all through its conduction. This is, we believe, a very important fact which should favor the systematic use of this system in routine practice. Also, unlike most studies on AE detection using administrative data, which only refer to PSIs and generally measure the validity of individual indicators, our research tested the advantages of both computerized screening and manual review, using a set of indicators derived from different experiences. However, our research reflects the findings obtained by other PSI validation studies, which suggest a generally high specificity and modest sensitivity. In particular, two studies similar to ours have been published, by Romano et al. [19], and by Kaafarani and Rosen [20], who have examined the use of these sorts of screening approaches using surgical AHRQ PSIs. Unlike our research, however, both papers consider selected PSIs individually, and do not combine individual values. Sensitivities found in these studies range from 19 to 63%, according to the PSI, against our overall value of 41%. Specificities reported by Romano et al. [19] range from 99.1 to 99.9%, higher than the overall value obtained in our study. This work has a few limitations, as evidenced in other similar studies. First, its findings cannot be totally reproduced, as chart review implies some degree of subjective judgment. Moreover, the sample is relatively small, and limited to a restricted geographical area, therefore, direct inferences cannot be made for other settings. Finally, the use of administrative data for screening may lead to underestimation of a problem, because the ascertainment of AEs depends on the quality and completeness of coding, and only events for which there are corresponding ICD-9-CM codes can be found [17]. Despite these limitations, this study suggests that this detection system offers a true benefit for hospitals which intend to assess their AEs. 119 Downloaded from http://intqhc.oxfordjournals.org/ at Biblioteca della Facolta' di economia, Universita' di Pavia on February 28, 2014 Fq (% no. of records) No. of records 428 (100) 76 (100) No. of patients with at least one AE 77 (18) 3 (4) No. of AEs 88 (21) 3 (4) No. of preventable AEs 15 (4) 1 (1) Outcome Fq (% no. of AEs) The clinical event caused disability 19 (22) 0 (0) The clinical event prolonged LOS 43 (49) 1 (33) The clinical event influenced survival 12 (14) 0 (0) The clinical event caused death 7 (8) 0 (0) Patient Safety, Quality Measurement Caminiti et al. Acknowledgements Funding This work was supported by the Emilia-Romagna Regional Health Trust. References 1. de Vries EN, Ramrattan MA, Smorenburg SM et al. The incidence and nature of in-hospital adverse events: a systematic review. Qual Saf Health Care 2008;17:216–23. 2. Dankelman J, Grimbergen CA. Systems approach to reduce errors in surgery. Surg Endosc 2005;19:1017 –21. 3. van Bokhoven MA, Kok G, van der Weijden T. Designing a quality improvement intervention: a systematic approach. Qual Saf Health Care 2003;12:215–20. 4. Pronovost PJ, Berenholtz SM, Needham DM. Translating evidence into practice: a model for large scale knowledge translation. Br Med J 2008;337:a1714. 5. Thomas EJ, Petersen LA. Measuring errors and adverse events in health care. J Gen Intern Med 2003;18:61–7. 6. Michel P, Quenon JL, de Sarasqueta AM et al. Comparison of three methods for estimating rates of adverse events and rates of preventable adverse events in acute care hospitals. Br Med J 2004;328:199 –202. 7. Thomas EJ, Studdert DM, Burstin HR et al. Incidence and types of adverse events and negligent care in Utah and Colorado. Med Care 2000;38:261– 71. 8. Vincent C, Neale G, Woloshynowych M. Adverse events in British hospitals: preliminary retrospective record review. Br Med J 2001;322:517 –9. 120 10. Baker GR, Norton PG, Flintoft V et al. The Canadian Adverse Events Study: the incidence of adverse events among hospital patients in Canada. CMAJ 2004;170:1678 –86. 11. Zegers M, de Bruijne MC, Wagner C et al. Design of a retrospective patient record study on the occurrence of adverse events among patients in Dutch hospitals. BMC Health Serv Res 2007;7:27. 12. Soop M, Fryksmark U, Köster M et al. The incidence of adverse events in Swedish hospitals: a retrospective medical record review study. Int J Qual Health Care 2009;21:285 –91. 13. Wolff AM. Limited adverse occurrence screening: using medical record review to reduce hospital adverse patient events. Med J Aust 1996;164:458 –61. 14. Wolff AM, Bourke J, Campbell IA et al. Detecting and reducing hospital adverse events: outcomes of the Wimmera clinical risk management program. Med J Aust 2001;174:621 –5. 15. Zhan C, Miller MR. Administrative data based patient safety research: a critical review. Qual Saf Health Care 2003;12(Suppl 2):ii58 –63. 16. Agency for Healthcare Research and Quality. AHRQ Quality Indicators—Patient Safety Indicators: Software Documentation. Rockville, MD: Agency for Healthcare Research and Quality, 2002. http://www.qualityindicators.ahrq.gov/psi_overview.htm (August 2010, date last accessed). 17. Romano PS, Geppert JJ, Davies S et al. A national profile of patient safety in U.S. hospitals. Health Aff (Millwood) 2003;22:154 –66. 18. Flahault A, Cadilhac M, Thomas G. Sample size calculation should be performed for design accuracy in diagnostic test studies. J Clin Epidemiol 2005;58:859–62. 19. Romano PS, Mull HJ, Rivard PE et al. Validity of selected AHRQ patient safety indicators based on VA National Surgical Quality Improvement Program data. Health Serv Res 2009;44:182 –204. 20. Kaafarani HMA, Rosen AK. Using administrative data to identify surgical adverse events: an introduction to the Patient Safety Indicators. Am J Surg 2009;198(Suppl 5):S63 –8. Downloaded from http://intqhc.oxfordjournals.org/ at Biblioteca della Facolta' di economia, Universita' di Pavia on February 28, 2014 We thank the medical and nursing reviewers, Alessandra Bardiani, Barbara Benoldi, Stefania Bertocchi, Maria Cristina Cornelli, Guglielmo Delfanti, Paolo De Tullio, Carlo Maggi, Emilio Marchionni, Libera Notarangelo, Chiara Rocchetta, Michela Squeri, Angela Villa, Pietro Vitolo and Mario Zecchinato, for their invaluable contribution to the study. We are grateful to the medical managements of the participating institutions for their precious support. Special thanks go to Dr Maria Elisabeth Street, for her linguistic revision of the paper. 9. Brennan TA, Leape LL, Laird NM et al. Harvard Medical Practice Study I. Incidence of adverse events and negligence in hospitalized patients: results of the Harvard Medical Practice Study I. 1991. Qual Saf Health Care 2004;13:145–51.