Safety Science 50 (2012) 181189

Contents lists available at SciVerse ScienceDirect

Safety Science
journal homepage: www.elsevier.com/locate/ssci

A descriptive study of the OHS management auditing methods used by public sector organizations conducting audits of workplaces: Implications for audit reliability and validity
Lynda S. Robson , Sara Macdonald, Garry C. Gray 1, Dwayne L. Van Eerd 2, Philip L. Bigelow 2,3
Institute for Work & Health, 481 University Ave., Suite 800, Toronto, ON, Canada M5G 2E9

a r t i c l e

i n f o

a b s t r a c t
Past research has identied a research gap regarding studies of the reliability and validity of OHS management audit methods. This study describes 17 audit methods used by OHS auditing organizations in the broader public sector in the province of Ontario, emphasizing aspects relevant to audit reliability and validity. Wide variation was found in pertinent characteristics of the audit methods and in their corresponding programs. In addition, some discrepancies were found between actual auditing practices and international standards on management system auditing. Further research is needed to precisely determine the impact of these variations and discrepancies on the measurement properties of audit data and on the performance of audit programs. This study determined that such research is feasible with some, but not all, auditing programs. 2011 Elsevier Ltd. All rights reserved.

Article history: Received 29 October 2010 Received in revised form 17 June 2011 Accepted 8 August 2011 Available online 6 September 2011 Keywords: Audit Reliability Validity Qualitative study Quality Measurement

1. Introduction Occupational health and safety (OHS) management audits are used to evaluate workplaces OHS management structures and processes. The audits typically determine whether the organization is compliant with one or more standards; such as its own policies and procedures, applicable legislation and regulations, or another standard external to the organization (e.g. OHSAS 18001 (OHSAS Project Group, 2007)). OHS management audits may also examine OHS management effectiveness. Auditing is considered to be an important component of OHS management systems (ANSI/AIHA, 2005; OHSAS Project Group, 2007; ILO, 2001) and up to 95% of Fortune 2000 companies perform them (Nash, 2005). Auditing is less prevalent in smaller rms, but is relevant to them too (Grant and Brown, 2005). Typically, the processes carried out by auditors
Corresponding author. Tel.: +1 416 927 2027x2164; fax: +1 416 927 4167.
E-mail addresses: lrobson@iwh.on.ca (L.S. Robson), smacdonald@iwh.on.ca (S. Macdonald), ggray@ethics.harvard.edu (G.C. Gray), dvaneerd@iwh.on.ca (D.L. Van Eerd), pbigelow@iwh.on.ca (P.L. Bigelow). 1 Present addresses: Edmond J. Safra Center for Ethics, Harvard Law School, Harvard University, 124 Mount Auburn Street, Suite 520N, Cambridge, MA, USA, 02138. 2 Present address: Health Studies and Gerontology, University of Waterloo, 200 University Avenue West, Waterloo, ON, Canada N2L 3G1. 3 Present address: Dalla Lana School of Public Health, University of Toronto, 155 College Street, Toronto, ON, Canada M5T 3M7. 0925-7535/$ - see front matter 2011 Elsevier Ltd. All rights reserved. doi:10.1016/j.ssci.2011.08.006

include the following steps: (1) gathering evidence through systematic data collection, usually by reviewing documentation, conducting interviews and observing worksites; (2) evaluating the evidence against audit criteria; and (3) summarizing and reporting the results. The quantitative results of audits are often used by organizations as performance measures. For example, they are used to monitor organizational improvement in safety management (Bunn et al., 2001; LaMontagne et al., 2004; Nielsen et al., 2008; Pearse, 2002), compare organizational units, and determine whether a certain standard has been met (Eisner and Leger, 1988). Further, audit results can have consequences to organizations in terms of rewards or penalties (ACC, 2008; Government of Alberta, 2009; OSHA, 2010; WSIB, 2011a,b). Since audit data are used as performance measures, their reliability and validity are clearly important for sound organizational decision-making. However, a systematic search of the literature found surprisingly little research in this area (Robson and Bigelow, 2010). Furthermore, in the few cases where inter-auditor reliability has been formally investigated (Dyjack et al., 2003; Kuusisto, 2000; Matheson et al., 2008), it was poorer than expected. Our project addresses this research gap by studying a population of audit methods, using both qualitative and quantitative research methods. The aim was to lay the groundwork for future measurement studies that involve analyses of audit results. The specic objectives of this project were the following:

182

L.S. Robson et al. / Safety Science 50 (2012) 181189

Audit method
e.g. - content - sector specificity - clarity to user - data sources - sampling - decision support - scoring methods

Audit program
e.g. - resources - auditor recruitment -auditor training - quality control

Audit results
e.g. - reliability - validity - responsiveness

Auditor
e.g. - knowledge - skill - independence

External environment
e.g. - legislation & regulations - market competition

Workplaces
e.g. -need for audit - need for good audit results

Fig. 1. Conceptual model of the factors affecting the reliability and validity of OHS management system audits.

 describe and compare the OHS management system audit methods used by organizations in the Ontario Prevention System (a broader public sector entity, explained in Section 2.1),  investigate the content validity of the audit instruments,  assess the feasibility of further study of the measurement properties of the audit methods, This article focuses on the ndings related to the rst and third objectives. Findings related to the second objective have been reported elsewhere (Robson et al., 2010). In the preparatory phase of the project we reviewed the extensive prescriptive and the less extensive empirical literatures regarding the measurement properties and quality of OHS management audits (Beckmerhagen et al., 2003; Blackmore and Shannon, 1996; Cooper, 1998; Dyjack and Levine, 1996; Dyjack et al., 2003; Gay and New, 1999; Gillette et al., 2004; Glendon, 1995; Glendon and McKenna, 1995; Grant et al., 1996; HSE, 1997; Karapetrovic and Willborn, 2000a,b; Kennedy and Kirwan, 1998; Kuusisto, 2000; Matheson et al., 2008). We also reviewed some of the empirical literature on nancial audits that has relevance to OHS audits (Mayne, 2005; Messier et al., 2001; Meyer et al., 2007; Mohd-Sanusi and Mohd-Iskandar, 2007; Nelson and Tan, 2005; OLeary, 2004; Pugrath et al., 2007; Preuss, 1998; Richard, 2006; Sweeney and Pierce, 2004; Umar and Anandarajan, 2004). From these sources, we developed a conceptual model of the factors affecting OHS management audit reliability and validity; it guided our analysis and was rened iteratively during the course of the study. The current model shows how audit results are affected by the characteristics of the audit method, the auditor, workplaces, the auditing program and the external environment (Fig. 1). In this explorative study, we were not able to directly observe a set of audit results, but we were able to observe the other domains of the conceptual model. The ndings give a rich description of a dened population of auditing methods and their corresponding programs, with regards to characteristics relevant to audit reliability and validity. The results are relevant to other OHS management audits, as well as to audits of environmental management or quality management. 2. Material and methods The study protocol was approved by the Ofce of Research Ethics at the University of Toronto, Toronto, Ontario, Canada.

2.1. Study population The study population is comprised of the OHS management audit methods used by the organizations of the Ontario Prevention System (Robson et al., 2007). This is an aligned group of public and not-for-prot sector organizations with a mandate in primary prevention in occupational health and safety in Ontario. Together the organizations oversee to some extent approximately 93% of Ontario workers; the basis for the oversight is through provincial legislation (Ontario Health and Safety Act, Workplace Safety Insurance Act). At the time of the study, 20072008, the following 17 organizations were included in the Prevention System:  the Ministry of Labour (a department of the provincial government of Ontario),  the Prevention Division of the Workplace Safety & Insurance Board (WSIB; a provincial, statutory corporation that provides no-fault insurance to workplaces for the costs associated with work-related injuries and illnesses; and administers some prevention incentive programs to workplaces),  twelve sector-based Health and Safety Associations (non-prot organizations that provide consulting and training services to workplaces in their respective sectors, with costs offset by core funding from WSIB; the 12 HSAs have since amalgamated to four),  two worker-focused Health and Safety Associations (non-prot organizations that provide training and occupational health services to workers, with costs offset by core funding from WSIB), and  the Institute for Work & Health, a research organization receiving core funding from WSIB.4 A senior decision-maker in each organization was contacted by the lead author to determine whether their organization applied an audit method meeting the study criteria. If such a method was identied, a referral to a key informant within that organization was obtained. All organizations with an eligible audit method agreed to participate.

4 It should be noted that authors were all afliated with the Institute for Work & Health (IWH) at the time of the study. The IWH conducts and reports its research independently of the WSIB, though it receives core funding from it.

L.S. Robson et al. / Safety Science 50 (2012) 181189

183

The eligibility of an audit method was determined by the information provided by the senior decision-maker and key informant. Criteria for inclusion were as follows:  Assesses the management of OHS in a workplace.  Has a substantial focus on the management of OHS in the workplace. The following were the reasons for excluding a method from study:  Method assesses only selected aspects of OHS management (e.g. training only).  Method is used only by clients for self-assessment.  Method uses perception surveys as the primary source of data.  Method is developed by a 3rd party and used only on an ad hoc basis in response to a client request. 2.2. Data collection Semi-structured interviews were conducted in person with key informants and lasted approximately one-and-a-half hours. The content of the interview was based on the studys research objectives, the conceptual framework (Fig. 1), and an international standard for management system auditing (International Organization for Standardization (ISO), 2002). The key informant for a particular audit method was usually the manager of the program that provided auditing services to client workplaces. Typically, there was one interviewee reporting on one or two audit instruments, but in a few cases there were two to four interviewees in a group interview about one instrument; for two instruments, there were two interviews conducted separately with two different respondents. In total, 19 interviews were conducted; they were audio recorded and later transcribed. Key informants were asked to send documentation related to their organizations audit method(s) prior to the interview. Any additional documentation mentioned in the interviews was also requested. All organizations made documentation available to researchers. A nal source of documentation was organizations websites. By these means, the research team acquired various documents used by auditors in the eld, copies of procedures used in applying the audit methods, and illustrative nal audit reports. At the conclusion of the key informant interview, permission was sought for observing an upcoming audit. Of the 17 eligible audit methods, eight were observed when an actual audit was being conducted. The researcher accompanied the auditing team, but played no active role. Another two audit methods were observed through a walk-through demonstration by an auditor. The reasons for not observing the remaining seven methods were as follows: scheduling conict (n = 1 case), refusal (n = 2) and no audits conducted in Ontario during the researchers available period of
Table 1 Audit methods classied by content typology. Types of audit methods Audit content featuresa Legislative and regulatory requirements Legal Compliance Only Legal Compliance Plus OHSMSd Basic Comprehensive OHSMS Comprehensive Some Some Some Some Some

7 months (n = 4). The observation period was limited to a single day in cases when an audit involved multiple days of activities. Detailed eld notes were taken during the observation of the audits. 2.3. Analysis Our unit of analysis (case) was an individual audit method with its associated auditors and program. Because some organizations contributed more than one method to the study, cases were nested within auditing organizations. There were two levels of cross-case comparison: a high level, limited overview of all 17 cases and a more in-depth, comprehensive analysis of ve selected cases. The overview analysis was based on the interview transcripts, audit documents and internet material. It involved the extraction of data on specic topics:  whether the audit assessed compliance with legislation and regulations,  the extent to which the content of the audit method was intended to go beyond legislative and regulatory compliance (i.e. best practices),  whether the content of the audit method was portrayed as using a management system framework (Redinger and Levine, 1998),  time taken by the auditing organization to conduct the audit,  cost of the audit to the client workplace, and  purpose of the audit method. Data gathered on the rst three topics were used to construct a typology of audit content, which was developed for this project (Table 1). 2.3.1. Selection of in-depth cases The research projects funding did not allow in-depth study of all cases, and so cases were selected for in-depth study on the basis of whether measurement properties were relatively more important, in light of their intended application. In particular, it did not seem sensible to study instruments in-depth if they were only intended to be a rather imprecise or crude measure of OHS management quality. Furthermore, it seemed that the quality of information from the audit would have additional importance if larger risks to workers were involved or if there were organizational consequences. Thus, after considering the information from the key informant interviews, audit-related documents and auditing organization websites, the audit methods were assessed according to three criteria developed by the research team:  whether they were used in a performance measurement application (i.e. organizational comparisons; monitoring change in performance; determining whether a rm had met a particular quantitative standard),

Number of audit methods Best practices in OHSb No Some Some Comprehensive Comprehensive Management system frameworkc No No Yes No Yes 3 3 2 4 5

a Features were determined according to the nature and intended use of the audit method, as determined from audit program documentation, websites of auditing organizations and key informant interviews. b Best practices in OHS are practices based on the recommendations of OHS experts, but not explicitly specied in legislation or and regulations. c Management system framework refers to the conception of OHS management as involving the plan-do-check-act cycle. d OHSMS = occupational health and safety management system.

184

L.S. Robson et al. / Safety Science 50 (2012) 181189

 whether there were external consequences associated with the audit results (e.g. recognition rewards, nancial rewards, nancial penalties), and  whether the organization applying the method operated in a sector other than a low risk sector. The ve audit methods that met all three criteria were selected as in-depth cases; each was from a different organization. Observational data were available for four of the methods. 2.3.2. Case and cross-case analysis of in-depth cases A case summary for each of the ve cases was prepared by reviewing all transcripts, documents and eld notes related to it. A summary of the material was constructed using an outline with 35 topics based on the conceptual framework for the study (Fig. 1). Major sections of the outline included the following:  audit program context (internal and external to the auditing organization),  audit program structure and processes (e.g. structure, recruitment of auditors, training, quality control, data storage),  audit program challenges (internal and external),  auditor characteristics (e.g. knowledge and skills, beliefs and attitudes, independence, rigor),  audit method (e.g., content development, gaps in content, sector specicity, steps and coordination, data sources, sampling, data transformation, audit output, reliability assurance such as decision aids), and  feasibility of measurement study. (The complete outline is available to the reader upon request.) Although the outline was structured, there was an option of including new themes, which did arise. The cross-case analysis was carried out by two researchers. First, both reviewed all ve case summaries so they were familiar with the entire data set. Next, analysis was conducted on one case summary topic at a time, with the topics shared between the two researchers. A researcher reviewed the data on a given topic for all ve cases. A descriptive summary of themes, similarities and differences across the cases was prepared. In addition, the data on a topic were considered from three perspectives to see whether there were implications for audit reliability and validity:  whether an auditing practice had been identied by participants as valuable,  concordance of practices with the ISO 19011 standard for management system auditing (ISO, 2002), and  concordance with research literature on auditing and research methodology. 2.4. Feasibility analysis The feasibility of future studies of the reliability and validity of audit methods involving direct use of audit data was based on several considerations:  whether the nature of the current applications of the audit method warrants a measurement study (i.e. used in a performance measurement application),  whether the output from the audit method was qualitative or quantitative in nature,  acceptability of proposed research procedures to workplaces (i.e. repeating audit after short period of time, having researcher observer present during an audit, and having more than one auditor present during an audit),

 availability of audit data in a database, and  volume of activity of the auditing program. These criteria were applied to the ve methods selected for indepth analysis. 3. Results and discussion The results are presented in two sections. First is a high level overview of all audit methods that met the eligibility criteria. Second are the ndings from the in-depth cross-case analysis. It should be noted that although we use the term audit to refer to all methods, there were other terms in use by auditing organizations, such as evaluation and assessment. Similarly, we use the term auditor, when in fact those who carried out the audits sometimes had other job titles, such as consultant or evaluator. 3.1. Cross-case overview Seventeen audit methods, developed by 10 organizations, met the eligibility criteria of the study. The following section gives an overview of the methods by describing the reasons why workplaces undertook audits, the nature of their content, the time taken to conduct an audit and the cost of an audit to the workplace. 3.1.1. Reasons workplaces undertook audits A large majority of the 17 audit methods were applied in a context in which workplaces had voluntarily undertaken the audit. The reasons for rms undertaking these voluntary audits were varied:  Initial needs assessment, when rms were at an early stage in OHS management development.  Assurance of a rms legal compliance; i.e. risk management.  Assurance to corporate head ofce and shareholders of OHS management system quality.  Feedback on the quality of a rms OHS management system (for rms to know where they stand), often following improvement efforts.  Identication of further opportunities for continuous improvement.  Identication of opportunities for improvement before a scheduled mandatory audit for rms with poor OHS claims experiences delivered by the workers compensation (Workplace Safety and Insurance Board, 2011b).  Reduction of the likelihood of selection for the mandatory audit program.  Negotiated alternative to the mandatory audit program.  Assurance to external parties of OHS management system quality, in order to enhance public image generally or to bring about known benecial consequences, including contract opportunities, recognition rewards and nancial rewards. Eight of the 17 methods were used in a performance measurement application: determining whether a certain level of performance had been met, inter- or intra-rm comparisons of performance, or monitoring performance over time. 3.1.2. Types of audits by content All audit methods were classied using the content typology developed for this project (Table 1). Eight audit methods were classied into one of the three categories of Legal Compliance Basic, Legislative Compliance Plus and OHSMS Basic. These were simpler audits, typically used in a less formal manner and used primarily with small businesses or with larger rms just starting to develop their OHS management structure and processes. As such, these

L.S. Robson et al. / Safety Science 50 (2012) 181189

185

methods focused on basic legislative and regulatory requirements, reective of the limited OHS aspirations of the workplaces in which these audits were used. In contrast, the other nine audit methods were developed with an aim to being comprehensive with respect to best OHS practices (i.e. classied into one of the two categories of Comprehensive or OHSMS Comprehensive). These methods were typically used by auditing organizations in a more formal manner; and with rms that had aimed not only to simply meet their legislative and regulatory requirements, but to also develop the effectiveness of their OHS management structure and processes. Several auditing organizations administered both a simpler and a more comprehensive audit method, thereby allowing their auditors to select a method appropriate to their client workplaces needs. The auditors avoided overwhelming a small or undeveloped rm with an audit method that was too sophisticated. In a sub-study reported elsewhere (Robson et al., 2010), the content of ve of the instruments in the two comprehensive categories were compared to a recent national voluntary OHS management standard, CSA Z1000 (CSA, 2006), which is explicitly based on a management system framework, which involves the Plan -Do-Check-Act cycle. This standard is a recent expert consensus on what management elements should be included for effective control of OHS risks. As such, it was a suitable denitional standard for the analysis in the absence of a research evidence-based denitional standard. We found that 34% of the content of CSA Z1000 was represented completely in the ve audit methods on average (range 2249%); 40% was represented partially (range 3445%); and 26% was represented not at all (range 1736%). The management elements with the least complete representation on average were the following six elements characteristic of management systems: general OHSMS, which was concerned with integration with other management systems; documentation; objectives and targets; internal audits; management review input; and management review output. Variation across the ve methods was high for the last four of these elements. Instruments developed or revised more recently and those with a management system framework were more likely to have greater representation of this content. To the extent that some of the content of CSA Z1000 had no representation in some of the audit instruments, questions about validity are raised. 3.1.3. Time required for and cost of an audit The amount of personnel-time in the auditing organizations required for the typical application of the audit methods, including preparation and report writing, varied widely, from half a day to 15 days, with a median of two days. For a given method, there was also a fair amount of variation too, depending on the size and complexity of the rm being audited and the familiarity of the auditor with the rm. The cost to rms of having an audit done varied from no-cost (several methods) to thousands of dollars (several methods). The no-cost arrangement arose in cases where a sector-based Health and Safety Association provided services to a rm in its respective individual sector as part of its mandate within the Prevention System. In general, it was the more comprehensive methods and those involving more extensive data collection that had fees associated with them. 3.2. In-depth cross-case analysis Five audit methods were purposively selected for more indepth study. They were chosen on the basis of their measurement properties being relatively more important (Section 2.3.1). In this section, a description is provided for several characteristics of the audit methods, the auditors that used them, the programs in which the auditing methods were used, and the organizational context of

the auditing activities. The implications of these ndings for audit validity and reliability is also discussed. 3.2.1. Organization of OHS audit activities in the auditing organization Auditing comprised the major activity of the program to which it belonged in only one case. In the others, the program that administered auditing also provided OHS management consulting services. Consulting sometimes prepared rms for audits and it sometimes served to assist rms taking action on audit results. Typically, auditing was seen as one of several types of consulting services to workplaces and those conducting audits often had the job title of consultant. Auditing services also had a relationship with training services, since audit results often indicated a need for training. This embedding of auditing services within consulting has implications for audit validity. Research on nancial audits (Meyer et al., 2007) has shown that the relationship with the rm makes a difference on decisions requiring auditor judgment and ISOs Guidelines for Quality and/or Environmental Management Systems Auditing (2002) considers auditor independence to be one of ve fundamental auditing principles: Auditors are independent of the activity being audited and are free from bias and conict of interest. Some organizations were mindful of the conict of interest which arises when a consultant audits a rm to which he or she has consulted: of course its in their interest that the consultants want their rms to do well. One strategy for dealing with this was to intentionally separate consulting and auditing activities within a program by having certain staff who only conducted audits and other staff who only delivered consulting services. Another strategy was for auditors to have both consulting and auditing responsibilities, but not allow one to audit a workplace to which they had consulted. On the other hand, arguments have been made by others that a more valid audit result is achieved when there is an ongoing relationship with a rm because of the auditors greater understanding of the rm (Richard, 2006). This was echoed in our ndings too. Cited advantages to audit results of allowing the same personnel to deliver both audits and other consulting services to client rms were a greater openness of workplace representatives (they tell me things they dont tell others) and an enhanced ability for the auditor to detect a misrepresentation of the truth. Other cited advantages were having a better informed consultant, when implementing actions arising from the audit, and efciency in servicing a given geographical area. Regarding the assignment of auditors to work sites, there were three cases for which only one auditor was typically involved, with exceptions made for auditing large organizations. In a fourth case, there were often two auditors assigned, but they conducted different parts of the audit, and the results of these were loosely integrated by a third person. Finally, in a fth case, two auditors were again involved, but their roles were more tightly integrated: both carried out the document review and key informant interviews; each took a portion of the site inspection and verication interviewing tasks; and both created the report, with one playing the role of lead author and the other of reviewer. This last method of assigning auditors to work sites, while more resource intensive, is clearly a strategy to improve both the reliability and validity of the audit, since random mistakes committed and individual biases held by one auditor may be identied by the other auditor and modied. 3.2.2. Quality control and quality improvement Audit quality was primarily controlled through a procedure whereby an audit report was reviewed by the program manager before it was released to the client. The degree of scrutiny varied, but in at least three cases, the degree of scrutiny was enough to

186

L.S. Robson et al. / Safety Science 50 (2012) 181189

challenge an auditors ndings. In one case, a second auditor who assisted on the audit provided a review before the nal review by a manager. Two audit programs also incorporated random spot checks of auditing in the eld by the program manager, though the managers of these programs indicated time constraints impinged on this activity. Another program had a procedure of regular peer review instead. Programs did not measure inter-auditor consistency as a means of monitoring or evaluating program quality, though the importance of the issue was recognized. Indeed, research has indicated that consistency can sometimes be surprisingly modest even with well-qualied auditors (Dyjack et al., 2003; Kuusisto, 2000; Matheson et al., 2008). ISO 19011 (ISO, 2002) suggests that the consistency in performance between audit teams in similar situations be one of the inputs to reviewing an audit program. Some organizational effort was put into developing and maintaining inter-auditor consistency through initial training, team meetings, regular team review of material, and management or peer review activities. (Notably, in one organization trainees were asked to each make their independent judgment and assign audit scores based on the same information about a workplace. The individual scores of the group members were fed back to them and discrepancies discussed. As training progressed, trainees became more similar in their judgments.) We suspect that that many auditing programs need to address the issue of inter-auditor consistency: rst, determining the implications that variation among auditors or auditing teams have for nal auditing decisions such as passing an audit or not; second, if needed, putting in place new practices to maximize consistency; and third, monitoring consistency through periodic measurement. With regards to ongoing improvement in audit quality, organizations sought feedback from external parties, including workplaces and industry groups, when (re)developing the content of their audit. One sought feedback from workplaces about the entire audit process in the post-audit meeting; and another conducted a survey of clients annually. Less mention was made of seeking feedback from auditors about the audit method, though we expect that would naturally take place during regular meetings of the auditing staff. 3.2.3. Recruitment, training and ongoing development of auditors When recruiting for positions where auditing was a primary function, auditing organizations sought credentials related to auditing. For other positions, which had a dual consultant and auditor function, this was not the case. Industry or sector experience was strongly weighted in the hiring decisions. In all cases, previous OHS experience seemed desirable, but not essential, since auditing organizations developed the OHS expertise of new staff through in-house training. In one organization, there was the additional expectation that staff would achieve the designation of Canadian Registered Safety Professional within ve years of hiring, requiring additional, external study. Organizations were also prepared to develop auditing expertise in their staff, primarily through in-house training. This sometimes included learning ethical guidelines or discussing the potential for bias in an auditors judgment. In one case, trainees got role-playing experience receiving pressure from a workplace trying to inuence audit results; and in resisting that pressure. Research on nancial audits shows an association between the presence of ethical guidelines in the auditing organization and the more ethical nature of choices made by its auditors (Pugrath et al., 2007). The intensity of training on the audit methods varied widely. At one extreme, trainees rst shadowed an experienced auditor for 3 months and were then shadowed by an experienced auditor for three months. At the other end of the spectrum, in situ training consisted of shadowing someone only once or twice, and then

receiving no shadowing by an experienced auditor. This latter type of preparation was less than that outlined in standards for auditing programs and certifying auditors (ISO, 2002; RABQSA International, 2007), which recommends that auditors participate in at least four audits consisting of 20 days of activity as an auditorin-training under the direction and guidance of an audit team leader before conducting an audit on their own. There was no mention of explicit criteria or a test to determine whether an auditor was ready to be independent in the eld. Rather, this determination was made more informally, based on the observations of the trainer and the comfort level of the trainee. The frequency with which auditors applied a particular audit method also varied widely: some audited several days a week; others conducted only one or two audits per year. The degree of familiarity with the method would be expected to affect the reliability of its application. Notably ISO 19011 guidelines (ISO, 2002) state that auditors should maintain and demonstrate their ability through regular participation in audits. . . There were a variety of ways in which the skills of auditors were continuously developed, likely contributing to audit validity and reliability:  Regular (weeklymonthly) meetings among the auditors of the organization.  Reviewing the content of the audit or auditing principles on an ongoing basis in those meetings.  Organizational support for further external training in OHS, people skills or auditing.  Formal mentoring program.  Manager or peer evaluations of auditor performance in the eld (see Section 3.2.2).  Annual performance evaluation. 3.2.4. Auditor characteristics The aspect of auditor qualications has already been covered in the previous section. Observations validated the technical expertise in OHS implied by the reported practices: auditors were procient in their knowledge and took a systematic approach to the audit. Auditors were also generally adept with people, displaying the following attributes: an ability to put people at ease, good listening skills, patience, an encouraging and positive attitude, empathy, respectfulness, professionalism, and an ability to maintain control of the situation. Auditors felt sufciently prepared by their organizations for auditing, believed in the value of their audits to workplaces and felt condent in their abilities to withstand pressure from workplaces. This condence stemmed from the nature the audit method (in particular, the explicitness of the audit criteria), knowledge of the ethics involved, a belief that their organization supported them in resisting pressure, and a consideration of individual and corporate reputations. Occasions of an auditor successfully resisting pressure were recounted by both auditors and key informants. Auditors acknowledged that rms, to varying degrees, put on a faade during an audit, but believed the truth will come out. For example: You have it or you dont. You can back it up or not! A consultant nds that out. The ability of an auditor to see through a faade was variously attributed to the following: auditor experience with audits and workplaces, familiarity with the rm being audited, the multiple data collection methods, the practice of having a worker representative from the joint health and safety committee present during interviews with management key informants, persistent questioning during interviews, and the practice of increasing the stringency of evidence verication when suspicions arise. Nevertheless, there was some variation among auditors in their interviewing skills, which could impact inter-auditor consistency.

L.S. Robson et al. / Safety Science 50 (2012) 181189

187

For example, one auditor consistently followed up on weak or missing responses to questions, whereas another seldom did. One auditor occasionally asked questions in a confusing manner (which might have been related in part to his interview guide having an outline instead of explicit questions). Some auditors often asked leading questions, while others did not. Such questions should be avoided since they bias answers, according to ISO 19011 (ISO, 2002). This study did not investigate the extent to which auditors receive training in interview techniques, but the ndings suggest there may be room for a greater emphasis on this. We note that researchers who use interviews as a means of collecting data recognize the importance of training for interviewing effectiveness (Kvale, 1996; Patton, 1987). 3.2.5. Audit content All ve of the instruments were of the Comprehensive or Comprehensive OHSMS content type (see Section 3.1.2). The period of initial content development for the ve audit methods varied between the late 1980s and the 2000s. Common inuences on the audit content were as follows:  provincial legislation and regulations,  the International Safety Rating System audit, a proprietary audit applied worldwide by the International Loss Control Institute in the United States in the 1980s and currently by Det Norske Veritas in Norway,  sector-based opinion about best OHS practices, including the manufacturers of equipment and supplies, and  the mandatory, comprehensive audit by the Ontario workers compensation insurer that targets rms with poor OHS claim experiences (WSIB, 2011b) and which some consider to be a gold standard in the province. Other inuences included the British CHASE audits (Collison and Booth, 1993; Glendon et al., 1992), management system standards such as OHSAS 18001 (British Standards Institute, 1999), the safety culture concept, the internal responsibility system concept (whereby responsibility for OHS lies at all levels of the organization), and the priority hazards in a sector. Small changes in audit content were made on an ongoing basis to reect changes in legislation and regulations. In summary, audit content was usually based in part on legislation and regulations (i.e. a mandatory standard) and in part on prevailing best practice expert opinion (i.e. a voluntary standard). It was not based directly on locally generated risk data, with the exception of including content related to priority hazards. Audits were made specic to their sector through various means, which should have the effect of enhancing validity: using input from a technical committee with industry representation; drawing from the material of a sector-specic audit from another country; adapting to the typical rm size of the sector; tailoring the site inspection to hazards commonly found in the sector; and making changes on the basis of feedback from sector-based OHS specialists or workers. The sector-based Health and Safety Associations had close ties with its corresponding industry, which facilitated the transfer of expertise from industry. There was some variation in the degree to which the audits addressed musculoskeletal disorders, which is a signicant OHS issue in all industrial sectors. Some audit methods explicitly assessed this area of OHS management, whereas others did not. It was observed that the language used in the audit documentation sometimes excluded this category of risk factors, since the content was oriented toward accidents and acute onset injuries. Prevention of musculoskeletal disorders was the one content area commonly identied by respondents as a current gap in content or an area for future development. All of the audits focused on OHS risks

related to injuries and illnesses compensable in the Ontario workers compensation insurance system. Accordingly, no audit content was concerned with work psychosocial factors (e.g. social support) and only one of them had some content about individual health behaviors. There was great variation in the degree to which the audit documentation contained content on specic hazards and hazard controls. This was reective of the different roles that observations played in the audit. At one extreme there were extensive, fully specied forms used during the site inspection; and the completed forms contributed substantially to the overall results. At the other extreme, there was no specied content for the site inspection and the inspection ndings contributed little to the overall results. There are no evidence-based rules for the degree to which observations should contribute to a management audit result, but there is a general agreement that they should play a role (Cooper, 1998; Health and Safety Executive, 1997). Extensive examination of potential hazards and their controls are considered the domain of a technical audit. However, some examination of these is an important means of conrming the degree of implementation of the OHS management system and its effectiveness. In other words, this type of audit content is relevant to audit validity. 3.2.6. Generation of audit data All methods involved the auditor judging audit evidence against pre-specied audit criteria. In three audit methods, an ordinal level score was assigned for each criterion, with the maximum possible score varying among criteria. In the other two methods, auditors needed only to decide whether a criterion was met or not. There was large variation in the degree of guidance provided to auditors regarding the nature of audit evidence required (i.e. interview, document or observation): some organizations specied the type(s) of evidence needed and others did not. Some organizations provided explicit interview questions to auditors, whereas others provided only an outline of topics. There was also large variation in the degree to which the auditor was given explicit guidance in how to score the evidence against the criteria. Researchers thought that some of the judgments would be difcult, especially when partial marks out of a possible several marks were allowed and no explicit guidance was given. The corollary is that such circumstances would contribute to inconsistency between auditors. To this point, two of the older methods had evolved through revisions in the direction of adding greater specication of audit criteria related to the document review and key informant interview portions of the audit. This had been undertaken in an effort to improve the transparency and reliability of auditor decision-making. 3.2.7. Transformation of audit data to report results In most cases, the assessments for the individual audit criteria were weighted and summed to produce a nal quantitative score. The weights had been determined previously through expert opinion. One key informant said that part of the rationale in developing the weights was to give greater emphasis to key primary prevention elements (e.g. hazard identication and assessment) and less for other elements (e.g. injury reporting). This pattern of weighting was observed with other methods too. Some audit methods have set criteria for passing the audit; e.g. a minimum score. Some also have minimum scores for different sub-sections of the audit to ensure an evenness of competency across audit sections. One auditing organization used a combination of minimum scores and weighting to ensure that the site inspection component played a substantial role in the overall audit assessment. In the words of one key informant in relation to a workplace: They just couldnt have a great paperwork system and pass the audit. All audit methods included some explanation in the nal report when criteria were not fully met. Some but not all methods included recommendations.

188

L.S. Robson et al. / Safety Science 50 (2012) 181189

3.3. Limitations of the current research A limitation of this study is that it was conned to audit methods used by OHS organizations with a prevention mandate in the broader public sector of Ontario. Results can be generalized only with caution to other jurisdictions or to audit methods available through the private sector. However, we note that the methods studied here, at least upon rst examination, are similar to methods found in the research literature in terms of content and methods of scoring (Robson and Bigelow, 2010); and those in the research literature have been used in the eld elsewhere by both private and public sector organizations. A second limitation is that resources permitted in-depth study of only ve methods (though they were selected to be most pertinent to the project). Third, not all audit methods could be observed in the eld, but in cases where observations were possible, they conrmed that the information provided by key informant interviews was accurate.

4. Conclusions This exploratory study provides a rich description of the OHS management audit methods and respective auditing programs in a group of broader public sector organizations in one jurisdiction. This and its companion study (Robson et al., 2010) lay the groundwork for further research into the validity and reliability of OHS management audit methods. The ndings demonstrated wide variation in some aspects of audit methods or their corresponding programs. In addition, some discrepancies between actual auditing practices and ISO 19011 guidance (ISO, 2002) on auditing practices were found. These variations and discrepancies in practice have direct implications for the reliability and validity of audit results used in decision-making. Further research is needed to precisely determine the impact of these determinants on the measurement properties of audit data and on the performance of audit programs. This study determined that such research is feasible with some existing auditing programs. Acknowledgements The authors give thanks to the administrative assistance provided by Diana Pugliese, Lori Chambers and Lyudmila Mansurova, and the generous participation of key informants and auditors from several Ontario occupational health and safety organizations. The project was funded by a research grant provided by the Workplace Safety and Insurance Board (Ontario), but should not be interpreted to mean that the ndings are endorsed by the organization. References
Accident Compensation Corporation, New Zealand, 2008. Applying for the ACC Workplace Safety Management Practices Programme. <http://www.acc.co.nz/ for-business/small-medium-and-large-business/how-to-pay-less/BUS00055> (07.09.10). American National Standards Institute/American Industrial Hygiene Association (AIHA), 2005. ANSI/AIHA Z10-2005. American National Standard for Occupational Health and Safety Management Systems. Beckmerhagen, I.A., Berg, H.P., Karapetrovic, S.V., Willborn, W.O., 2003. Auditing in support of the integration of management systems: a case from the nuclear industry. Managerial Auditing Journal 18, 560568. Blackmore, G.A., Shannon, H.D., 1996. Risk-based safety management auditing. Process Safety and Environment Protection 74, 3844. British Standards Institute, 1999. OHSAS 18001:1999. Occupational health and safety systems. Specication. BSI, London. Bunn III, W.B., Pikelny, D.B., Slavin, T.J., Paralkar, S., 2001. Health, safety, and productivity in a manufacturing environment. Journal of Occupational and Environmental Medicine 43, 4755. Canadian Standards Association, 2006. CSA Z1000-06. Occupational Health and Safety Management. CSA, Mississauga, ON. Collison, J.E., Booth, R.T., 1993. An evaluation of two proprietary health and safety auditing systems. Journal of Health and Safety 9, 3138. Cooper, D., 1998. Safety management system auditing. In: Cooper, D. (Ed.), Improving Safety Culture A Practical Guide. John Wiley & Sons Ltd., Chichester, UK, pp. 144176. Dyjack, D.T., Levine, S.P., 1996. Critical features of an ISO 9001/14001 harmonized health and safety assessment instrument. American Industrial Hygiene Association Journal 57, 929935. Dyjack, D.T., Redinger, C.F., Ridge, R.S., 2003. Health and safety management system audit reliability pilot project. American Industrial Hygiene Association Journal 64, 785791. Eisner, H.S., Leger, J.P., 1988. The international safety rating system in South African mining. Journal of Occupational Accidents 10, 141160. Gay, A.S., New, N.H., 1999. Auditing health and safety management systems: a regulators view. Occupational Medicine 49, 471473. Gillette, D., Campbell, P., Busby, B., 2004. The evolution of a radiation safety audit program for a research institution. Radiation Safety Journal 86, S80S84. Glendon, I., 1995. Safety auditing. Journal of Occupational Health and Safety, Australia and New Zealand 11, 569575. Glendon, A.I., McKenna, E.F., 1995. Human Safety and Risk Management. Chapman and Hall, London. Glendon, A.I., Boyle, A.J., Hewitt, D.M., 1992. Computerized health and safety audit systems. In: Matilla, M., Karwowski, W. (Eds.), Computer Applications in Ergonomics, Occupational Safety and Health. Elsevier, Amsterdam.

3.4. Feasibility and need for future research Future research on the reliability and validity of audit data derived from this population of audit methods was deemed feasible by the research team. First, the way in which some of the audit methods were used in a performance measurement application associated with external consequences for workplaces means that such studies are appropriate. Second, audit results were all in quantitative form, allowing quantitative analyses. Third, some of the necessary research procedures were acceptable to some (though, not all) organizations. Specically, there was sometimes concern about the presence of a researcher during audits. Less problematic was the proposal to have a second of their own auditors present in cases where there would normally be one, which would allow measurement of inter-auditor consistency, though the resource implications was of some concern. We note that efciency could be achieved if the measurement of inter-auditor consistency was an incremental addition to existing in-eld review activity by a peer or manager. On the other hand, the determination of testretest reliability, which would involve repeating audits in workplaces about 24 weeks after the initial assessments, was considered unacceptable by all, because of the burden to both the auditing organization and workplaces. Only one organization had created a database to consolidate the results from all audits. Although there were electronic records kept in other organizations, the need to compile this information would be a barrier to further measurement studies. Finally, some, but few, programs had a sufcient volume of activity to yield sufcient data for research analysis. No organization was conducting its own studies of reliability and validity. This suggests that there is truly a knowledge gap in this area, as was suggested by a review of the research literature (Robson and Bigelow, 2010). Specically, there is a need to study the ability of audit results to discriminate between low and high performing rms and to predict future outcomes such as injury rates (i.e. construct validity). In addition, the weighting of scores when aggregating sub-sections of audit results (considered an aspect of content validity) has until now been based on expert opinion, but this should be compared with an analysis using injury outcome data. Also important is the study of inter-auditor consistency, under realistic eld conditions, as this has important practical implications for program decisions based on audit results, such as the awarding of organizational penalties and rewards. Ultimately, audit methods with optimized properties of reliability and validity should enhance organizational decision-making around prevention, both in an individual workplace or across multiple workplaces.

L.S. Robson et al. / Safety Science 50 (2012) 181189 Government of Alberta, 2009. What is a Certicate of Recognition? <http:// employment.alberta.ca/SFW/334.html> (30.09.09). Grant, J., Brown, D., 2005. The inspector cometh. Canadian HR Reporter 18, 1317. Grant, J., Bricker, R., Shiptsova, R., 1996. Audit quality and professional selfregulation: a social dilemma perspective and laboratory investigation. Auditing: A Journal of Practice & Theory 15, 142156. Health and Safety Executive (HSE), 1997. Successful health and safety management: HSG65. HSE, Sudbury, UK. International Labour Organization (ILO), 2001. Final Report, Meeting of Experts on ILO Guidelines on Occupational Safety and Health Management Systems, Geneva, Switzerland. International Organization for Standardization, 2002. ISO 19011, Guidelines for quality and/or environmental management systems auditing. ISO, Geneva, Switzerland. Karapetrovic, S., Willborn, W., 2000a. Generic audit of management systems: fundamentals. Managerial Auditing Journal 15, 279294. Karapetrovic, S., Willborn, W., 2000b. Quality assurance and effectiveness of audit systems. International Journal of Quality and Reliability Management 17, 679 703. Kennedy, R., Kirwan, B., 1998. Development of a hazard and operability-based method for identifying safety management vulnerabilities in high-risk systems. Safety Science 30, 249274. Kuusisto, A., 2000. Safety management systems: audit tools and reliability of auditing, dissertation. Tampere University of Technology, Tampere, Finland. Kvale, S., 1996. InterViews: An Introduction to Qualitative Research Interviewing. Sage, Thousand Oaks, CA. LaMontagne, A.D., Barbeau, E., Youngstrom, R.A., Lewiton, M., Stoddard, A.M., McLellan, D., et al., 2004. Assessing and intervening on OSH programmes: effectiveness evaluation of the Wellworks-2 intervention in 15 manufacturing worksites. Occupational and Environmental Medicine 61, 651660. Matheson, A., Mughal, W., Thomas-Olson, L., Spiwak, R., Wasdell, M., 2008. Interrater reliability assessment of an OH&S management systems audit tool, National Occupational Injury Research Symposium, Pittsburgh, PA, 2123 October, 2008. Mayne, J., 2005. Ensuring quality for evaluation: lessons from auditors. The Canadian Journal of Program Evaluation 20, 3764. Messier Jr., W.F., Kachelmeier, S.J., Jensen, K.L., 2001. An experimental assessment of recent professional developments in nonstatistical audit sampling guidance. Auditing: A Journal of Practice & Theory 20, 8196. Meyer, M.J., Rigsby, J.T., Boone, J., 2007. The impact of auditorclient relationships on the reversal of rst-time audit qualications. Managerial Accounting Journal 22, 5379. Mohd-Sanusi, Z., Mohd-Iskandar, T., 2007. Audit judgment performance: assessing the effect of performance incentives, effort and task complexity. Managerial Auditing Journal 22, 3452. Nash, J.L., 2005. Who is auditing your safety auditors? Occupational Hazards 67, 31 34. Nelson, M., Tan, H.-T., 2005. Judgment and decision making research in auditing: a task, person, and interpersonal interaction perspective. Auditing: A Journal of Practice & Theory 24S, 4171.

189

Nielsen, K.J., Rasmussen, K., Glasscock, D., Spangenberg, S., 2008. Changes in safety climate and accidents at two identical manufacturing plants. Safety Science 46, 440449. Occupational Safety & Health Administration, United States Department of Labor, 2010. Voluntary Protection Programs. <http://www.osha.gov/dcsp/vpp/ index.html> (07.09.10). OHSAS Project Group, 2007. OHSAS 18001:2007. Occupational Health and Safety Management Systems Requirements. British Standards Institute, London. OLeary, C., 2004. The consistency of individual auditors in performing evaluations. Managing Auditing Journal 19, 597605. Patton, M.Q., 1987. Depth interviewing. In: How to Use Qualitative Methods in Evaluation. Sage Publications, Newbury Park, CA, pp. 108143. Pearse, W., 2002. Club Zero: implementing OHSMS in small to medium fabricated metal product companies. Journal of Occupational Health & Safety 18, 347356. Pugrath, G., Martinov-Bennie, N., Chen, L., 2007. The impact of codes of ethics and experience on auditor judgments. Managerial Auditing Journal 22, 566589. Preuss, L., 1998. On ethical theory in auditing. Managerial Auditing Journal 13, 500 508. RABQSA International, 2007. Criteria for Qualication-based Occupational Health and Safety Auditor Certication, 6th ed. Redinger, C.F., Levine, S.P., 1998. Development and evaluation of the Michigan occupational health and safety management system assessment instrument: a universal OHSMS performance measurement tool. American Industrial Hygiene Association Journal 59, 572581. Richard, C., 2006. Why an auditor cant be competent and independent: a French case study. European Accounting Review 15, 153179. Robson, L.S., Bigelow, P.L., 2010. Measurement properties of occupational health and safety management audits: a systematic literature search and traditional literature synthesis. Canadian Journal of Public Health 10, S34S40. Robson, L.S., Macdonald, S., Van Eerd, D.L., Gray, G.C., Bigelow, P.L., 2010. Something might be missing from OHS audits: ndings from a content validity analysis of ve audit instruments. Journal of Occupational and Environmental Medicine 52, 536543. Robson, L.S., Speers, J.C., Kusiak, R.A., Burns, B.B., 2007. Development of a performance measurement report for the Ontario Prevention System. Policy and Practice in Health and Safety 5, 318. Sweeney, B., Pierce, B., 2004. Management control in audit rms: a qualitative examination. Accounting, Auditing & Accountability Journal 17, 779812. Umar, A., Anandarajan, A., 2004. Dimensions of pressures faced by auditors and its impact on auditors independence. Managerial Auditing Journal 19, 99116. Workplace Safety and Insurance Board, 2011a. The Safety Groups Program. <http:// www.wsib.on.ca/en/community/WSIB/ArticleDetail?vgnextoid=7ac6e35c819d 7210VgnVCM100000449c710aRCRD> (03.06.11). Workplace Safety and Insurance Board, 2011b. Workwell. <http://www.wsib.on.ca/ en/community/WSIB/230/ArticleDetail/24338?vgnextoid=83394c23529d7210 VgnVCM100000449c710aRCRD> (03.06.11).