Until very recently the effectiveness of training and education in controlling occupational health and safety hazards was largely a matter of faith rather than systematic evaluation (Vojtecky and Berkanovic 1984-85; Wallerstein and Weinger 1992). With the rapid expansion of intensive federally-funded training and education programmes in the past decade in the United States, this has begun to change. Educators and researchers are applying more rigorous approaches to evaluating the actual impact of worker training and education on outcome variables such as accident, illness and injury rates and on intermediate variables such as the ability of workers to identify, handle and resolve hazards in their workplaces. The programme that combines chemical emergency training as well as hazardous waste training of the International Chemical Workers Union Center for Worker Health and Safety Education provides a useful example of a well-designed programme which has incorporated effective evaluation into its mission.
The Center was established in Cincinnati, Ohio, in 1988 under a grant which the International Chemical Workers Union (ICWU) received from the National Institute for Environmental Health Sciences to provide training for hazardous waste and emergency response workers. The Center is a cooperative venture of six industrial unions, a local occupational health centre and a university environmental health department. It adopted an empowerment education approach to training and defines its mission broadly as:
… promoting worker abilities to solve problems and to develop union-based strategies for improving health and safety conditions at the worksite (McQuiston et al. 1994).
To evaluate the programme’s effectiveness in this mission the Center conducted long-term follow-up studies with the workers who went through the programme. This comprehensive evaluation went considerably beyond the typical assessment which is conducted immediately following training, and measures trainees’ short-term retention of information and satisfaction with (or reaction to) the education.
Programme and Audience
The course that was the subject of evaluation is a four or five-day chemical emergency/hazardous waste training programme. Those attending the courses are members of six industrial unions and a smaller number of management personnel from some of the plants represented by the unions. Workers who are exposed to substantial releases of hazardous substances or who work with hazardous waste less proximately are eligible to attend. Each class is limited to 24 students so as to promote discussion. The Center encourages local unions to send three or four workers from each site to the course, believing that a core group of workers is more likely than an individual to work effectively to reduce hazards when they return to the workplace.
The programme has established interrelated long-term and short-term goals:
Long-term goal: for workers to become and remain active participants in determining and improving the health and safety conditions under which they work.
Immediate educational goal: to provide students with relevant tools, problem-solving skills, and the confidence needed to use those tools (McQuiston et al. 1994).
In keeping with these goals, instead of focusing on information recall, the programme takes a “process oriented” training approach which seeks “to build self-reliance that stresses knowing when additional information is needed, where to find it, and how to interpret and use it.” (McQuiston et al. 1994.)
The curriculum includes both classroom and hands-on training. Instructional methods emphasize small group problem-solving activities with the active participation of the workers in the training. The development of the course also employed a participatory process involving rank-and-file safety and health leaders, programme staff and consultants. This group evaluated initial pilot courses and recommended revisions of the curriculum, materials and methods based on extensive discussions with trainees. This formative evaluation is an important step in the evaluation process that takes place during programme development, not at the end of the programme.
The course introduces the participants to a range of reference documents on hazardous materials. Students also develop a “risk chart” for their own facility during the course, which they use to evaluate their plant’s hazards and safety and health programmes. These charts form the basis for action plans which create a bridge between what the students learn at the course and what they decide needs to be implemented back in the workplace.
Evaluation Methodology
The Center conducts anonymous pre-training and post-training knowledge tests of participants to document increased levels of knowledge. However, to determine the long-term effectiveness of the programme the Center uses telephone follow-up interviews of students 12 months after training. One attendee from each local union is interviewed while every manager attendee is interviewed. The survey measures outcomes in five major areas:
- students’ ongoing use of resource and reference materials introduced during training
- the amount of secondary training, that is, training conducted by participants for co-workers back at the worksite following attendance at the Center course
- trainee attempts and successes in obtaining changes in worksite emergency response or hazardous waste programmes, procedures or equipment
- post-training improvements in the way spills are handled at the worksite
- students' perceptions of training programme effectiveness.
The most recent published results of this evaluation are based on 481 union respondents, each representing a distinct worksite, and 50 management respondents. The response rates to the interviews were 91.9% for union respondents and 61.7% for management.
Results and Implications
Use of resource materials
Of the six major resource materials introduced in the course, all except the risk chart were used by at least 60% of the union and management trainees. The NIOSH Pocket Guide to Chemical Hazards and the Center’s training manual were the most widely used.
Training of co-workers
Almost 80% of the union trainees and 72% of management provided training to co-workers back at the worksite. The average number of co-workers taught (70) and the average length of training (9.7 hours) were substantial. Of special significance was that more than half of the union trainees taught managers at their worksites. Secondary training covered a wide range of topics, including chemical identification, selection and use of personal protective equipment, health effects, emergency response and use of reference materials.
Obtaining worksite improvements
The interviews asked a series of questions related to attempts to improve company programmes, practices and equipment in 11 different areas, including the following seven especially important ones:
- health effects training
- availability of material safety data sheets
- chemical labelling
- respirator availability, testing and training
- gloves and protective clothing
- emergency response
- decontamination procedures.
The questions determined whether respondents felt changes were needed and, if so, whether improvements had been made.
In general, union respondents felt greater need for and attempted more improvements than management, although the degree of difference varied with specific areas. Still fairly high percentages of both unions and management reported attempted improvements in most areas. Success rates over the eleven areas ranged from 44 to 90% for unionists and from 76 to 100% for managers.
Spill response
Questions concerning spills and releases were intended to ascertain whether attendance at the course had changed the way spills were handled. Workers and managers reported a total of 342 serious spills in the year following their training. Around 60% of those reporting spills indicated that the spills were handled differently because of the training. More detailed questions were subsequently added to the survey to collect additional qualitative and quantitative data. The evaluation study provides workers’ comments on specific spills and the role the training played in responding to them. Two examples are quoted below:
Following training the proper equipment was issued. Everything was done by the books. We have come a long way since we formed a team. The training was worthwhile. We don’t have to worry about the company, now we can judge for ourselves what we need.
The training helped by informing the safety committee about the chain of command. We are better prepared and coordination through all departments has improved.
Preparedness
The great majority of union and management respondents felt that they are “much better” or “somewhat better” prepared to handle hazardous chemicals and emergencies as a result of the training.
Conclusion
This case illustrates many of the fundamentals of training and education programme design and evaluation. The goals and objectives of the educational programme are explicitly stated. Social action objectives regarding workers’ ability to think and act for themselves and advocate for systemic changes are prominent along with the more immediate knowledge and behaviour objectives. The training methods are chosen with these objectives in mind. The evaluation methods measure the achievement of these objectives by discovering how the trainees applied the material from the course in their own work environments over the long term. They measure training impact on specific outcomes such as spill response and on intermediate variables such as the extent to which training is passed on to other workers and how course participants use resource materials.