| The Investigation Process Research Resource Site A Pro Bono site with hundreds of resources for Investigation Investigators | |||||
| Home Page | Site Guidance | FAQs | Old News | Site inputs | Forums |
|
INVESTIGATING
INVESTIGATIONS to advance the State-of-the-Art of investigations, through investigation process research. Research Resources: Search site for:: Launched Aug 26 1996. |
ASSESSMENT OF ACCIDENTINVESTIGATION METHODS
FOR
WILDLAND FIREFIGHTING INCIDENTS
BY
CASE STUDY METHOD
BY
STEVE MUNSON
PRESENTED
IN PARTIAL FULFILLMENT OF THE REQUIREMENTS FOR THE DEGREE OF MASTER OF SCIENCE
IN FORESTRY
UNIVERSITY OF MONTANA
1999 APPROVED
BY:
------------------------------------------ CHAIRPERSON ------------------------------------------ DEAN,
GRADUATE SCHOOL
------------------------------------------ DATE Munson, Steve. M.S. January 2000 ForestryApplication
of Accident Investigation Methods to Wildland Firefighting by Case Study Method
(112 p.)
Director:
Ronald H. Wakimoto
Wildland
firefighting is an inherently risky occupation. Firefighter entrapments,
burnovers, and fire related fatalities continue to occur on an annual basis
throughout the United States. The purpose of this thesis was to identify, from
USDA Forest Service recommendations, an accident investigation method that
would be most applicable to wildland firefighting. The most applicable
method(s) would best be able to pinpoint causal factors and identify areas
where future occurrences could be reduced.
This
thesis examined a single event as a case study, the 1994 South Canyon Fire. Due
to the volume of published material and its position as an extreme case, this
fire was determined to be a suitable study. The South Canyon Fire was
reinvestigated utilizing each of the U S Forest Service proposed methods.
Wildland fire experts evaluated each method according to six criteria. This
determined an overall ranking used to determine the applicable method(s).
Results
suggest that two methods, The Sequential Timing and Events Process (STEP) and
Fault Tree Analysis were acceptable accident investigation techniques. Each
method had strengths (and weaknesses) in distinct areas. The third evaluated
method, Controls/Barriers Analysis was determined to be not as applicable to
wildland firefighter entrapments. Used individually or as a composite/cross
reference application, these two methods would be valuable tools in
investigating wildland firefighter entrapments.
Research
indicates that a composite model that utilizes the strengths of each method
would be the most valuable in determining the accident causal factors that once
identified, would lead to reduced incidents/accidents in the future. It is
recommended the future accident investigations should use both methods
separately, in conjunction, and as a third composite method in order to
revalidate thesis findings. In addition, this thesis identified the need to
utilize a reliable, established accident investigation method for wildland
firefighting entrapments. An applicable method would determine causal factors
for near misses and accidents in order to track, mitigate, and identify areas
in need of revision. These areas must be identified if future firefighter
entrapments are to be reduced.
ACKNOWLEDGEMENTS
The
author would like to gratefully acknowledge all those people that gave their
time and energy to assist in this thesis. First, to Professor Ron Wakimoto, for
all his guidance, ideas, and insight into the project. And to the other
committee members, Professors Mike Patterson and William Borrie, and Dr. Ted
Putnam. Their support and assistance were invaluable and stretched my thoughts
and education across other fields. And to my friends and family who offered
constructive criticism, proof reading skills, and asked some tough questions, I
greatly appreciate the help. Also the grad school partners, Lisa and Kristen,
both of which where there on the long road. And finally to Bernie, who has
given me more than I could ever repay.
TABLE OF CONTENTS
CHAPTER 2 GOALS AND OBJECTIVES CHAPTER 3 PREVIOUS WORK AND PRESENT OUTLOOK USDA FOREST SERVICE PROPOSED METHODS CHAPTER 4 METHODS
LIMITATIONS AND GENERALIZABILITY CASE STUDY BACKGROUND CASE STUDY APPLICATION ANALYSIS CHAPTER 7 DISCUSSION LITERATURE CITED Appendix A EVALUATION PACKAGE Appendix B USDA FOREST SERVICE DOCUMENTS LIST OF FIGURES
LIST OF TABLES
|
CHAPTER 2 - GOALS AND OBJECTIVESThe
purpose of this thesis is to identify a comprehensive, easily utilized, and
systematic accident investigation method derived from US Forest Service (1998)
recommendations that could determine most causal factors that have led to
wildland firefighter fire burnover accidents (entrapments). The goal was to
define a method that identified causal factors. Once identified, these factors
would be addressed to prevent future accidents, reduce risk and hazard, and
monitor safety programs. This method should be; 1) directly applicable in the
field environment with a minimum of formal instruction, 2) required to be
objective, proceduralized, and systematic to reduce or eliminate investigator
bias and subjective analysis, 3) discipline the investigator and promote
logical interpretation by others, and 4) reliable (testable) and document the
accident process and identify any gaps in knowledge discovered in the
investigation. The hypothesis of this thesis is that one of the three accident
investigation methods derived from current USDA Forest Service recommendations
would be the most applicable to wildland firefighting incidents/accidents using
the proposed evaluation methodology. The investigation methods would each be
used to evaluate the 1994 South Canyon Fire and the twelve “West Flank
Group” fatalities and assigned ratings as to their overall ability to
meet
the proposed criteria. This will be the initial test of the selected accident
method proposed for future wildland firefighting entrapments. Subsequent direct
application to ongoing accidents/incidents would increase the validity and
reliability of the proposed method.
|
CHAPTER 3 - PREVIOUS WORK AND PRESENT OUTLOOKACCIDENT METHODS OVERVIEW Accident
investigation methods have developed significantly throughout the industrial
age into the “age of the organizational accident” (Reason, 1990).
As technologies advanced and systems became more complex, many disciplines have
researched the sources of human, machine, and organizational failures that have
led to accidents. Engineers, economists, psychologists, sociologists,
attorneys, insurance companies, and industry managers were among the most
prominent disciplines to actively seek out causes of accidents. The realm of
accident investigation research currently encompasses risk management, problem
solving, decision-making, human error, organizational safety culture, safety
systems, and other human, machine, and environmental interactions. As Kjellen
(1987) remarked, “The development of the necessary means to reduce risk
of accidents involves a multidisciplinary approach and a close cooperation
between theory and practice.” The integration of disciplines has
facilitated a broader perspective and the ability to examine causative factors
more accurately. This collaboration will remain essential to the ongoing search
for the understanding and insights into the causes of accidents.
The
perceptions of accident causes have been categorized into five main areas
according to their history, limitations, and applications. Figure 2 illustrates
the various approaches to accident investigations. Three additional approaches
to accident investigations will be presented following the five perceptions
summary. They do not meet the five previous categories criteria due to their
investigative nature. These three approaches are Change Analysis, Managerial
Failures approaches, and Multi-faceted/proactive approaches. General overviews
of the various methods will be included.
|
Figure 2. Diagram showing the dominate five perceptions of accident causation (Benner
1975).
SINGLE EVENT CONCEPTHistory The
first perception of accident causation is the single event concept. This
concept focuses on the premise that accidents are caused by a single event.
This simple model exemplifies the quest for the “cause” of what
occurred. The search for a scapegoat and taking care of the scapegoat would
solve the problem. This concept is the most widely perceived and least complex.
The public and media typically utilize this concept when they ask “what
caused the accident?”
Limitations The
single events concept is limited in its ability to see the accident as a
process or sequence of events in time. The factors that may contribute to the
accident are not identified or pursued due to the fact that the
“real” cause is obvious and visible. Causes that may underline
human behavior are rarely determined.
Application Current
applications are primarily apparent in how the public and media view accidents.
This viewpoint is reinforced by findings such as when an airline accident was
caused by “pilot error”. Police citations are another example of
the perception.
CHAIN OF EVENTS CONCEPTHistory
The
chain of events concept or domino theory was originally developed by Heinrich
(1941). The basic concept implied that accidents resulted from a sequence of
events that led to an accident. Like a row of dominos, once the sequence began
each event led to the next until an accident occurred. Intervention at any
point along the events sequence could halt the accident process and eliminate
the unwanted results. An unsafe act starts the chain of events that began with
an unsafe condition.
Limitations This
concept is limited by the linear progression characteristic of the model.
Interactions among events, contributing causes, and the duration and timing of
each event limit the identification of all causal factors.
Applications The
current use of this concept is prevalent in the legal field that attempts to
reconstruct the sequence of events that led to the accident.
STOCHASTIC EVENTS CONCEPTHistory The
prevailing idea behind this concept is the gathering of data and facts in order
to isolate the factors not due to chance. The model searches for variables
common to all accidents. This approach utilizes statistical comparisons to
search for causal factors present in accidents.
Limitations The
Stochastic Events approach is limited by its dependency on the data reported by
the accident investigators. The gathering of the facts supercedes any attempts
at analysis. The validity is lacking in this procedure because of investigators
assumptions about the cause bias the reporting of the facts. The procedure is
undisciplined and unstructured.
Applications This
concept is practiced to a large degree by the USDA Forest Service. A form is
completed that has pre-identified contributing and causal factors to consider
and record (see Appendix Table B.2). The “just the facts” approach
is a commonly accepted way of investigating accidents in industry, law
enforcement, and the medical profession.
BRANCHED TREE PERCEPTIONHistory
The development of the logic tree perception is illustrated by the following
various accident investigation methods. The Management Oversight and Risk Tree
approach encompassed several analytical techniques in a logic tree format as
integral aspects of the investigative process. These techniques include Fault
Tree analysis, The Haddon Matrix, Barriers Analysis, and Events and Causal
Factors Charting.
Management
Oversight And Risk Tree
|
|
Figure 3. National Safety Council (1984) diagram of the “home accident ”
sequence and possible causal factors that can lead to an accident.
investigation
methods. Interactions and relationships between factors could be easily
depicted and questioned if information was absent or questions arose.
Multifaceted problems with long or complex causal factor chains could be better
analyzed by this method (Gertman and Blackman 1994). This is because the
accident sequence can be visibly outlined and worked backward from the accident
to reveal causal factors as they lead to and interact with, each other. Benner
and his associates (in Ferry 1988), while working for the National
Transportation Safety Board were innovators in the development of sequence
diagrams and the charting processes such as ECFC. The charting process visually
allowed for evaluation of factors that sequentially led to an accident.
Limitations Limitations
of this method include the amount of time required to conduct the analysis and
the need for investigator familiarity with the process in which the accident
occurred (Gertman and Blackman 1994). The absence of a time scale to relate
simultaneous events to each other is another limitation of the method.
|
|
Figure 4. Event and Causal Factors Charting (ECFC) diagram showing the integration of
systematic factors with contributing factors leading to direct causal factors
(Ferry 1988).
|
Figure 5. Event and Causal Factors Charting diagram of an example of the accident
sequence where a firefighter gets burned.
MULTILINEAR METHODOLOGIES
Multilinear
Events Sequence (Mes)
|
|
Figure 6. Multilinear Events Sequence (MES) diagram showing the analysis process in
reconstructing the accident sequence. Note the time scale at the bottom and the
incorporation of simultaneous conditions and/or events (Benner 1975).
|
|
Figure 7. Multilinear Events Sequence example illustrating accident process of
firefighter receiving burns.
The
second major contribution the MES process has embodied is a more
distinct
time frame than was present in antecedent linear models. The timeline has aided
investigators by structuring the search for relevant factors and events. Newly
discovered conditions or events could be easily tested and then inconsistencies
and gaps in knowledge could be more readily determined. The Civil Aeronautics
Board (1962) in the early 60’s incorporated a time line when flight data
recorders came into use.
Limitations This
method may be limited by its perceived complexity in developing the framework
to process all the information gathered. Underlying human factors may also be
more difficult to identify if experience in the relevant work tasks is limited.
Application Currently
the National Transportation Safety Board utilizes a similar concept as part of
a hybrid approach. Their approach involves a quantitative assessment of
engineering structures, the environment, and the time line analysis (Gertman
and Blackman 1994).
Additional
Approaches
|
Figure 8. Change analysis diagram depicting the concept and process that compares the
pre-accident situation to the post-accident consequence. The process aids in
determining the changes to the system that had to occur for an accident to be
initiated. (Ammerman 1998).
Managerial Failures ApproachesHistory
and General Overview
Many
prominent accident investigators have stated the position that accidents have
their roots in managerial and organizational failures (Fine 1976, Weaver 1973,
Grimaldi
and Simonds 1984, Petersen 1975, Vaughan 1996a, 1996b). Fine (1976), for
instance, summarized this concept when he stated, “all accidents and
hazards are indicators of management failure.” Vaughan (1996b) directly
related that concept to the USDA Forest Service firefighting community when she
said that they are politically vulnerable and the policy decisions that they
make directly affect how operations are done on the ground and how lower level
employees make decisions. She concluded by saying “top decision makers
are thus irrevocably responsible for safety.” Just as with the Challenger
disaster, the USDA Forest Service has had warning signs of potential danger
latent within the organization prior to the South Canyon Fire. These latent
conditions brought about the transition of seemingly small, minor decisions
towards what was described as an “incremental descent into poor
judgement” (Turner 1978). Reason (1991) used the medical term
“resident pathogens” to describe latent conditions in an
organization that may have laid dormant for years until a triggering mechanism
broke through the system defenses and barriers to cause an accident. He
emphasized that these resident pathogens could be identified with
“adequate access and system knowledge.”
One
of several investigation techniques that looked more deeply into management
failures and their contribution to accidents was TOR, the Technic of Operations
Review (Weaver 1973). TOR was developed for the Wausau Insurance Companies to
identify management oversight and omissions. Findings from accident
investigations were analyzed using a four-step process. The process led
investigators through a work sheet of eight general categories. The
investigative team was to identify a direct cause to initiate the process. They
then followed the factors that contributed to the direct cause that the
worksheet proposed. This identified possible contributing factors to the
accident and investigators eliminated factors that did not apply. The
sequential process was used to locate the potential problem areas within the
organization. Weaver recognized that though simple to use, TOR required an
objective mid-level management team to be effective at exposing organizational
deficiencies.
Another
systems approach to accident investigations that has directly implicated
management failure was Fine’s method (1976). While working for the Naval
Surface Weapons Center, he had developed an approach based on the premise that
for each causal factor identified in an investigation, the question needed to
be asked, “Where did management fail?” His technique proposed
fifteen possible management failures linked to each causal factor found in any
mishap. Fine stipulated that expertise and sound judgement by the investigators
was required in order to trace all the direct and indirect factors attributed
to higher level management.
Multi-Faceted/Proactive
Approaches
|
Figure 9. The US Army and Department of Defense “3W’s” approach to
accident investigation, analysis, and prevention (PAM 385-40 1998).
ENERGY AND TRACE HAZARD ANALYSISEnergy
and Trace Hazard Analysis is an integral aspect of the Management Oversight and
Risk Tree process previously discussed. Gibson (1961) introduced the concept of
energy flow and barriers in the classification of accident process. This
concept focused on various vectors of potentially harmful energy sources
(chemical, kinetic, electrical, and thermal) and the barriers provided to
protect from their harmful effects (Figure 10). Identification of these
barriers that have been compromised aided development of improved or additional
defenses. Gibson’s search into safety analysis looked for a more
behavioral approach in that these barriers can be supervisory, managerial, or
organizational/cultural as well as physical. He stressed that these barriers
may have worker behavioral implications in that these non-physical barriers are
less visible and easier to violate without immediate adverse consequences.
Administrative barriers such as rules and regulations are much easier to
transgress than physical barriers such as containment walls or wire insulation.
Examples of administrative barriers present in the wildland firefighting
profession are the 10 Standard Firefighting Orders and 18 Watch Out Situations
(see Appendix Table B.2). Examples of physical barriers would be fire shelters
and personal protective equipment, such as fire resistant clothing, hard hats,
gloves, neck shrouds, and leather boots. But a physical barrier would include
any boundary of thermal protection between the firefighter and the fire itself.
|
|
Figure 10. Barrier Analysis conceptual framework where barriers/controls prevent the
unwanted transfer of energy from a hazard to a valued target. These barriers
may be physical (protective clothing) or administrative(safety rules)(EG&G
Idaho 1985).
Haddon
(1973) further developed the unwanted transfer of energy concept and its
control by various measures or barriers. Again, sources of energy were derived
from chemical, kinetic, electrical, and thermal vectors. He specified ten types
of barriers to the accidental transfer of energy. These barriers are intended to:
An
analysis of an accident sequence can be initiated by investigating a) the
energy source(s) and their paths, b) the people or objects that are vulnerable
to the unwanted energy flow, c) the barriers and controls that were designed to
protect vulnerable people and objects, and finally, d) the precursor events of
energy transfers and barrier failures that lead to the accident. The ten types
of barriers outlined above show examples of their applicability to firefighting
operations. Barriers Analysis also allows safety personnel or investigators to
examine the sequence of events/causes that may have led up to the accident.
Ammerman (1998) provided a worksheet to document and track accident
consequences, barriers in place and the reasons for barrier failure in any
accident where there was loss of property or injury. The Department of Energy
(1992) expanded on the barriers concept by implementing a six-step process that
identified the barriers, found the ones that failed, identified how they
failed, then why, where barriers may have prevented the accident and finally
validated the findings from the information learned. This process was
incorporated into this thesis and documented using DOE’s recommended
worksheet (see Appendix Table A.1).
The
Barrier Analysis method is currently one aspect of the accident investigation
process (and MORT process) utilized by the Department of Energy (Trost and
Nertney 1985, Buys and Clark 1995) and proposed by the USDA Forest Service
(USDA Forest Service 1998). Though recommended, this method has not been
utilized as of this date by the Forest Service. Though the concepts and
processes are identical, Barriers Analysis is also called Energy Trace Hazard
Identification, Control Barriers Analysis, and similar variations of those names.
Barrier
Analysis is limited by requiring investigators to have a good working knowledge
of the task process in order to properly identify and evaluate
barriers/controls and possible avenues of barrier penetration (Gertman and
Blackman 1994). Since barriers may be administrative, managerial, and
supervisory, as well as physical, a competent overall knowledge of the work
process is essential.
FAULT TREE ANALYSIS Heinrich
(1941) developed the methodology that preceded and formed the basis for Fault
Tree Analysis. He illustrated the linear sequence of factors in accident
causation by using a domino theory. The theory stated that a disturbance that
caused any one of the five identified components of the sequence to fail would
set off a chain-of-events that led to an accident. The five in the sequence
were 1) ancestry and social environment, 2) conditions and fault of person, 3)
unsafe act, 4) unsafe condition and 5) injury. He showed that by intervention
at any point along the sequence an accident/injury could be prevented. This
theory has been modified and updated (Baker 1953, Marcum 1978, Heinrich et al
1980), and has wide applicability in current automobile accident and law
enforcement investigations.
Similar
linear sequence models such as Critical Path Analysis (CPA), Gantt Charts, and
Program Evaluation Research Task (PERT), were initially used in the
1950’s and 60’s as planning tools (Lockyer 1964). Though many names
were given to their process they were very similar in their goals and methods.
They provided a graphical display of activities linked to events by arrows in
order to plan complex projects. The process illustrated a flow (path) from one
task sequence to the next and incorporated time frames and interrelationships
between tasks. Projects could then be analyzed by task, the amount of time
needed for each segment and the relationship a task may have with another task.
These methods offered an effective means of project planning, costs analysis,
and time frame considerations by visually outlining the task process (Lockyer
1964). These processes also provided the means to better understand the
interrelationships between and among tasks. This logical depiction of process
flow related directly to analyzing an accident sequence and the precursor
events.
In
the 1960’s Bell Laboratories expanded upon the linear chain of events
concept through missile system safety. They arranged events in a flow chart
that used a proceed/follow logic pattern. Their concept, Fault Tree Analysis
(Figure 11), is generally credited to Watson (1971). Figure 12 illustrates
the fault tree concept as applied to a hypothetical accident where a wildland
firefighter was burned. This analysis concept helped provide a sense of
management by objectives by identifying unwanted events (the top event) and
then systematically and sequentially determining the precursor events. The
objective is the top event and the identification of the preceding causal
factors aid in the management achievement of that objective. Watson’s
Fault Tree Analysis investigation methodology provided a visible, easily
understood and defendable format (1971). The methodology extended the linear
chain of events into a “branched events chains” concept through the
use of “and/or” logic gates. It uses basic Boolean logic in a
hierarchical tree format. Other Boolean terms such as “not” are not
used in Fault Tree Analysis. For example, “C” can only occur when
both “A”
and
“B” occur. If two or more events are required for a cause to happen
then an “and” symbol is used. Another possibility is when only one
of the factors need be present. For “C” to occur, then
“A”
or
“B” occurred. If only one event of two or more are necessary then
an “or” gate is used. The “top event” is the unwanted
result of the accident and causal factors branch out below leading to it. The
downward sequence is continued until the root causes are found or the tree
cannot be further developed. This technique, according to Benner (1975),
“contributed a powerful tool for the investigation of accidents –
both historical and postulated.” Accidents could be investigated or
reinvestigated in the search for causal factors utilizing this method. It
assisted in illuminating areas that may have previously been overlooked by
other means. Numerous approaches to determining accident causal factor using
“branched events chains” reflected the discipline of the
investigations employing it; thus medical doctors
used
an epidemiological approach (agent/host/environment), while psychologists
focused on human factors.
|
|
Figure 11. Fault Tree diagram illustrating a typical failure process, symbols
used,
and the logic sequence leading to an undesired event, a dark room (in Ferry
1988).
|
|
Figure 12. Fault Tree diagram illustrating the deductive process using an example of a
sequence of events in which a firefighter receives burns.
One
key limitation of Fault Tree Analysis is the inability to model time sequences
that are concurrent and interactive (Hendrick and Benner 1987). Brown (1993)
added that only one event could be analyzed at a time and thus primarily
applicable to catastrophic events. Benner (1975) cited similar deficiencies,
most notably that charting analysis methods focus on a single undesired event
and provided no means to indicate the chronological relationships (and the
subsequent concurrent interrelationships) of events. Another limitation is the
restriction inherent in the method whereby causes must be either successes or
failures and degrees of each are not accounted for (Tulsiani and others 1990).
SEQUENTIAL TIMING AND EVENTS PLOTTINGEvaluations
of the multilinear systems safety approach have led to a procedure developed by
Hendrick and Benner (1987). Multilinear systems approaches view accidents as
multiple avenues of causal factors that react to previous factors and may
interact with others throughout the system to ultimately lead to an accident.
The Sequentially Timed and Events Plotting (STEP) procedure was a comprehensive
approach to reconstructing an accident. It was based primarily on Events and
Causal Factors Charting and Multilinear Events Sequencing previously cited. The
key component of the accident reconstruction process was the STEP worksheet
(see Appendix Figures A.3). The worksheet was the documentation that provided
structure, visibility, and organization to data gathering and analysis. It
illustrated the beginning and end of the accident sequence along columns that
represented time. The rows of the worksheet listed the actors, either people or
things, which acted to produce the harmful outcome. Each actor performed one
action, termed an event, that when displayed along a timeline visually showed
the interactions among actors and events. The process subsequently accommodated
events that occurred at the same time. Each event was represented by a block
diagram that displayed the time the event occurred, the information source, the
actor and the action (Figure 13). These event building blocks allowed
investigators to visually recreate the mental motion picture and determine gaps.
By
performing three tests, the accuracy and validity of the entire worksheet could
be assessed. Test number one is the column test to make sure the events
sequence is accurate. Test two is the row test for completeness. The third test
is the necessary and sufficient test to validate what events were necessary and
sufficient to have caused the next event(s). These tests also helped
investigators look for knowledge that may be lacking. This extended the cause
and effect linear model into one that took into account contributing causes and
conditions that occurred simultaneously. Interruptions and questionable cause
and effect relationships could be more readily recognized and investigated than
previous logic diagram techniques.
The
STEP concept was substantially based on the development of a “mental
motion picture” of the accident sequence as a reconstructive tool. The
building blocks of actors and their actions were the “frames” in
which to recreate the “motion picture”. Figure 14 illustrates the
incorporation of building blocks onto the STEP worksheet in order to visualize
the accident “motion picture”. Hendrick and others (1987) proposed
an additional benefit to the STEP methodology. They added that the
identification and utilization of an applicable decision-making model along
with concrete terminology to specifically classify human error would expand the
capabilities of STEP. The decision making model could provide the basis for the
development of a data base to track and analyze human error that was unique to
an occupation, task, and industry. In this thesis, the STEP method is applied
without the human error classification since it is currently unrefined.
|
|
Figure 13. STEP CARD used to consolidate information used to reconstruct an accident
sequence. (Hendrick and Benner 1987).
|
Figure 14. STEP Worksheet illustrating the placement of individual building blocks (of
actors and events) their influence and interaction with one another, and
relative sequence in time (Hendrick and Benner 1987).
CHAPTER 4 - METHODSSTUDY APPROACHThe
application of an accident investigation method to wildland firefighting was
evaluated by examining a single case, a portion of the South Canyon Fire. Case
studies have provided an established, valuable method of study. Yin (1989)
stated that the case study is a “frequent mode of thesis and dissertation
research in... psychology, sociology, political science, anthropology, history
and economics”. The frequent use of case studies is due to the fact that
they allow close in-depth analysis and understanding of specific cases, aid in
understanding unique realms of inquiry, and provide insight into cases that
could not be duplicated experimentally. Reason (1990) has stated that when
sufficient evidence regarding a single case is available, “we are able to
study the interaction of the various causal factors over an extended time scale
in a way that would be difficult to achieve by other means.” This case
study allows for an evaluation of an extreme incident that could not be
replicated by experimental means. Reason added that case studies have taught
us “disasters are very rarely the product of a single monumental
blunder.” He further states that human-made disasters are generally the
result of accumulating, apparently negligible consequences that compound to
contribute to the undesired result. In reference to this specific case study,
Prineville Hot Shot Superintendent Tom Shepard (South Canyon Fire survivor)
echoed Reason when he said, “There was a whole series of events and
circumstances, a change in any one of those would have produced a different
outcome”(Long and Hoover 1994).
Case
studies, as applied to this thesis, involve the investigation into a single
social phenomenon, in this study, the 1994 South Canyon fatality fire. Case
studies can generate innovative new interpretations and concepts by selectively
analyzing a single case (Feagin and others 1991). As laboratory experiments
lend themselves to low-level generalizations, case studies can provide the
means to generalize to theory (Yin 1989). This approach allows for a broader
explanation of how cases that are deviant can provide insight into accident
causation and investigative techniques.
Yin
(1989) mentioned four procedures in order to construct case studies. The first
is asking the right question you would like answered, what is the theory you
are attempting to clarify and investigate? Data collection is the second phase
and must attempt to include as many sources as possible in order to triangulate
(come at the important data for various sides). This provides construct
validity to the research. Data analysis is the third phase and techniques such
as pattern matching, explanation building, and a logic model are techniques
used to establish a chain of evidence that provides validity to the
researcher’s conclusions. This phase provides internal validity. External
validity is accomplished by entertaining rival explanations of the proposed
hypothesis throughout the study. The last phase is the reporting of the
findings and conclusions in which data in the form of tables, spreadsheets,
statistical outputs, interviews, coded worksheets, etc. provide the evidence
necessary to defend the conclusions. Reliability is established through the
case study protocol developed in the research design phase.
Three
accident investigation methods were selected for evaluation. These methods are
the “units of analysis”. The method of investigation is under
examination in this thesis, not the fire itself. As previously mentioned, three
methods, Energy and Trace Hazard Identification (also called Energy Trace
Hazard Analysis, Control/Barrier Analysis or Barrier Analysis), Fault Tree
Analysis and STEP were recommended by the US Forest Service as their newly
established preferred methods (USDA Forest Service 1998). Failure Modes and
Effects Analysis (FMEA) was a fourth recommended method but was not integrated
into the thesis. This was due to its primary quantitative application to
hardware component failure rates in systems (DOE 1992, USDA Forest Service
1998). That particular method utilizes experimentally derived rates of failure
in various components to obtain an overall failure rate. Benner (1985) rated
these three methods highest among 14 methods he evaluated from 17 federal
agencies. Benner utilized 10 criteria derived from OSHA statutes and policy to
rate these methods. Five of these criteria were utilized in this thesis as
Benner’s other criteria were directly applicable to satisfying
OSHA’s mission and not directly to the accident methodology.
Benner’s additional criteria were, a) satisfying, b) functional, c)
direct, d) noncausal, and e) definitive. Criteria utilized in this thesis were
selected because they satisfied standard assessments of reliability and
validation (Benner 1985, Feagin and others 1991). Benner’s reasoning
behind the high ratings was that they were focused on accident causation as a
process where events occurred in a logical sequence. These “events
process” methods showed the interactions between actors and events and
the influence of contributing factors on the accident sequence.
LIMITATIONS AND GENERALIZABILITYOne
caveat Reason (1990) mentioned is the limited information that is available
from past accident investigations and the tendency of documentation to be
“digitized” as opposed to an original, more complex and continuous
nature of “analog” events. Past accident reports lack the
information that was potentially available. The broader, richer, and more
complex possibilities of the original account can be compromised in the written
form. Though this thesis is constrained by this limitation, the systematic
process of each method can not only identify contributing and causal factors
but gaps in knowledge that need further inquiry. The identification of these
gaps is a positive tool for improvement of future investigative procedures and
could identify areas that a particular method takes into account or,
conversely, fails to recognize. Thus the advantages and disadvantages of each
method can be determined. One method may be more applicable to specific causal
areas whereas another may be stronger in another. Determining the strengths and
weaknesses of each method as applied to wildland firefighting entrapments could
provide the theoretical framework for applications to subsequent
accidents/incident investigations. The method determined most applicable would
also provide possibilities for additional research to overcome any inadequacies
inherent in that method.
The
reliability and validation procedures of this case study approach, as applied
to wildland firefighter entrapments, was an important factor in selecting this
method of analysis. Each method used has been previously utilized in various
industries and was found to be valuable tools in accident cause determination
(Benner 1985). Therefore they have been found valid and reliable in other high
risk occupations. This thesis utilized inductive reasoning where specific
observations lead to theory generalization. Yin (1989) stated that “case
studies, like experiments, are generalizable to theoretical propositions, and
not to populations or universes and... the investigator’s goal is to
expand and generalize theories (analytic generalization) and not to enumerate
frequencies (statistical generalization)”. Just as statistics from
experiments that were derived from population samples are generalized to a
larger population, case studies can similarly generalize to a larger theory of
the investigated process itself. As in any scientific experiment, replication
of the results in other case studies can offer additional information and
validation into the phenomenon studied. By examining an extreme case,
generalizing to less complex, less extreme cases could be applicable, reliable,
and valid.
In
addition, case studies can provide invaluable modes of understanding (Yin
1989). Insights gained from their analysis can be incorporated into theories of
error production. Case studies can expand on principles that “can
reasonably be expected to reduce either the occurrence of errors or their
damaging consequences” (Reason 1990). The study of individual cases can
provide understanding into the breadth and scope of human performance
capabilities that laboratory environments could not emulate. It would be
impractical (and unethical) to attempt to replicate extreme circumstances that
model the real world in a laboratory environment. The ability to investigate
and learn from these extremes in human capabilities, high risk decision making
processes, and problem solving under life threatening situations can only be
studied in their complete context from case studies.
CASE STUDY BACKGROUNDBackground
knowledge of the South Canyon Fire studied in this thesis can provide insight
into the work processes, complex interactions, and the accident sequence
itself. This is a general overview from the fire’s inception to the time
of the accident. A more complete account can be found in the published
literature previously cited.
This
fatality fire provided the detailed, published documentation to compare and
contrast the proposed methods of analysis. It was inarguably the most
documented wildland firefighter fatality fire investigated up until that time.
On July 2, 1994 seven miles west of Glenwood Springs, Colorado, lightning
started the South Canyon Fire. Due to the large number of fires that burned on
BLM’s Grand Junction District at that time, the fire was monitored until
July 4 when increased public concern and resource availability lead to the
decision to begin suppression of the fire (IMRT 1994). Local resources made up
of a seven-person BLM/ Forest Service crew arrived and began suppression
activities early on July 5. This group was supervised by Butch Blanco who was
designated the Incident Commander for the fire. Eight smokejumpers with Don
Mackey as “jumper–in-charge” reinforced the local crew later
that evening. When mechanical problems disabled their chainsaws the BLM/Forest
Service crew hiked back down to Interstate 70 to do repairs and return the
following morning. The smokejumper crew worked on the fire till early morning
on the 6
th
when the rolling of burning logs and pine cones made line construction too
hazardous in the dark. They continued line construction on the southeast flank
after dawn (Figure 15). Later that morning, eight additional smokejumpers
parachuted to the fire with jumper–in-charge Eric Hipke. Hipke turned
over the jumper-in-charge responsibility to Dale Longanecker who then became a
line scout. A helispot was cleared near the fire and transport by helicopter of
the twenty members of the Prineville Interagency Hotshot Crew began.
Superintendent Tom Shepard led the Hotshots. After a 0930 reconnaissance
helicopter flight by Blanco and Mackey was made to determine suppression
tactics, Mackey instructed the smokejumpers to begin line construction down the
west flank from the north. They refused the assignment due to the fire activity
at that time, but when the 8 additional jumpers arrived, line construction
began. They were reinforced by nine of the Prineville Hotshots upon their
arrival at 12:30. After eating at the “Lunch Spot” at approximately
14:00, some smokejumpers worked to the south while the Prineville group of nine
and several smokejumpers worked the West Flank. This group has been designated
the “West Flank Group” (Butler et al 1998). The remaining
Prineville Hotshots and the BLM/FS crew worked on the Main Ridge improving
fireline and monitoring for spot fires. At approximately 15:20 that afternoon,
a dry cold front passed the fire area producing increased winds and fire spread
and intensity escalated. At 16:00 the fire had crossed the bottom of the west
drainage and spread up the west side. The firefighters on the west flank were
ordered “to get out of there” by Shepard (IMRT 1994, Butler and
others 1998). The fire then spotted back across to the east side beneath
retreating firefighters on the west flank fireline. The fire moved uphill in
dense Gambel oak vegetation and overran firefighters attempting to escape up to
the Main Ridge on the west flank. Of the forty-nine firefighters assigned to
the fire, twelve perished on the west flank and two helitack personnel perished
when they were overran northwest of the fire. A third group, called the Lunch
Spot Ridge group (Butler et al 1998), deployed fire shelters and survived. A
fourth group, called the Main Ridge group, escaped down the east drainage. An
initial investigative report was published by the USDA, USDI, and USDC (1994)
and followed up by two reports by the Incident Management Review Team (IMRT
1994,1995). The Occupational Safety and Health Administration also investigated
the South Canyon Fire and published a report (OSHA 1995).
|
Figure 15. Photograph of South Canyon Fire area and selected points where major events
occurred. Photograph by Jim Kautz USDA Forest Service. Top of photo is Southeast.
CASE STUDY APPLICATIONThis
case study has investigated a major subunit
of
the South Canyon Fire, the twelve fatalities that occurred on the West Flank of
the fire. It was selected due to its distinct environment separate from the
three other subunits identified and the severity of the consequences (Butler et
al 1998). The other three subunits were the “Main Ridge Group”, the
“Lunch Spot Group”, and the “Helitack Group”. Accident
investigators and safety professionals have agreed that the difference between
a near miss (or near serious) incident and an accident is mainly a matter of
luck or adequate recovery efforts (whether intentional or not)(van der Schaff
et al. 1991). Therefore by examination of the accident subunit the
applicability of the resulting accident method to wildland firefighting
incidents/accidents ranging from relatively minor fire burnovers to severe
consequences may be transferable. When safety barriers are breeched in
occupations where the risk may be high, numerous controls must be circumvented.
When the consequences are extreme and complex (e.g. fatalities) then the most
stringent barriers must be eluded in order for an accident to occur. Therefore
less serious accidents (near misses, etc) which have violated less stringent
controls can be investigated with corresponding success. Inferences drawn from
an accident investigation method could be applied to the range of failures from
near misses to fatality accidents. This thesis would be an “instance of a
broader phenomenon, as part of a larger set of parallel instances.”
(Feagin et al 1991).
ANALYSIS The
following section outlines the operations used in evaluating the three
investigation methods. This section illustrates the three methods of
validation, construct, internal, and external, and the case study protocol as a
means of reliability. This section includes the criteria selected to aid in
determining the most applicable method and the operational procedures utilized.
In addition, techniques were included to measure the consistency and
reliability of the results.
Criteria and Procedures Criteria
have been established in order to determine each method’s applicability
to wildland fire entrapment situations. These criteria were; is the method 1)
realistic, 2) comprehensive, 3) systematic, 4) consistent, 5) visible, 6)
simple and easy to learn? Criteria were derived from the only known source of
accident investigation methods’ evaluations (Benner 1985) and were
adapted to this case study. The criteria will be used by independent judges to
evaluate the three methods. The judges were given a copy of the methods
section in this thesis in order to understand the procedures and criteria.
These ideal criteria are defined as follows:
The
judges utilized a ranking system of these six criteria and assigned a ranking
of 0,1,or 2 to each criterion. They applied each set of criteria to evaluate
each of the three methods. A “0” meant that they did not meet this
criterion. A “1” meant that they addressed the criteria but not
completely and improvement would be required. A “2” meant that they
fully met the criteria. This rating is a summated scale. The narrow span of
the scale was used to reduce indecisiveness on the part of the subject matter
experts, since they were not accident investigation experts. The 3-point scale
allowed them to be more exact in their determinations of how each method
satisfied that particular criterion. The rating scale also follows
Benner’s (1985) approach in that until a more comprehensive scale is
developed to better differentiate levels of compliance to the criterion, a more
simple direct measurement scale is appropriate. This rating process limited the
possible more precise evaluation of each method but provided a basis for
determining accident investigation methods that would require further testing
in field applications. Since each criterion is independent of the others no
weighing factor was applied. Again this follows Benner’s (1985) format he
derived from government statute and “did not conflict with one
another.” Each criterion was evaluated as having no more importance to
determining an appropriate accident investigation method than another. Each was
a valuable determinant to the overall assessment.
Five
subject matter experts evaluated the three analysis techniques by rating how
well they satisfied each criterion. They examined the three methods utilized in
the author’s reinvestigation of the S. Canyon Fire. This was a
purposeful sampling of occupational experts. Since no known list of these
experts exists, they were chosen from the first available. These experts were
not accident investigation experts but wildland firefighting experts. This was
utilized to most accurately emulate real world situations where investigators
may have some investigative experience but their primary occupation and
training is not in these techniques. Each expert had at least fifteen years of
wildland fire suppression experience and is at a minimum qualified at the
Strike Team Leader level. The Forest Service considers experts as those who
teach a particular subject, so the author’s classification can be
considered conservative. This definition of expert was developed by the author
and is a source of author bias but reduced by the conservative definition. To
reduce evaluator bias, none of the subject matter experts consulted or
coordinated with each other in rating the methods. Preconceived impressions
about the South Canyon Fire and its possible causes may have introduced
evaluator bias into the ratings. Since they were evaluating the investigation
methods and not the reinvestigation of the fire, bias should have been reduced.
The
author (also qualified as an expert) reinvestigated the South Canyon Fire using
the three methods so that the Subject Matter Experts could see how the methods
could be used. They then rated the each method using the criteria previously
mentioned and rating how well each method met those criteria. It was stressed
that they were to evaluate the methods themselves and not how well I did used
each method. The subject matter expert’s evaluation package included a
summary explanation of each proposed method compiled from the standard
reference literature (DOE 1992, Ferry 1988, Hendrick and Benner 1987). The
package also included the author’s output worksheets of his
reinvestigation of the South Canyon Fire (see Appendix Figures A.1, A.2, A.3,
and Table A.1). A copy of the study approach along with the author’s
results of the reinvestigation using the three methods was included. The
subject matter expert’s familiarity and knowledge of the basics of the S.
Canyon Fire was assumed.
The
overall totals for each accident investigation method were assessed to
determine the best method. In addition, the degree of agreement among
evaluators (interrater reliability) was utilized to assist in the determination
of the most applicable method and each method’s individual strengths
according to each criterion. Comments by subject matter experts were solicited
to obtain individual impressions and evaluations that may have not been covered
in the assessment process. As no additional literature was found on the
importance of the selected evaluation criteria towards accident investigation
methods (and methodologies), the study used a variation of Benner’s
(1985) criteria, methods and equal-weight approach. No evaluator or criterion
was given more weight than another. This aided in strengthening the internal
validity.
As
a measure of the reliability between evaluators, percentage agreement was
calculated. This was done to access the degree of reliability among scores
assigned to each criterion. The higher the agreement the more valid the ratings
are. Also an additional measure was computed using the index of Perreault and
Leigh (1989). Their index is computed using the following formula:
Ir
= {[(F/N) – 1/k)][k/k – 1)]}
0.5
,
where F is the frequency of agreements between the evaluators, N is the total
number of judgements, and k is the number of categories.
Multivariate
techniques to interrelater reliability were not conducted as the number of
evaluators and criteria were considered too small and would not constitute any
meaningful insight.
The
case study analysis used the pattern matching logic to strengthen and validate
the results (Yin 1989). By comparing patterns predicted for each method, that
is, that they meet selected criteria and are applicable to wildland
firefighting, and matching with the predicted patterns (that they fully meet,
or not meet the criterion), evidence is accumulated in determination of a most
applicable method. Yin stated that though empirically based, the comparison of
patterns and their ability to coincide with established criteria can
“strengthen its internal validity”. The development of empirical,
logically tested evidence would provide internal validation to the case study
analysis process.
As
mentioned previously, this system of evaluating accidental investigation
methods followed an earlier attempt by Benner (1985). He acknowledged the
assumptions and bias inherent in such an evaluation. The systematic accident
method process, documentation of each event sequence, and independent
consultation of subject matter experts reduced the bias. The five evaluators
had no prior experience with any accident investigation method so there was
minimal pre-study bias as to which method may be more applicable. Although the
author used each method to reinvestigate the South Canyon fire, each evaluator
was instructed to use the reinvestigation as a means to evaluate the methods
and not to evaluate how well the author performed the analysis. The author is
not an accident investigator, so the way the method can be used to investigate
an accident is critical not the way the author used it. The experts were
presented the tools to understand the process and application of each method,
and the author’s working example illustrating how it can be used. They
were instructed to analyze how well each method did, and possibility could,
fulfill each criterion.
CHAPTER - 5 RESULTSThe
results of the subject matter experts’ evaluations are presented in Table
1. The STEP accident investigation method received the highest overall rating
with a score of 52 total out of a possible 60 (87%). The Fault Tree Analysis
method received a rating of 51 (85%). The Control/Barriers Analysis method
received a rating of 42 (70%).
For
each criterion evaluated by the experts, the STEP analysis method was rated
highest along with Control/Barriers as the most realistic with a score of 9 out
of a possible 10 (90%). It was rated as the most comprehensive (100%), most
consistent (100%), and tied with Fault Tree Analysis as the easiest to use
(90%). Fault Tree Analysis was rated as the most systematic method with a score
of 12 (100%), the most visible (90%), and tied as the easiest to use (90%). Two
of the evaluators rated Fault Tree Analysis highest in overall applicability
across all criteria to wildland firefighter entrapments, two rated STEP
highest, and one rated both STEP and Fault Tree Analysis as equal. No evaluator
rated Control/Barrier Analysis highest.
The
majority of reliability indexes and percentage agreements calculated were
acceptable within the limits prescribed by Perreault and Leigh (1989) and
Kassarjian(1977). They reported that indexes above .85 were very good and below
.80 may require reevaluation. The total agreement among evaluators for STEP and
the six criterion used to evaluate this method was 80% (0.84, Perreault and
Leigh’s (1989) reliability index). The Fault Tree Analysis method was 83%
(0.86 reliability index). For the Control/Barriers method agreement was 75%
(0.79 reliability index). The overall reliability of evaluator agreement for
all three methods was 80% as a percentage and 0.84 when computed using the
reliability index. These agreement indexes showed that consensus among
evaluators as to their ratings was acceptable.
In
reference to each criterion evaluated, the Control/Barriers and STEP method
were the highest rated and most agreed upon methods for being realistic. Fault
Tree Analysis did not achieve an acceptable agreement index (.63). In
evaluating comprehensiveness, the STEP method rated highest for both overall
score (10) and agreement (1.0, Perreault and Leigh’s reliability index)
and the other two methods had high agreement on a value of “1”
(addressed the criterion but needed improvement). In rating each method as
systematic, Fault Tree Analysis rated highest in both score and agreement.
Agreement was high that Control/Barriers rated a “1” in
inadequately meeting that criterion. The STEP method did not receive an
acceptable overall agreement index (.63). Consistency was highest using STEP
with Fault Tree Analysis and Control/Barriers second. Agreement on the scores
for all three methods was acceptable at above the .80 level. Visibility was
highest using Fault Tree Analysis and of acceptable agreement. Agreement was
high that STEP and Control/Barriers were not adequately visible in their
application. Both the STEP method and Fault Tree Analysis were rated equally
high as to ease of use and evaluator agreement was high. Although it received
a lower score than the other methods, there was not acceptable agreement that
C/B was easy (or not easy) to use. When the criterion “easy to use”
was eliminated from evaluation, overall percentage agreement scores were Fault
Tree Analysis (84%), STEP (80%), and Control/Barriers (70%) and therefore did
not alter the findings.
Evaluator’s
comments were solicited as to the applicability of each method beyond what each
criterion addressed. One evaluator liked the way Fault Tree Analysis visually
presented complex events and the way it showed accidents as a chain-of-events
as opposed to a single random occurrence. Another commented on the way Fault
Tree Analysis led backwards from the accident itself to logically uncover
causes or reveal questions that may have been otherwise overlooked. They
thought that this method might be better at uncovering
managerial/administrative latent factors contributing to the incident than the
other two methods. In contrast, one evaluator responded that the STEP method
appeared more stringent in revealing underlying human causal factors. They
commented that STEP (and Control/Barriers Analysis) provided an approach that
was more likely to distinguish more abstract human factors from hard factual
data considerations and therefore be better at raising questions into human
error causes. The STEP method was cited by one evaluator as the approach that
most visually displayed the actor/action sequence of events and identified
knowledge gaps in the sequence. All evaluators expressed concern that
Control/Barriers Analysis was inadequate in determining causal factors when
applied to wildland firefighting. It had strengths in identifying needed and/or
compromised barriers at an administrative level but the dynamic and highly
variable aspect of the firefighting environment made its application to
investigations inadequate. They commented that it did not appear to be an
adequate tool to probe deeper into possible human error (and
administrative/managerial oversights). The method was good at defining what
control or barrier failed but not why it failed.
Table
1. Results of subject matter expert’s evaluation of the three
investigation methods. A score of “0” meant that the criterion was
not met, a score of “1” meant that this accident method had the
ability to meet that criterion with some improvement. A score of
“2” meant that the criterion was fully met. Consult text for
definitions
.##
table 1
CHAPTER 6 - CONCLUSIONS The
results of this inquiry into an accident investigation method applicable to
wildland firefighter entrapments showed the Sequential Timing and Events
Plotting (STEP) method to be the most desirable method, followed very closely
by Fault Tree Analysis. Both methods together met the majority of the goals and
objectives of the thesis. The total overall scores obtained for both the STEP
and Fault Tree Analysis methods showed the two methods are not likely
significantly different. Both were rated higher overall than Control/Barriers
Analysis. Each accident investigation method had its strengths and weaknesses
as verified by the resulting evaluations of each criterion.
The
STEP method received the highest score and highest number of selected criteria
that evaluators rated highest and in which they concurred. The criteria that
the STEP method rated highest on (realism, comprehensiveness, consistency, and
ease of use) showed this method to be the best investigation process in these
areas. Therefore, overall, the most applicable method would be STEP. Fault Tree
Analysis would be the most desirable method when accident investigators
required a systematic process that was highly visible and easy to implement.
APPLICATIONS Possible
application could involve utilizing each method’s strengths in
combination to overcome the inadequacies found with each method. A possible
co-method approach where STEP is initially used to develop the timeline with
actors/actions and events, identify gaps in knowledge, and provide consistency.
Fault Tree Analysis would then be incorporated to provide the systematic
framework to logically sequence the causal factors, identify any knowledge gaps
not uncovered by STEP, and allow for additional multi-investigator input to be
utilized. The resulting analysis obtained from STEP and Fault Tree Analysis
could then be visually displayed using the Fault Tree diagram to provide an
easy to see accident event sequence that would be understandable and
informative. Since both methods were evaluated as easy to use, a co-method
approach could be relatively easy to implement. In addition, this approach
could produce a valuable cross-check and verification approach for each method.
An
alternative approach to determining the most appropriate method would be to
conduct investigations using both methods individually under a variety of
entrapment circumstances to actively assess each method’s capabilities in
causal factor determination. This would further validate each method’s
applicability to wildland firefighting.
FUTURE RESEARCH It
may be desirable to research and develop a new integrated method that
incorporated the strengths of STEP and FTA into a third more comprehensive
method. Thus the weaknesses of each model could be accounted for (and the
inherent biases of the author and subject matter experts) and eliminated to
produce a method better suited to the unique, dynamic work environment of
wildland firefighting. Current research into organizational/managerial and
human factors involved in accident causation would need to be incorporated into
the new model. In order to identify those factors that lie at the root causes
of accidents, an updated model should integrate research that focused on high
reliability organizations (e.g. air traffic control or aircraft carrier
operations), decision making models, risk management concepts (e.g. risk
homeostasis), intentional standards violations, and human error mechanisms (See
Rasmussen 1997 for research into these converging fields). These fields of
research could combine various academic and safety professional disciplines
into a singular, encompassing causation model that would more accurately and
effectively reflect more deeply rooted failure mechanisms.
Another
alternative would be to investigate more thoroughly the possible application of
Accident Fault Trees (AFT diagrams) to wildland firefighter entrapments. Love
(in press) has extended the capabilities of traditional Fault Tree diagrams to
include temporal properties, accident severity considerations, and possible
interactions during the course of the accident. These enhancements account for
the most significant shortfalls inherent in traditional Fault Tree Analysis.
In
hindsight it may be more effective to survey additional evaluators in order to
accumulate more evidence as to the best method. It would be more insightful to
have investigators who are experts with each method investigate entrapment
fires and subsequently compare results.
An
additional criterion would also prove more informative in the evaluation. This
criterion could be the practical utility of the method to wildland firefighter
entrapments. This would allow the evaluators to provide input into the overall
applicability.
CHAPTER 7 - DISCUSSION Results
of this thesis have shown the applicability of specific accident investigation
methods to wildland firefighter entrapments. Subject matter experts have rated
the Fault Tree Analysis and STEP method as the most applicable to this specific
high risk work environment. STEP received the highest overall rating, the
largest number of high rated criteria, and had acceptable evaluator consensus
on all criteria except being systematic(face validity). Fault Tree Analysis was
rated nearly as high and with the exception of being realistic (ecological
validity), evaluators reached consensus. When “ease of use” was
eliminated from the evaluation process (to determine whether that criterion
affected the rating), results remained unchanged. Both Fault Tree Analysis and
STEP scored a “9” for ease of use and evaluators acceptably agreed
on the score. Fault Tree Analysis has limitations in representing sequential
and simultaneous events along a time line. It also was limited in only
portraying success/failure modes and not the varying degrees that are often the
norm in human interactions. But its ability to systematically and logically
solicit the cause of a particular event was its overall strength. The STEP
process was not deemed as sound in that respect. The STEP method was determined
to be a method that was valuable in organizing and collecting data at the onset
of the investigation. It was valuable in its ability to follow actors and
events along the causal chain and illuminate breaks in the sequence. It also
illustrated possible interactions among actors and graphically depicted the
accident process in an easy to follow and updateable format. The STEP method
would be a powerful tool in the data collection/developmental stage of an
investigation. When followed up by a Fault Tree Analysis, any subsequent
questions uncovered may be investigated. This second stage analysis could also
provide validation or invalidation of the STEP procedure. Upon completion of
the accident investigation, the visual display of the root causes, contributing
factors, and chain-of-events would be in the hierarchical tree format used by
Fault Tree Analysis. Any interested party could then see graphically the
sequence of events that have led to the accident. The appropriate and
responsible administrators could then have accurate focus points on which to
mitigate hazards and institute corrective measures. By the application of both
methods in concert, latent agency inadequacies, managerial omissions and
oversights, and human factors issues would more likely be identified than by
utilizing a single approach.
INVESTIGATING NEAR MISSES The
premise has been raised as to the critical need to investigate incidents that
have not led to disaster, the near-misses (Lucas 1991, Reason 1991, Vaughan
1996b). This approach would identify the sequence of events that could have
led to a disastrous result were it not for luck and/or extraordinary recovery
efforts. Identification of adaptive processes, modes of recovery, and
conditions at the boundary of near-miss versus harmful accident situations
could be invaluable in proactive measures to prevent future occurrences.
Rasmussen (1997) discussed this point when he said that individual workers
navigate freely within a work system shaped by objectives and constraints
(administrative, functional, safety related). He stated that a worker searches
freely within those boundaries “guided by process criteria such as work
loads, cost effectiveness, risk of failure, joy of exploration, etc...”
Managers supply the “cost gradient” in which a worker searches to
identify an “effort gradient”. Therefore, in their search, a
worker will systematically migrate toward the boundary of functionally
acceptable performance, and when crossing the boundary is irreversible, an
accident may result. In a dynamic work environment such as firefighting, many
degrees of freedom exist where firefighters must continually validate
boundaries and adapt to changes that may be frequent, rapid, and life
threatening. Rasumssen (1990) said that removal of human errors cannot and
should not be the goal of safety programs. He stated that “ the ability
to explore degrees of freedom should be supported and means of recovery from
the effects of errors should be found.” Through training (such as
simulators), firefighters could subsequently learn better coping skills at
critical boundaries and a more effective array of tools in which to
successfully identify, adapt, and successfully recover from potentially
hazardous situations. Agencies continue to add more rules to cover situations
where disasters occur in an environment where all the conditions can never be
known (Vaughan 1996b). Skills to avoid entrapments and assess risk at critical
boundaries could provide alternatives to the addition of more rules that make
completion of the job more difficult (Rasmussen 1997). At the
managerial/administrative level, near-miss investigations would aid in locating
those latent factors that may have lain dormant at various levels awaiting
triggering actions that may result in an accident. Defenses, barriers, and
safeguards within an organization can be circumvented when a particular
combination of events occurs. Proactive identification of possible
“loopholes” within the organization could aid in closing those gaps
in the defenses through which accident sequences may occur.
The
reporting, identification and subsequent investigation of near-miss incidents
is the primary focus of the National Interagency Fire Center’s (NIFC)
SAFETYNET ’99 pilot program. This program responds to the Tri-Data
report’s recommendation (1998) to develop a system for anonymously
reporting safety concerns. Another interagency response to Tri-Data’s
recommendations is the Center for Lessons Learned that is currently being
developed at NIFC. This Center is a focal point and clearinghouse for
information related to firefighter safety. Both the safety data reporting
system and the Center for Lessons Learned are vital components of the
study’s principle “collect reliable safety data and use it”.
Lucas
(1991) proposed “systemic safety management” whereby perceived
potential problems, as well as near misses and accidents, are actively
solicited from throughout the organization as an integral component of the
organizational culture. She cited three key elements vital to the success of a
systems approach. The employees must have anonymity and freedom from
prosecution in reporting near misses. They must have confidence in
management’s policy of forgiveness so they have no fear in losing their
jobs. And finally, there must be feedback to employees in the form of
implemented error control strategies so that they can see the results of their
input. Thus, it is critical that organizations objectively evaluate their
underlying safety culture if they wish to institute a near miss reporting
system and benefit from the results.
In
addition, by reducing the number of unsafe acts, agencies can reduce the number
of harmful accidents. For every accident reported, there are numerous unsafe
acts that were unreported (Reason 1991). This reduction of unsafe acts could be
accomplished by clearly defining employee (firefighter) tasks, adequately teach
them how to do each task, validly measure performance of each task, and
subsequently reward workers for high-quality completion of the assigned tasks
(Kenney 1993). This is clearly a line management responsibility and function.
These include, but are not limited to, safety, environmental, and risk
management activities. Acquisition and allocation of resources, proper
training, and stewardship programs are subsequently dependent on the decisions
of top level managers and administrators (Vaughan 1996b, Kenney 1993).
Another avenue of approach that can add more reliability and validity to
accident investigations is one in which the National Transportation Safety
Board (NTSB) has a prominent role. As Johnson (1999) points out, the NTSB
position is outside the Federal regulatory mechanisms that protect the agencies
and companies that are investigated. Thus the Board has the independence and
autonomy to analyze managerial and regulatory practices that may otherwise be
overlooked. Any agency that investigates itself may very well be suspect in its
conclusions (whether right or wrong), particularly when the agency itself may
be lacking in adequate policy, regulations/standards, and oversight. As in
previous wildland firefighter accident investigations, wider issues pertaining
to organizational and administrative practices have been generally obscured by
more prominent causal factors such as high workload, situational awareness,
distributed cognition, and mode confusion (Johnson 1999).
As
environmental conditions change adversely, fire ground complexities increase,
and agencies continue to be subject to changing political climates, a
comprehensive, systematic process to reveal possible failure points is vital.
Investigations of entrapment near-misses would provide the insight as to where
the system needs reassessment and updated controls. It would likewise provide
insight into the adaptive/coping skills that may have kept a near-miss from
becoming an accident. There is a need to learn from the lessons that occur
without the resulting disaster so that firefighter safety is not so much
reactive to major disasters but continually adapting to the dynamic nature of
the overall work environment. Reason (1997) calls for an
informed
culture,
one
comprised of a
reporting
culture
(accident/incident reporting), a
just
culture
(where
rewards and punishments are viewed as just), a
flexible
culture
(where
an organization shifts from a bureaucratic conventional operating mode to
professional, task expert control during emergency situations), and finally a
learning
culture
(where the organization has the willingness and competence to identify correct
conclusions and implement needed reforms). Adequate investigations of near-miss
incidents and the development of avoidance/coping skills at vital trigger
points (boundaries) could greatly decrease the harmful outcomes to wildland
firefighters. By identification of boundaries where successful recovery actions
cannot be implemented and the necessary coping/recovery skills near these
boundaries, harmful accidents could be significantly reduced. The STEP method
used as a primary investigation tool and followed up by Fault Tree Analysis
could greatly aid in the ongoing search for a safer firefighting work
environment. Both methods offer the means to incorporate more abstract
causative factors and more specific root causes as new “composite”
error identification models become functional.
This
thesis showed, not only the two most applicable accident investigation methods
for wildland firefighter entrapments, but the critical need for requiring a
method to be utilized. In order to determine the proximate causes of
entrapments, the trends, and monitoring of safety measures, it seems equally
critical to develop the data base of human factors causes and near miss
incidents. To develop and institute a safety culture through a learning
culture, a total commitment by management must be a leading priority. The
current trend in firefighter entrapments can only be reduced by top level
administrative support and low level firefighter dedication to safety. A change
in the way we do business, a change in the safety culture, can be and must be
achieved by the combined efforts of all levels of the wildland firefighter
community.
|