The Investigation Process Research Resource Site
A Pro Bono site with hundreds of resources for Investigation Investigators
Home Page Site Guidance FAQs Old News Site inputs Forums
. . . .. . . . . . . . last updated 8/8/09

INVESTIGATING
INVESTIGATIONS

to advance the
State-of-the-Art of
investigations, through
investigation process
research.



Research Resources:

Search site for::


Launched Aug 26 1996.

 

 

Comments on RAND -ICJ Report about NTSB

Pro bono comments from a former Intermodal Specialist and mid-level manager at NTSB about the study <"safety in the skies, personnel and parties in ntsb aviation accident investigations: master volume" by the rand institute for civil justice. 12/99.

View another commentary on the rerport posted at www.iprr.org/comms/co_randcom.html. See also a paper "Forensic Software Engineering and the Need for New Approaches to Accident Investigation" for additional commentary.

(Rebuttals/opinions/invited. Go to AIPRE Researcher's Forum. I am also willing to post additional reviews here if anyone cares to submit them to me.

Foreword
The views expressed are solely those of the author. These observations are based on the summary report of the RAND ICJ study of the National Transportation Safety Board's investigation-related practices. The comments might change after the full technical report becomes available.[1] The views are based on extensive research of investigation processes, work products, tasks and uses.


Overview.
The Study is good news and bad news. The good news is that serious problems at the current NTSB' are now in the public domain.The bad news is that the study misdefines some of the problems, that it overlooks some key problems, that its proposed solutions are based on some false premises and flawed logic, and that some solutions are often misdirected. The aims of the RAND recommendations are at the end of this commentary.

The good news.
The Study asserts that "The NTSB must embrace new methodologies, new management approaches, and a new awareness of its working environment if future demands are to be met" and supports that contention with generous evidence. It calls for management and training reforms, and the need to modernize investigation practices and procedures.

These points have been widely recognized privately. To the present Chariman's and Board Members' credit for undertaking the project, they are now publicly acknowledged, with sufficient supporting detail to command attention.

Bringing these issues into the public arena makes the report a valuable contribution. The NTSB does indeed have serious management, staffing and methodological problems.

The bad news
As I read the report, I find that to make the points persuasively I would have to perform work commensurate with the study. Therefore, I will simply highlight the points I think need to be made to bring about long term changes needed in this agency. Upon request, I would be happy to provide further details.

While the report brings the problems to light, its recommendations well not resolve the management and methodological problems described.

1. Statutory mandates
.
A major study deficiency is its failure to address the NTSB's statutory mission problems. The study does not challenge the statutory mission - to investigate all aircraft accidents and determine their cause - which only the Congress can fix. Investigation of all aircraft accidents is, on its face, an impossible mission for any agency. Congress needs to fix this statutory defect to reflect reality - and do investigations that are befitting a national agency - by making the selection of accidents to investigate the Board's decision, and clarify what the Board is expected to accomplish.

2. The NTSB objectives


The second mission problem only lightly addressed is the framework for Board work - to establish the facts, circumstances and cause or probable cause of accidents.The statute thus casts the work of the agency into a legal framework, (facts, circumstances and cause" or probable cause" invoke blame, and can not be validated technically or scientifically. The inevitable result is that they incite controversy which requires political or judicial rather than technical resolution in today's environment. The five presidential Board appointees serve as a de facto jury to make judgments about cause/blame, further reflecting a legalistic approach to the mission. Controversy undermines confidence, and so by design the agency is placed in a difficult position. This could be fixed by making it clear that the best service the Board can provide is to report what happened and why it happened. Rather than broaden cause as the study suggests, eliminate from the statute and substitute words like "determine what happened and why it happened." While the study states the determination of cause is the area that has been most vigorously challenged, it does not get at the roots of this problem.
3. Examine techniques and methods
The third mission problem is what the study did not examine vigorously - the Board's mission to examine techniques and methods of investigation and periodically publish recommended procedures for accident investigations. When is the last time the Board spent any money on this neglected mandate. This neglect is one of the reasons the Board's own methods had to be rebuked by the Study team. Further, if the Board were to establish its practices as standards for investigation without a research basis, it would perpetuate failing practices. Unfortunately this research neglect has far reaching consequences as other agencies are created using the NTSB model. Rather than providing the best techniques and methods, we now learn that agencies modeled after NTSB practices are saddled with the problems cited in the Rand Report.

The study did not touch the requirement for the Board to report its appraisal of the accident investigation and prevention activities of other government agencies mandated in the statute. Given the Study's comments on the NTSB's processes, it is difficult to fathom how such a requirement (1117(3)) could be satisfied credibly. This reinforces the need for the NTSB to focus its research on investigation processes rather than research unknowns discovered during investigations that should be the responsibilities of the aviation community to know. By subsidizing industry with such research, the NTSB is neglecting its own duties.

2. Resource increases
.
The study makes some unjustified assumptions and leaps of logic to propose that the agency be provided more funds. It neglects the continually improving safety performance of air carriers over time, the ability of the NTSB to establish the scope of its investigations, and the NTSB's case selection options in assessing the future workload demands on the NTSB. . The question posted in the study about what the Board will investigate if efforts like the National Civil Aviation Review Commission are successful is a good one, and related to this issue.

The investigation scoping option is particularly important. The TWA 800 case, illustrates this point. Where is it written that when something is not understood, the NTSB has to research the problem, expending its resources to do so? It seems to me that when the NTSB discovers a problem, it can satisfy its duty to the public to define it well, and propose that the proper party fix it, thus reserving NTSB's resources for other investigations. Even more importantly, the NTSB would not become an interested party in defining the fixes or the ultimate success or failure of the fixes. The utilization of parties to acquire resources is acknowledged in the Study, but the effects of the investigation methodology the Board has selected (parties, eliminate all other possible causes) on resource demands of the agency is not treated adequately, in my view. Case selection in private civil aircraft investigations also needs further treatment, to balance the value of lessons learned in each investigation against the cost. Frankly, until these kinds of management options are adequately analyzed, the recommendation to increase agency funding seems to be influenced more by politics than reason

3. Management issues
.
The study correctly points out administrative problems, but there are other management issues that affect NTSB performance. In addition to the need for leadership in defining the agency's mission, the impact of management's influence on defining what constitutes the agency's success, how to measure that success, and then monitoring activities to determine if success is being achieved are given skimpy treatment. A good healthy cost/benefit analysis of activities would be beneficial to all concerned. I am surprised that Congress has not asked for such an analysis, in view of recent trends in that body to identify performance.

Related to this is the performance efficiency issue. Output questions are related to people and training, but not to the methodology which management has selected for the agency - or perhaps tolerated would be more appropriate. To its credit, the Study brings out communications issues, but is vague about how the methodology selection influences this problem. It is also ambiguous about how the internal report generating process and its management affects productivity and timeliness of agency outputs. At one time, I had 22 people in the agency that could veto my work products, and one helper to generate it. No wonder the agency has almost one administrative employee for every two production employees.

Another aspect of management deficiencies is the process by which investigation teams are managed, which is addressed by the Study, but it doesn't connect all the dots. The selection of the methdology which drives tasks performed by teams contributes to the management difficulties, but this issue is not identified in the report. Thus this criterion is less likely to influence the selection of "better" methodologies for investigation as remedial efforts proceed.

4. Methodology selection
.
The report lays open methodology issues, but there is one consideration it doe not address. Should the agency focus on investigation technology, or try to cover both investigation and aviation system technology?. One of the conceptual reasons for the party system is that it enables NTSB to access the latest knowledge about systems and their operation, to answer the investigator's questions. The investigator must know what questions to frame, and how to test the validity and relevance of the answers, which are investigation knowledge and skill demands. It seems to me the Study failed to distinguish between these knowledge needs adequately in its discussion of methodology and training needs. Train investigators to know and do what? For a small agency to know everything that is going on worldwide is a forbidding demand. Better they know how to define what information they need, how to get it, and how to tell if it is adequate for their purposes in any case with any kind of systems.

Gratuitous afterthoughts

Given the complexity of the investigation demands involving the NTSB, I wonder if the most effective use of public resources for investigation purposes might not be a scheme that changes the agency's mission substantively. How about designating it as the agency that prescribes the specifications for investigation of transportation accidents, and the quality criteria for the work products they produce. Then give it a quality assurance role over others' investigations, like its mission in the hazardous materials field?

This seems like a good time to reexamine prior assumptions and paradigms about the NTSB and similar Federal agencies' roles in the safety assurance processes.

I hope before the NTSB gets more resources, these needs and options will be thoroughly examined, and other potential operational models explored.. Properly aimed and managed, the agency can serve a valuable public purpose,


L. Benner
January 29, 2000
rev 3/29/00

-o-

							
							
							
							
From the Study:

The recommendations aim to accomplish the following eight objectives:

• strengthen the party process

• create a more expansive statement of causation

• modernize investigative procedures

• streamline Internal operating procedures

• better manage resources

• maintain a strategic view of staffing

• streamline training practices

• improve facilities for engineering and training.


	

 


[1] The final written Rand Report was originally scheduled to be delivered on or about February 1, 1999. NTSB paid the last of the $400,000 charge for the report under the MOU in Februrary 1999. As of the most recent update of this page the Technical Report has not been released.

LB.