Back to index
Safeware: System Safety and Computers
Nancy G. Leveson, University of Washington
Nature of Risk: (likelihood) x (severity)
- Shift of responsibility from individual to groups/experts
- Shift from craftsman to assy line (and accompanying increase in
complexity of production systems) has engendered a shift in
safety responsibility from employee to employer, and by
extension, from members of the general public to "experts".
- "Whereas in the past, component failure
was cited as the major factor in accidents, today more accidents
result from dangerous design characteristics and interactions
among components". (Paraphrased from Willie Hammer, Product
safety management and engineering. Englewood Cliffs, NJ:
- Can't measure system before it's built/designed, so risk
assessment tries to quantify risk up front. Acceptable
risk is threshold level, but often is determined
irrespective to those receiving benefits of risk
protection. Ex: Pinto gas tanks were known to be
volatile, but the occasional explosion of a car was
considered acceptable risk rather than pay cost of reengineering.
- Role of computers and software
- Technology eliminates some hazards, but creates some new ones
that are more pervasive and harder to find and eliminate than the
ones that were reduced.
- Computers make this worse by enabling interactive,
tightly-coupled, error prone designs:
"by increasing the complexity of the processes that can be
controlled, [computer programs] have increased the scope for the
introduction of conventional errors" (Trevor Kletz: Wise after
the event, in Control and Instrumentation 20(10):57-59,
- Operators often isolated from phys system (for safety
reasons, or because the scale of the system is
incomprehensible to individual users, as in the case of an
oil rig) and must rely
exclusively on indirect/physically-unverifiable info
or are "out of the loop" till
too late. Ex: 1985 China Airlines jet losing power in
right engine, but autopilot automatically corrected. Crew
didn't find out till it was too late (autopilot could no
- We learn more from bridges that collapse than from those
that stand. Increased pace of change leads to decreased
learn from failure. Empirical design rules derived by
trial and error are being replaced by reliance on a
priori hazard identification and
- Software emphasizes only the steps to be achieved, without
worrying about how to achieve them (unlike mechanical
systems). Proven system-safety eng techniques haven't
been applied to software, nor can they be because of the
essence of the medium.
- SW "flexibility" encourages redefinition of tasks to shift
responsibility from HW to SW late in the process.
Interfaces in HW systems tend to be simpler than SW ones
because physical constraints make the costs of complex HW
interfaces immediate and obvious.
- SW is logically brittle (not physically brittle like HW),
so harder to see how easily it can be broken, esp. since
partial success is trivial to achieve (compared to
HW) and rarely indicative of complete success.
- Software myths exploded:
- Cost of computers is not lower than cost of
electromechanical/analog devices, when maintenance is
considered. Space shuttle software is about 800Kbytes
but costs $100M/yr to maintain.
- Software is not easy to change if safety is to be
preserved. Plus, software errors tend to lurk dormant for
a long time -- there is no "infant mortality" for
software, thus the belief that safety programs can be
"phased out" after a few years is fundamentally
misguided. (There is no way to measure the number of
accidents a good program has avoided.)
- Increasing SW reliability will not increase safety:
often the SW is working as designed, and the bug comes
from unexpected component interactions. "Safety is a
system property, not a software property." SW is considered
safety-critical only if a SW failure alone can lead to an
accident, but in real accidents this is rarely
- Reusing software does not increase safety: may
actually decrease it, because of complacency it engenders.
Also, most SW is specially constructed
for its application and
reusing it elsewher is probably dangerous (Therac-6 SW
ported to Therac-25).
- Hierarchical view of accident causation: mechanisms ("the car hit
the tree"), conditions ("it was raining and the car didn't have
antilock brakes"), and root causes.
- Root causes: inherent weaknesses that can also affect
future accidents. Important because accidents rarely
repeat themselves in the same manifestation, so addressing
root causes is more important than "patching". (eg Mars
Pathfinder thread-priority inversion) "Often one finds
triple locks on those doors from which horses have been
stolen, while other doors are wide open."
- Most root causes
involve 2 or more low-probability events happening in the
worst possible combination.
- Overreliance on redundancy for low-probability events that
turn out not to be independent. Ex: in Challenger
accident, primary O-ring failure caused conditions that
led to failure of secondary (backup) O-rings.
- Overreliance on redundancy for poorly-design device: worse
than no device at all, since it engenders complacency.
- Technologies are developed because they have clear and
quantifiable benefits, but often the risks are ambiguous
and elusive, so up-front hazard management is often
replaced by "downstream protection" (adding safety to a
completed design). Note the similar situation in
- Organizational factors
- Large engineered systems usually reflect the structure,
management, culture, etc of the organization that created
- Other factors: diffusion of responsibility and authority
in large organizations; lack of independence or lowly
status of safety personnel, or requiring them to report to
the organization whose safety is being checked.
- Need a
reference channel that communicates safety
goals/policies downstream, and an independent measuring
channel that communicates actual state of affairs
Intro. System Safety/Definitions and Models
Elements of a Safeware Program
Back to index