05/08/01 Armando's Notes from HDCC
Jim Gray, Internet reliability.
- C.1995: SW masks mosts HW faults, HW was great. "Hidden"
SW outages (new s/w, online upgrades, etc) motivated spending on improving
SW; Heisenbugs seemed to define 30-yr MTTF ceiling. "A good goal
for availability today would be 100 year MTTF"
- Tandem technology today: <30sec restart/failover
- Today: RAID standard, "cluster in a box" (commodity failover),
remote replication standard. Hope was for five nines
- Today: cellpones 90%, websites 98%, day-long outages (problems harder to
diagnose and fix). (1990: normal phone 5 nines, ATMs 4 nines,
hour-long outages) Also, hackers are now main source of Internet
- Why complexity compared to old xact servers? Many heterogeneous layers,
each of which under constant churn: ISP,firewall, Web,DMZ, app server, DB,
..., plus administrator skill level very reduced
- Typical MS data center: 500 4 or 8 headed servers, crossbars, etc.
Hotmail: 7K servers, 100 backend stores totaling 120TB spread across 3 data
centers, 1B msgs/day, 150M mailboxes of which 100M active, 400K new
mailboxes/day, software upgrades every 3 mo. (eg recently switched from
FreeBSD to Windows...[I wonder what the experience was like])
- New threats from hackers: all systems open (can be attacked from
anywhere); complexity makes them hard to protect; concentration of wealth
makes them targets.
- Check out Iinternet health report and measurements from SLAC
- Most large sites build their own instrumentation - vast repeated work!
(despite commercial attempts to systematize it)... is part of the problem
the lack of availability benchmarking standards?
- Netcraft: asks system how long it's been since site was rebooted [does
that mean per node average? or...?] [Check Netcraft website!]
- MS.com: each individual backend server is four 9's! BUt overall
system is LESS...system too complex, software/architecture changing
quarterly, less skilled operators (took 18 hrs to diagnose an induced router
misconfigurations), etc. Ebay is most honest, they publish their
operations logs. [Geo, check it out!] Recent results: 99% of scheduled
uptime, about 2 hrs scheduled downtime/week.
- We have great node-level availability, terrible system-level availability.
Interesting observation from Ed Adams: most bugs in mature systems are 50K-MTTF
bugs. BUt there are so many of them that you'll hit one every couple
- In Tandem days, people wanted faster and cheaper than IBM but just as
reliable. Today, people are willing to tolerate stale data,
insecurity, etc. in return for features, basic availability, etc.
- Recommendation: live in fear! we have 10K-node clusters, headed for
- Recommendation: more systematic measurements, automated management
- Recommendation: change security rules! No anonymous access, unified
auth/authorization model, only one kind of interface. Otherwise we
can't win. Firewalls at packet level can't stop current
attacks. (end-to-end argument) Detecting attacks and responding to
them must be app-level.
- Recommendation: single-function "appliance" servers with low
churn. [But how to accommodate new features?]
- "Hard to publish/hard to get tenure" in holistic dependability;
journals want proofs, etc.
Questions from audience:
- We should have a computational model analogous to transactions, for
Observation from me:
- We need a computation model that (a) admits of long system MTTF
times despite shorter MTTFs of components, (b) captures "Internet
semantics" (soft state, stale data, etc), (c) allows avaiabilty
benchmarking that is well-defined with respect to a particular app (by
virtue of mapping the app to the model).
Bill Scherlis: projects for new NASA/CMU/etc testbed, eg open source
dependability tools, etc
David Garlan: in charge of new SWE curriculum @ CMU. Tell him to teach
restartability-centric, fault-isolated, etc design in ugrad SWE classes.
Lynn Wheeler, CTO, First Data (the actual operators of Visa and MC's
- 90% of all Visa/MC xacts done at 6 processing centers (not all operated by
- Started out in batch/xact world, recently providing support for
e-commerce. TCP/IP has no infrastructure on service side for
troubledeterminination; service spec is not defined anywhere (only the
protocol is). FDC built a "trouble desk grid" with about 40
failrue modes and 5 states,for use in troubleshooting TCP-related trouble
calls; in each case trouble desk must demonsrtate recoverability, isolation,
and/or problem determination.
- Similarly: ISO standards for a credit card xact assume the connection is
ckt-based with end-to-end diagnostics built in. This doesn't exist with
TCP-based infrastructure, so couldn't just "reimplement the spec"
over TCP. Inventing these for payment gateway resulted in about 5x
code increase, plus ancillary procedures put in place.
- Rule of thumb: Given a good trained programmer who can write apps to spec,
adding dependability increases code size by a factor of 4 to 5.
Can we actually trade cycles (nearly free) for dependability? Simple
example: in one case with FDC, string buffer overflow cost billions of dollars
in downtime, diagnosis, etc. Original problem: programmers thought they
could save a few machine instructions by making assumptions about string length,
proper termination of strings, etc. rather than coding defensively. To
what extent could we integrate code like this into libraries, runtime,
etc.? Would this be a good contribution to the open-source project testbed
(a "dependable" or "safer" libc that is more expensive)?
Steve Gonzalez, Chief, Ops Rsrch and Strategic Devel, NASA JSC
- Before: breaks between missions (continuous operation only required during
a mission), redundant isolated flight control rooms, each console hardwired
for one function (cannot fail over to another console), full HW isolation
(if flying on one system, others can be offline, under test, etc)
- Now: everything distributed, incl. application development (each mission
team may do its own control, visualization, etc), hybrid HW/SW isolation
[was this a good idea??]
- Custom vs COTS: expertise is offsite, yet they need <1min failover
during missions...they are often pushing the envelope of performance for
- With ISS, env is constantly changing/evolving, making 24x7 support very
hard. Had to come up with a scheme to allow live upgrades. Current rec
is to go back to full HW isolation!
Bruce Maggs, Akamai/CMU
- Failures seen at Akamai: congestion at public peering points,
misconfigured routers/switches, inaccessible networks. Congestion at
public peering points is bad because for contractual reasons the operators
can't change route tables to ease congestion. Esp. in less developed
NW infrastructure area, sometimes subnets become unreachable for weeks at a
time. Makes software upgrades hard (policy is to make everything
- Streaming: streams sent from source to as many as 4 different backbones
via "reflectors". The reflectors redundantly distribute
streams to the edges, i.e. a given edge cluster may have redundant feeds
from different backbones. Duplicate filtering, etc is done at edge
cluster, where stream is then redistributed. Note, no buffering!
A little bit at the edge clusters, but it's OK to lose the occasional
- HW/server failures: they operate 10K machines on 500+ ISP's. All
machines either Linux or win2k controlled over ssh. Interesting
failures: SIMMS pop out of sockets (and machine doesn't crash!), network
cards not in slot, switches configured to drop broadcasts.
- IP related software failure: machine 1 dies; machine 2 does IP-addr
stealing, but then gets infected with whatever killed machine 1; then local
DNS server started shutting off other machines there. Problem: someone
had turned on a router with weird MTU, and Linux had a bug where it couldn't
compensate for weird MTU size.
- Bugs vs features: IP-stealing to react to server failures is usually a
feature, but turned out to cause cascading failure in case above.
[Similar problems in TACC, or any system where a systematic bug can take out
one node at a time]
- Reporting tools must be as reliable as system itself - if tools say it's
not working, that's just as bad as watching the failure in action.
- Content providers get both weekly traffic reports and near-real-time data
on where the hits are coming from.
- Attacks: internal bugs have been much larger source of problems.
Weak spot in system is top-level nameservers (as is true for the DNS root
- Testing: one aspect of testing allows them to deploy stuff on the live
system in a "passive" or shadow mode: it does everything except
send bytes. (Like make -n)
- Design: no single point of failure; rule of thumb is require 4
simultaneous failures on different networks to cause a sytemwide failure,
and 2 simultaneous on different networks to cause a single user-perceived
failure. This doesn't count "systematic vulnerabilities"
like the case above; they've considered porting to multiple OS's, etc, but
there's a lot of overhead involved. Currently they rely on Linux for
everything except serving Windows Media streams.
- No humans in operations loop except for SW upgrades.
- Failover at multiple scales (steal IP, move to another local site, shut
down DNS subtrees, etc). Sophisticated optimization algorithms and
comprehensive network maps, doing data cleaning on probes to figure out
which ones indicate real problems, etc.
- Multiple and disjoint reporting systems (ie the monitoring system is not
used to actually load balance, and the measurement network used for billing
is also separate).
How to define a computational model? One possibility -start relaxing
Acid constraints one by one, since internet systems are query-like.
Another - instead of capturing something absolute, like conisistency, capture
something like eventual consistency (consistency x latency), or basic
availability (consistency x availability), or something like Aaron's avail
benchmarking or Amin's conits, or "OK to say no"-ness.
Jim Gray: snapshotting filesystems - rotating RAID-0 on 2 out of 3 disks
Can machine virtualization be used to achieve the fault-isolation for
restartability through modularity (FIRM)?
Plans for near term
- RR app + runtime system -> iroom
- Talk to Mendel about VM stuff: fault isolation? fault injection (fake
- how much does wolfpack do now?
- Which mechanisms will we use in platform
- what end-to-end and module-specific probes to use? how to express
- Jamie: application to ground station s/w
- find out what kind of fault injection we need
- Restructuring apps to make components restartable by separating out the
state (generalized transformations) - from Fox/Mazer ms, and then isolating
the pieces and doing fault injection (need to pick an app)
- Error handling models for Paths: carry exceptions along? What has been
done in dataflow? pipelining? are "precise exceptions"
needed? Talk to Jim Larus also.
- Computation model to emerge from readings: approximate answers, eventual
- Setup a day for: iRoom demos, talk w/Mendel, talk w/Geo, talk
w/Emre/Laurence, + lunch and social.
- Target next fall for "The Case For ROC" papers
- low level isolation mechanisms
- "synthesis" paper (common ideas across different communities)
"Resilient data structures" - internally redundant so they can be
fixed/tolerate losses - would this be useful?
A thought for inexact answers: a utility function that measures answer
"quality" vs. resource consumption (memory, time ...). It may
have a cliff, or the user may have a threshold, for answer quality vs. resource.