Active Connection Management for Internet Services
Mike Chen, Stanford CS548 seminar, 5/22/02
- Problem: failover strategies and conn. mgt. in L4-L7 switches are
hardwired. A mismatch for the dynamism of large services (service
growth, machines get replaced, etc.) Soln: extend API of LB
switches to allow apps/infrastrcuture to dynamically control mapping of
client conns to phsyical machines . Hypothesis: improves
availability, load conditioning, admin/mgt.
- Goals include: dynamic rsc alloc to handle peaks; automate switch
config. to eliminate operator error; support graceful starting/stopping
services to support rolling reboot and online upgrades; help detect/recover
from svr failures.
- Cisco Dynamic Feedback protocol: can set weighting vector for LB,
keepalive interval, server added/removed.
- IBM Director: relies on failover mech. of underlying sys. Eg to
reboot a node behind an LB switch, Director has to shut off the node
and rely on the switch to detect this first.
Existing L4 connection mgt primitves:
- Add: bind physical to virtual IP/port (after verifying health of resource)
- Remove: stop forwarding new conns (force remove: breaks existing conns
- Not supported: Drop some new conns (according to some criterion)
for admission ctrl
Implementation of Mike's soln:
- Each app server generates events ("I want to bind to a virtual
address"), a separate Conn Mgr takes these and uses Java wrapper for
switch to implemenet these requests.
- CM is centralized with standby peers, maintains soft state (heartbeats
from servers), sends config deltas to switches
- Switch fails: CM adds known bindings to switch after recovery (switch
forgets its bindings on hard reboot!)
- Note - it takes as long for the standby to take over as it does to
restart the primary and repopulate its soft state! Mike admitted
standby is not as useful as he'd expected. A way to achieve seamless
transition would be to have 2 primaries and have the switch just ignore
messages from the standby, until it starts seeing messages from the standby
with no corresponding messages from the primary...but then you have to deal
w/the possibility that standby and primary will give different instructions,
and besides the switches are designed to accept messages only over a single
dedicated TCP connection for now.
- CM failure: Since all config of switch is via CM, if CM fails and then
another server fails before CM back up, switch will contain stale binding
for that failed server. So after CM comes back up, collects known-good
bindings from heartbeats, then removes unknown bindings on switches.
("The heartbeats are the only truth")
- Supports resource replacement through graceful removal: quiesce server,
then reboot it. ("100% availability" - except it's really
not, since there is temproary thruput reduction as standby warms up.
- App-level exceptions can actively remove failed resources (in addition to
middleware's existing health checks). This is one more way to detect
errors. (This implies the exception leaves the app in good enough
state that it can request to be removed; if it is in that good shape,
perhaps it should proactively restart itself, since that's what the switch
is going to do anyway?)
- Note rolling reboot vs. "big flip": Rolling Reboot only works if
new upgrade is software-compatible w/old upgrade, since the old and new
versions must coexist in same system.
- Neat hack: since current switches can't do admission control, add drop
servers, which are like /dev/null - it immediately RST's the TCP
connection, or holds the client connection open, but is a "bottomless
sink". Can use for preferential tretament, ie E-trade prioritizes
paying customers over free servers.'
- Interesting tidbit - COTS switches have a group of procs for data routing,
and a separate proc for SNMP/config. Add and remove take 20-25ms each
on Foundry ServerIron.