Jim Gray, Prashant Shenoy
(Summary by George Candea)
Rules of Thumb in Data Engineering,
Proc. ICDE, Feb. 2000
This paper is a reexamination of the rules of thumb that drive the
design of data storage systems.
- Many rules of thumb are a consequence of Moore's Law, so I'll
reiterate it here: "circuit densities increase 4x every 3 years"
(a.k.a. "things get 4x better every 3 years").
- In-memory data growth leads to the need of one extra bit of
addressing every 18 months (64-bit addressing should be sufficient for
another 2-3 decades).
- Storage capacity improves 100x / decade, while storage device
throughput increases 10x / decade. At the same time the ratio between
disk capacity and disk accesses/second is increasing more than 10x /
decade. Consequently, disk accesses become more precious and disk data
becomes colder with time, at a rate of 10x / decade.
- Disk page sizes increase 5x / decade. In ten years, the typical
small transfer unit will be around 64 Kb, while large transfer units
will be 1 MB or more.
- The cost per random tape access is about 100K times higher than
for disk. In five years this will probably be 1M times higher
(calculation includes cost of tape drive). Tape capacities are
expected to improve faster than tape speed, and access time is
expected to stay about the same.
- The nearline tape : online disk : RAM storage price ratio
today is 1:3:300. Historically it has been 1:10:1000.
- In ten years 1 MB of DRAM will cost what 1 MB of disk costs today.
- Disks are replacing tapes as backup devices. You may be able to
backup 1 TB to tape, but it takes very long to restore it. As
sysadmins see petabyte stores looming on the horizon, they start
replacing tape backups with the maintenance of multiple disk versions
online (perhaps geographically distributed) so that they never have to
restore from tape.
- Storage prices have dropped to the point where storage management
costs exceed storage costs, the same way PC management costs exceed
hardware costs. A sysadmin can generally administer $1 milion worth of
storage. That used to be 1GB, is 1TB today, and will be 1PB in 10
- Amdahl's revised balanced system law: a system needs 8 MIPS per
MB/sec of I/O bandwidth (but the instruction rate and I/O rate need to
be measured on the relevant workload).
- The MB of RAM / MIPS of CPU ratio is rising (is about 1-4
- Random I/O happens once every 50K instructions. For sequential I/O
this number is much higher.
- Gilder's law: deployed network bandwidth triples every year.
- Network link bandwidth improves 4x every 3 years.
- The CPU cost of a SAN (System Area Network) message is 3K clocks +
1 clock/byte. Historically, a network message cost 10K instructions +
10 instructions/byte, while a disk I/O cost 5K instructions + 0.1
- Currently, it costs more than $1 to send 100MB via a WAN, while
local disk and LAN access are 10K times less expensive. This price gap
is likely to decline to 10:1 or even 3:1 over the next decade.
- The memory subsystem cannot keep the CPU's pipelines full, even
with 3-level caches, unless programs have good data locality. As CPU
speeds continue to outpace memory speeds, it will become increasingly
important for software to have small instruction cache footprints,
predictable branching behavior, and good data locality. There is a
trend to build huge multiprocessors using shared memory - these
systems are prone to instruction stretch, in which bus and cache
interference from other CPUs cause each CPU to slow down.
- 5-minute random rule: cache all randomly accessed disk pages that
are touched at least once every 5 minutes.
- 1-minute sequential rule: cache all sequentially accessed disk
pages that are touched at least once every minute.
- Spend 1 byte of RAM to save 1 MIPS of CPU.
- Cache web pages if there is any chance they will be re-referenced
within their lifetime (calcaulation made based on people cost
savings). A major assumption is that server performance will continue
to be poor (3 seconds on average). With declining costs, however, web
site owners may buy more bandwidth and hardware, reducing response
times to under 1 second. In that case, the need for caching would be
purely to save network bandwidth and download time.