Disaster Recovery

Picture a database server that dies at 10:00. Two questions decide how badly that hurts: how much data did we just lose, and how long until customers can use the service again? Disaster recovery (DR) is the discipline of restoring service after a major disruption, and almost every DR decision is driven by the answers to those two questions, expressed as two targets you choose in advance.

RPO: how much data can we lose

The recovery point objective (RPO) is the maximum acceptable amount of data loss, measured as a span of time. The authoritative definition from NIST SP 800-34 Rev. 1^[1] is "the point in time to which data must be recovered after an outage." If your last good backup is from 09:00 and the server died at 10:00, you lost an hour of data. An RPO of one hour means that loss is acceptable; an RPO of five minutes means it is not.

The consequence is the rule worth memorizing: RPO sets backup frequency. Your backup or replication interval can never be longer than your RPO, because anything written between backups is what you stand to lose. A one-hour RPO requires backups at least hourly; a near-zero RPO requires continuous replication, not nightly tape.

RTO: how long can we be down

The recovery time objective (RTO) is the maximum acceptable downtime. NIST SP 800-34 Rev. 1 defines it as "the overall length of time an information system's components can be in the recovery phase before negatively impacting the organization's mission." It is the clock that starts the instant service drops and stops when service is restored. An RTO of four hours means the business can tolerate being down for up to four hours; beyond that, the damage is unacceptable.

The consequence here: RTO sets the recovery method. A long RTO can be met by restoring from backup tape. A short RTO cannot wait for a restore and forces faster machinery: a standby site, replicated systems, or automated failover. The shorter the RTO, the more you must pre-stage and the more it costs.

Why both, and the one trap

RPO and RTO are independent and you set them separately, because they protect different things: RPO protects against data loss, RTO protects against downtime. A bank might tolerate hours of downtime (a generous RTO) but almost no transaction loss (a near-zero RPO); a marketing site might tolerate losing a day of edits (a loose RPO) but need to be back up in minutes (a tight RTO). The exam trap is to swap them. Hold the one-line split: RPO is how much data can we lose; RTO is how long can we be down. Both are derived during a business impact analysis (BIA), the exercise that identifies critical processes and the cost of their disruption.

Where RPO and RTO are targets you decide, the next two acronyms are statistics you measure from real equipment history. That single distinction is the cleanest way to keep all four straight: RPO and RTO are objectives you set before any failure; MTTR and MTBF are properties calculated after observing failures.

MTTR: how fast can we fix it

Mean time to repair (MTTR) is the average time it takes to restore a failed component once it breaks. Add up the repair times across many failures and divide by the number of failures. MTTR measures recoverability, so it drives operational decisions: how many spare parts to stock, how many on-call staff to keep, whether a vendor's four-hour support contract is fast enough. A lower MTTR is better, because it means each failure is resolved faster, which in turn helps you meet a tight RTO.

MTBF: how reliable is it

Mean time between failures (MTBF) is the average operating time between failures of a repairable system. A drive rated for an MTBF of 1,200,000 hours is expected, statistically across a population, to run that long on average between faults. MTBF measures reliability, so it drives design decisions: which component to buy, and when to schedule redundancy or proactive replacement before the part is likely to fail. A higher MTBF is better, because failures are rarer.

Keeping the pair straight

The one-line tell separates them by the question each answers: MTBF answers how reliable is it (time between failures); MTTR answers how fast can we fix it (time to repair). They work together. MTBF tells you how often you will be doing repairs; MTTR tells you how long each repair takes. A system with a high MTBF and a low MTTR is both reliable and quick to recover, which is exactly the combination that lets you commit to an aggressive RTO. The exam trap is to treat MTTR or MTBF as a target you simply declare in a plan. They are measured from failure data, not asserted; only RPO and RTO are objectives you choose.

When an RTO is too short to rebuild from scratch after a disaster, you pre-stage an alternate facility and decide how to run it relative to the primary. Two independent choices follow: which site type to keep ready, and which redundancy mode to run.

Site types: a cost-versus-speed ladder

The three recovery-site types are a deliberate trade of ongoing cost against recovery speed; higher cost buys a lower RTO. Learn them as one ladder, using the NIST SP 800-34 Rev. 1 definitions.

• A cold site is, per NIST^[2], "a backup facility with necessary electrical and physical components but without computer equipment in place." You get space, power, and cooling, nothing more. It is the cheapest to keep on standby and the slowest to activate, because after the disaster you must ship in, install, and configure all the hardware and then load data. Recovery takes days.
• A warm site is, per NIST^[3], "an environmentally conditioned work space that is partially equipped with information systems and telecommunications equipment." The gear is largely there; you load current data and finish setup before cutting over. It is the middle of the ladder on both cost and speed, with recovery measured in hours.
• A hot site is, per NIST^[4], "a fully operational offsite data processing facility equipped with hardware and software." It mirrors production and is kept continuously updated, so it can take traffic almost immediately. It is the fastest recovery and the most expensive to run, with recovery in minutes.

The selection rule falls straight out of the ladder: pick the cheapest tier that still meets your RTO. A long RTO on a tight budget tolerates a cold site; a near-zero RTO with budget to match needs a hot site; a warm site splits the difference.

Redundancy modes: active-active vs active-passive

Separately from the site tier, you choose how two or more nodes (or sites) run relative to each other.

In active-active, every node carries live traffic at the same time. The design both spreads load and survives the loss of a node, because the survivors simply absorb the failed node's share. This gives the shortest recovery (often no perceptible outage at all) but demands that every node be sized to carry extra load and kept fully synchronized, which is the most expensive and complex option.

In active-passive, the primary handles all traffic while the secondary stands by idle (or warm) until a failover promotes it to active. This is simpler and cheaper because the standby does not serve users day to day, but recovery includes the failover delay and the idle capacity is unused until something breaks. A hot site fronted by automated failover is the active-passive design that gets closest to active-active's recovery time.

The two choices compose: a hot site can be run active-active (the lowest possible RTO, highest cost) or active-passive (slightly higher RTO at the cost of the failover, lower steady-state cost).

A DR plan you have never exercised is a guess. Stale runbooks, expired credentials, an undersized replication link, a backup that never actually restores: these go unnoticed until the real disaster, when there is no time left to discover them. So DR plans are tested on a ladder of increasing realism and increasing risk, and the rule is to start low and climb only as confidence grows.

The four test types, lowest to highest fidelity

• A tabletop exercise (also called a walkthrough) is a discussion. The recovery team sits down with the plan and a scenario and talks through who does what, surfacing gaps and ambiguities. No systems are touched, so it is the cheapest and safest test and the natural first step. Its weakness is that it only proves the plan reads correctly, not that it works.
• A simulation models the disaster and the response more concretely, exercising procedures and decision-making against a constructed scenario, without performing a real production cutover. It catches process problems a pure discussion misses while still keeping production untouched.
• A parallel test stands the recovery systems up for real and runs them alongside live production. Real data flows into the recovery environment and you confirm it produces correct results, but production stays primary the whole time, so users are never at risk. This proves the recovery site actually functions without betting the business on it.
• A full failover test (also called a full-interruption test or cutover) actually shifts production to the recovery site, exactly as a real disaster would. It is the only test that proves end-to-end recovery, including the failover mechanism and real user load, and it is the most disruptive and risky, so it is run rarely, in a maintenance window, with a tested back-out to the primary.

The pattern and the trap

The ladder trades realism for risk: each rung proves more but costs and endangers more, so you climb it as the plan earns trust. The exam reflex: a question that wants the lowest-cost, no-impact validation points to the tabletop; one that wants proof the recovery site truly works under real conditions points to the parallel test or, for the most complete proof, the full failover. The trap is assuming a documented plan equals a working capability. Until a parallel or full-failover test exercises it end to end, the plan has untested gaps no matter how polished the runbook reads.

Disaster-recovery questions on N10-009 are mostly term-to-definition and scenario-to-metric items. The stem gives a number, a constraint, or a situation and asks for the matching acronym, site type, or test. Train these reflexes.

Metric reflexes

• "How much data can we afford to lose" / "how often must we back up" = RPO. A stated tolerable data loss ("no more than 15 minutes of transactions") is an RPO, and it sets backup cadence.
• "How long can the service be down" / "how fast must we be back online" = RTO. A stated tolerable outage ("restore within 4 hours") is an RTO, and it drives the recovery method and site tier.
• "Average time to fix a failed component" = MTTR (recoverability; lower is better).
• "Average operating time between failures" = MTBF (reliability; higher is better).
• Set-vs-measured cue: RPO and RTO are objectives you choose; MTTR and MTBF are statistics you measure from failure history.

Site and redundancy reflexes

• "Space, power, and cooling but no equipment; cheapest; slowest" = cold site.
• "Partially equipped; needs current data loaded; moderate" = warm site.
• "Fully operational, continuously updated, ready immediately; costliest" = hot site.
• "All nodes serve traffic at once; load-shared and survives a node loss" = active-active.
• "Standby waits idle until a failover promotes it" = active-passive.

Test reflexes

• "Discussion-based, no systems touched, find gaps cheaply" = tabletop / walkthrough.
• "Models the disaster and response without a real cutover" = simulation.
• "Recovery runs alongside live production to prove it works" = parallel test.
• "Production is actually shifted to the recovery site" = full failover / cutover (most realistic, most disruptive).

The traps

• Swapping RPO and RTO. This is the single most common DR distractor. Backups and data loss = RPO; downtime and time-to-restore = RTO.
• Treating MTTR/MTBF as targets. They are measured, not declared. An answer that "sets" an MTBF the way you set an RTO is wrong.
• Choosing a hotter site than the RTO needs. The correct site is the cheapest tier that still meets the RTO, not always the fastest one. A loose RTO with a tight budget points to a cold or warm site, not a hot site.
• Equating a documented plan with a tested one. If a stem asks what proves the plan actually works, the answer is a test that exercises it (parallel or full failover), not the existence of the runbook.