DR & Backups9 min read•October 2026•Kernelry Staff

Disaster Recovery, RPO, RTO & High Availability Scenarios

In senior reliability engineering roles, interviewers probe how candidates design for worst-case regional outages. Core concepts include quantifying Recovery Point Objective (RPO) and Recovery Time Objective (RTO), evaluating active-active versus active-passive failover tiers, ensuring immutable ransomware-protected backups, and conducting routine game days.

Architecture Blueprint

Disaster Recovery Strategies: Cost vs RTO/RPO Tradeoff

Spectrum of DR architectures from low-cost Backup & Restore to Pilot Light, Warm Standby, and Multi-Region Active-Active.

The Core Concept

Disaster Recovery (DR) is the set of engineering strategies, backup procedures, and failover mechanisms that restore mission-critical systems following hardware outages, human error, ransomware attacks, or cloud datacenter disasters.

Key Rules of Thumb for Production Systems

•RPO (Recovery Point Objective) measures maximum acceptable data loss in time; RTO (Recovery Time Objective) measures maximum acceptable downtime.

•Multi-region active-active architectures eliminate failover downtime but introduce complex cross-region data consistency challenges.

•Backups must be stored in immutable storage (Object Lock/WORM) in an isolated account to survive credential compromise and ransomware.

•Regular automated restore drills and team game days are the only reliable way to guarantee disaster recovery capability.

Why Production Systems Demand This Depth

An untested backup is not a backup; it is merely an unverified hope. When a primary cloud region experiences a widespread fiber cut or power loss, organizations without automated DNS failover, cross-region database replication, and proven runbooks face catastrophic business loss and reputation damage.

Common Diagnostic Pitfalls

•Storing database backups in the same cloud account and region as production systems, leaving them vulnerable to ransomware.
•Failing to regularly test backup restoration, only discovering corrupted archives or missing encryption keys during an actual crisis.
•Promoting an asynchronous read replica to primary without accounting for data replication lag (violating RPO).
•Assuming DNS TTL expiration guarantees instant client traffic failover (some resolvers, client libraries and JVMs cache records for longer than the TTL).

Best Practices for Scenario Drills

•Begin any disaster recovery discussion by asking the interviewer to define target RPO and RTO thresholds, as they dictate architecture and cost.
•Walk through the four classic cloud DR tiers: Backup & Restore, Pilot Light, Warm Standby, and Multi-Site Active-Active.
•Highlight the operational complexity of split-brain scenarios when configuring automated cross-region database failover.

Worked Production Scenarios

Study real-world outage patterns, diagnostic trees, and clarifying questions.

Architecting a Multi-Region Failover Strategy

Difficulty: senior

Scenario Prompt:

“Your company primary cloud region hosts an e-commerce platform processing $50,000 an hour in orders. Executive leadership mandates an RTO under 15 minutes and an RPO under 1 minute for a total regional catastrophe. How do you design the architecture and failover plan?”

Substantive Technical Concepts to Hit:

RPO Database Replication

Implement cross-region asynchronous database replication (e.g., Aurora Global Database or PostgreSQL replication) maintaining sub-second replication lag.

Warm Standby Infrastructure

Maintain a scaled-down warm standby cluster in the secondary region with minimum core services running, ready to autoscale upon failover.

Global Traffic Steering

Use Route 53 Application Recovery Controller (ARC) or Cloudflare with health-checked DNS and Anycast routing to redirect traffic within minutes.

Split-Brain Prevention

Implement fencing tokens and runbook gates to ensure the failed primary region cannot process writes if it recovers unexpectedly during failover.

Senior Clarifying Questions to Investigate:

What is the budget tolerance for maintaining warm standby compute capacity in the secondary region 24/7?
Can stateless frontend services run active-active while keeping stateful transactional writes pinned to an active-passive model?
What is the rollback procedure once the primary region is fully recovered?

A Database Restore Takes Longer Than the RTO

Difficulty: mid

Scenario Prompt:

“During a quarterly drill, restoring the 2 TB PostgreSQL database from its nightly pg_dump takes nine hours. The agreed RTO is two hours. What do you change?”

Substantive Technical Concepts to Hit:

Logical vs Physical Backups

Restoring a pg_dump replays every row and rebuilds every index, so it scales badly with size. A physical base backup (pg_basebackup, pgBackRest) restores by copying data files and is much faster for large databases.

Point-in-Time Recovery with WAL Archiving

A base backup plus continuously archived WAL lets you restore to any moment, so data loss is limited to the last archived WAL segment instead of the last nightly dump.

Snapshot Restores Have a Warm-Up Cost

EBS and RDS snapshots restore quickly but load blocks lazily, so the first reads after a restore can be slow. Fast snapshot restore or reading the volume once before cutover avoids a slow first hour.

Promote Instead of Restore

For a two-hour RTO, a streaming replica that can be promoted is often the main recovery path, with backups kept for corruption and deletion that replication would copy.

Senior Clarifying Questions to Investigate:

Does the two-hour RTO cover only the database, or the whole service including DNS and application warm-up?
Are backups and the WAL archive stored in a separate account so a compromised production account cannot delete them?
Is restore time recorded at every drill so a trend toward the RTO is caught early?

Interactive Scenario Drills

That's where we come in.

Explaining architectural trade-offs under time constraints is very different from reading about them.

Kernelry helps you rehearse technical questions, evidence-led debugging, safe next steps, and the clear communication teams expect. The drills adapt to your answer, so you practice the reasoning rather than memorize a company-specific script. During early access, free accounts include ten daily scenarios across our mixed question pool, with no credit card required.

Start Today's Free Drill

Back to All Blog Posts