# High Availability vs Backup vs Disaster Recovery: Building a Resilient Enterprise Infrastructure

Design enterprise HA, independent backups and disaster recovery with VMware, SQL Server, seven scenarios, RPO/RTO targets and a practical recovery runbook.

## 1. Three incidents, three different recovery paths

At 03:00 an ESXi host stops responding. In this hypothetical organisation, HA restarts its ERP VM elsewhere, but the open accounting session is gone. A week later, an operator deletes invoice rows: both SQL replicas faithfully retain the deletion. Then the entire building loses power. The servers are redundant, yet the branch offices cannot reach authentication, DNS or the ERP. These are three different incidents; buying another cluster does not answer all three.

From an Infrastructure & DevOps Engineer’s perspective, the useful question is specific: which business transaction can resume, using which data, through which network, and after how long? This guide builds that answer from failure domains and dependencies. All organisations, timings and thresholds below are design examples, not reports of personal deployments or promises of recovery.

Version scope: VMware vSphere/ESXi 8.0, PowerCLI 13.x, SQL Server 2022 (16.x) on Windows with WSFC, Ubuntu 24.04/systemd, Kubernetes 1.36 API examples, Docker Engine Swarm mode and Zabbix 7.0 trigger syntax. These are explicit reference baselines, not a claim that every branch is the newest. Check licensing, current lifecycle, patch and hardware compatibility before implementation. SQL 2025 backup capabilities must not be assumed on 2022.

## 2. What each protection layer actually buys

| Concept | Operational meaning and boundary |
| --- | --- |
| High Availability | Reduce interruption through redundancy and failover; a restarted VM still has boot and application recovery time. |
| Fault Tolerance | Maintain execution through a covered component failure; vSphere FT keeps a secondary execution instance, within supported limits. Logical mistakes remain mistakes. |
| Backup | Retained, independently recoverable historical copies; restore an invoice from before deletion. Its value depends on accessible keys and a working restore chain. |
| Replication | Transfer changes to another copy for availability or DR; deletion may propagate. Lag is not a retention policy. |
| Disaster Recovery | Recover IT services after loss of a site or major platform using an alternate environment and a rehearsed sequence. |
| Business Continuity | Keep business activity viable, including people, premises, suppliers and authorised manual work while IT recovers. |
| Failover | Transfer service ownership to a surviving target; define the single writer and client reconnection path. |
| Failback | Controlled return after repair, reverse synchronisation and validation; the old primary may contain stale data. |
| Redundancy | Extra components, such as two PSUs on independent power feeds; two devices sharing one feed still share its failure. |
| Data Protection | Combined access controls, integrity, encryption, retention and recoverability across the data lifecycle. |
| Disaster Recovery Plan | Named decision makers, dependencies, recovery steps, credentials access, evidence, acceptance criteria and failback for IT. |
| Business Continuity Plan | Business-owned priorities, alternate procedures, staffing, communications and maximum tolerable disruption; DR supports it. |

| Technology | Main Purpose | Protects Against | Does Not Protect Against |
| --- | --- | --- | --- |
| VMware HA | VM restart | Covered host loss with accessible disks and spare capacity | Shared array loss, deletion, ransomware, session loss |
| vSphere FT | Execution continuity | Supported primary host failure | Bad writes, shared dependency failure, backup retention |
| SQL Always On AG | Database role continuity | Replica failure under quorum/sync policy | Accidental DELETE, every corruption type, historical recovery |
| Storage replication | Current block copy elsewhere | Some array/site failures depending on independence | Logical deletion or encrypted writes being copied |
| Kubernetes / Swarm | Reconcile workloads | Worker failure when control, capacity and dependencies survive | Lost local volume, application mistakes, whole-site loss |
| Independent backup | Historical recovery | Deletion/corruption before a retained clean point | Instant service continuity or guaranteed clean recovery |
| Multi-site DR | Business-service recovery | Declared disaster within tested scope | Untested dependencies or an already-compromised recovery site |

A deleted SQL table is a committed change that the AG normally replays. A deleted VMDK on shared storage is not reconstructed by moving the VM to another host. A mistaken Kubernetes rollout can be reconciled very efficiently on every worker. Availability mechanisms reproduce the requested state; historical protection must preserve a state from before the mistake.

## 3. RPO and RTO: turn tolerance into an acceptance test

![RPO looks back from incident to the last recoverable point; RTO looks forward to validated service recovery](/assets/img/articles/content/high-availability-backup-disaster-recovery-enterprise-guide/rpo-rto.svg)

RPO is the maximum acceptable age of the recovered data relative to the disruption. If the only usable backup is at 01:00 and the server fails at 04:00, changes from that three-hour window may be lost. A stated RPO of 15 minutes does not change this result: the design has missed its target. A later intact log backup, reachable replica or safe tail-log backup may reduce actual loss; inspect the evidence before declaring a number. Three hours is an exposure window, not proof that three hours of transactions existed.

RTO is the target elapsed time from disruption until the business service meets its agreed recovery criteria. Detection, declaration, infrastructure, data restore, application start and validation consume this budget. Four hours of ERP outage may stop warehouse dispatch and payroll approval even when database restoration took only forty minutes. Record actual elapsed recovery time separately. In this article, RTA means Recovery Time Actual and RPA means Recovery Point Actual (the measured recoverable data age); these labels are not universal, so write their definitions in the DR plan.

| Service | Illustrative RPO / RTO | Defensible design and business check |
| --- | --- | --- |
| A: File Server | 4 h / 8 h | Four-hour backups plus offsite copies; owner accepts re-entering ordinary documents, but protected legal shares may need tighter targets. |
| B: Enterprise ERP | 15 min / 2 h | Log backups plus independently recoverable app/configuration; test identity-to-order-entry in two hours at expected load. |
| C: Critical financial database | 1 min / 30 min | Assess synchronous local HA plus remote durability/log protection and warm recovery capacity; reconcile external settlements. These targets cost more and require measured evidence. |

These numbers need Business Owner approval after a business impact analysis, including peak periods and maximum tolerable disruption. An asynchronous remote copy cannot be advertised as zero-loss DR. Size WAN capacity against sustained log generation and catch-up bursts; measure queue age, not just link speed.

## 4. VMware: the host can fail while storage must remain usable

![Three ESXi hosts, vCenter and shared storage: host failover restarts VMs but shared storage failure blocks every host](/assets/img/articles/content/high-availability-backup-disaster-recovery-enterprise-guide/vmware-ha.svg)

ESXi executes VMs; vCenter configures the cluster and supplies central management. Once configured, host HA agents can respond to a host failure without a running vCenter, but losing management still complicates operations. HA is a restart mechanism. vMotion moves a running VM during planned work when the source is alive; Storage vMotion migrates its disks. DRS balances placement, with licensing and automation settings controlling its behaviour. Neither migration tool replaces recovery after the source has died.

### Host failure: what the three-host cluster really does

Suppose H1 loses power. HA evaluates host/network/datastore heartbeat evidence and the configured responses, then restarts eligible protected VMs on H2/H3 once failure handling and placement allow it. There is no universal restart time: detection, locks, datastore accessibility, admission policy, boot and database crash recovery all matter. In-memory sessions and unfinished requests do not survive a cold restart. Applications need retries and idempotency rather than an assumption that every interrupted payment failed.

Reserve failure capacity with HA Admission Control and validate N+1 under peak CPU, RAM, storage IOPS and licensing constraints. VM Restart Priority orders attempts; it does not prove that AD or SQL is ready before ERP starts. Use application health gates. Host Isolation Response needs deliberate management-network and isolation-address design. VM Monitoring uses VMware Tools heartbeats and associated activity checks; a healthy Tools heartbeat is not evidence that an ERP transaction succeeds. FT can reduce covered execution interruption for supported VMs but shares data and many dependencies; verify vCPU, device, networking and licensing limits for the exact build.

### Storage failure: another host cannot read an unavailable array

If every host uses one unavailable array, restarting on a different host provides no accessible VMDK. Compute HA needs storage resilience: independent paths, redundant controllers, tested multipathing and, where justified, independently recoverable storage at another site. VM Component Protection distinguishes APD from PDL and applies configured actions, but recovery still needs a host that can access valid disks. A second switch does not make two paths independent if they share a controller, power source or fibre route. Follow vendor VMCP guidance rather than changing advanced timeout settings during an incident.

[Source: vSphere 8 Availability, HA and FT](https://techdocs.broadcom.com/us/en/vmware-cis/vsphere/vsphere/8-0/vsphere-availability-8-0.html)

[Source: VMCP, APD and PDL configuration boundaries](https://knowledge.broadcom.com/external/article/406100/best-practice-for-vsphere-high-availabil.html)

### Snapshot growth: investigate before consolidation

A VM snapshot retains a disk change chain; it is usually tied to the same base disks and datastore. It is not an independent backup. Growing deltas can exhaust a datastore, stun VMs and interrupt writes across unrelated guests. Start by checking free bytes, growth rate, snapshot age, active backup jobs, VM disk-chain health, array capacity and storage latency. Check consolidation-needed alarms and the backup platform state. Snapshot SizeGB is a clue, not a complete forecast of consolidation scratch space.

Before an approved consolidation, confirm a usable independent backup, adequate working space based on vendor guidance, a maintenance window and expected I/O impact. Escalate broken chains or uncertain ownership to support. Do not delete delta files, remove locks or launch concurrent consolidation/backup tasks to free space. If an operation stalls, retain task/log evidence and use vendor support; there is no safe generic “undo” for manual VMDK manipulation. Monitor projected time-to-full so the decision occurs before writes fail.

## 5. SQL Server Always On: a second writer is not a time machine

![Primary SQL sends transaction logs to a synchronous local secondary and an optional asynchronous DR replica; retained backups follow a separate path](/assets/img/articles/content/high-availability-backup-disaster-recovery-enterprise-guide/sql-always-on.svg)

The baseline two-node Windows AG has a Primary Replica accepting writes, a Secondary Replica replaying transaction log records, WSFC membership/quorum and an Availability Group Listener for client reconnection. Each replica has its own database storage. Put the nodes on independent hosts and plan witness placement so one site loss does not leave the intended survivor without quorum. A listener is an endpoint, not a transaction buffer; clients still need supported drivers, retry handling and, for multi-subnet designs, validated DNS/connection-string behaviour.

Synchronous Commit waits for the synchronous secondary to harden the log before acknowledging the covered commit; redo need not already have applied it. Asynchronous Commit avoids waiting for remote durability and therefore permits a loss window. Log Send Queue measures unsent log; Redo Queue measures hardened log awaiting redo. High redo backlog can lengthen recovery/readable-secondary freshness even with healthy transport. Inspect rates, queue age, disconnects and synchronization state together.

### Incident 1: the primary server fails

Automatic failover requires a supported automatic-failover pair configured for synchronous commit, a healthy synchronized target, WSFC quorum and a qualifying failure under the failover policy. A lost file or individual database fault is not automatically every AG failover trigger; database-level health detection is a separate configuration. Planned manual failover without data loss also needs a synchronized synchronous target. Forced failover to an asynchronous or unsynchronized target can lose acknowledged transactions and requires explicit incident authority, fencing and later reconciliation. This article does not issue that command.

[Source: SQL Server AG failover modes and prerequisites](https://learn.microsoft.com/en-us/sql/database-engine/availability-groups/windows/failover-and-failover-modes-always-on-availability-groups?view=sql-server-ver16)

### Incident 2: invoices are deleted

A valid DELETE or DROP is a logical change, so the log carries it to replicas. Stop the faulty application path through the incident process, preserve logs, identify the commit time and restore an earlier point into an isolated, differently named database. Compare keys and business totals before an authorised data repair. Failing over merely changes the server serving the same deletion. Do not assume all third-party dependencies or multiple databases share a transactionally consistent point without an application-aware recovery design.

### Incident 3: corruption needs diagnosis, not an automatic promotion

An infrastructure outage, logical corruption and a physical page fault need different investigations. Some supported page errors can use AG automatic page repair from an intact partner page; this is not a general repair guarantee. A storage fault may affect only one replica; an application bug can generate wrong but physically valid data on both. Preserve error logs, page/checksum evidence and backup chains. Run DBCC CHECKDB on a controlled restored copy where possible, assess resource impact before production checks, and choose a validated clean restore or supported repair path with the DBA/vendor. Do not jump to REPAIR_ALLOW_DATA_LOSS.

[Source: automatic page repair scope](https://learn.microsoft.com/en-us/sql/sql-server/failover-clusters/automatic-page-repair-availability-groups-database-mirroring?view=sql-server-ver16)

For SQL 2022, use regular Full backups as differential bases, Differential backups for changes since that base, and an unbroken Transaction Log Backup chain under FULL recovery for point-in-time recovery. Backup Preferences express placement intent; jobs must implement it, for example with sys.fn_hadr_backup_is_preferred_replica. In this baseline, secondary full backups are copy-only and differential backups are not supported there. Keep the normal full/differential plan on a supported primary workflow. SQL Server Standard Basic AG has materially narrower database/replica/features than Enterprise AG; verify the edition before copying this design.

[Source: version-specific secondary backup support](https://learn.microsoft.com/en-us/sql/database-engine/availability-groups/windows/active-secondaries-backup-on-secondary-replicas-always-on-availability-groups?view=sql-server-ver16)

## 6. SQL backup and point-in-time restore in an isolated lab

Platform: SQL Server 2022 Developer, Windows, SSMS/sqlcmd with GO support. This lab creates new databases and writes new backup files; run only on an isolated instance with change approval. Provision empty C:\DRLab\backup and C:\DRLab\data directories, grant the SQL service account required access, check capacity and use a fresh lab instance. No credential is embedded. Never redirect these commands to production. The guard refuses an existing lab source or restore database; no WITH REPLACE, file deletion or production overwrite is used.

```sql
:ON ERROR EXIT
USE master;
GO
IF DB_ID(N'ResilienceLab') IS NOT NULL
 OR DB_ID(N'ResilienceLab_Restore') IS NOT NULL
    THROW 51000, 'Use a fresh isolated lab instance.', 1;
GO
CREATE DATABASE ResilienceLab
 ON PRIMARY (NAME=N'ResilienceLab', FILENAME=N'C:\DRLab\data\ResilienceLab.mdf')
 LOG ON (NAME=N'ResilienceLab_log', FILENAME=N'C:\DRLab\data\ResilienceLab_log.ldf');
GO
ALTER DATABASE ResilienceLab SET RECOVERY FULL;
GO
USE ResilienceLab;
CREATE TABLE dbo.Invoice (Id int PRIMARY KEY, Amount decimal(12,2) NOT NULL);
INSERT dbo.Invoice VALUES (1,100.00);
GO
BACKUP DATABASE ResilienceLab TO DISK=N'C:\DRLab\backup\lab-full.bak'
 WITH CHECKSUM;
GO
INSERT dbo.Invoice VALUES (2,200.00);
BACKUP DATABASE ResilienceLab TO DISK=N'C:\DRLab\backup\lab-diff.bak'
 WITH DIFFERENTIAL, CHECKSUM;
GO
INSERT dbo.Invoice VALUES (3,300.00);
GO
-- Copy this server-local timestamp into STOPAT below.
SELECT CONVERT(varchar(23), GETDATE(), 121) AS RecoveryTarget;
WAITFOR DELAY '00:00:03';
UPDATE dbo.Invoice SET Amount=999.00 WHERE Id=3;
BACKUP LOG ResilienceLab TO DISK=N'C:\DRLab\backup\lab-log.trn' WITH CHECKSUM;
GO
RESTORE VERIFYONLY FROM DISK=N'C:\DRLab\backup\lab-full.bak' WITH CHECKSUM;
RESTORE VERIFYONLY FROM DISK=N'C:\DRLab\backup\lab-diff.bak' WITH CHECKSUM;
RESTORE VERIFYONLY FROM DISK=N'C:\DRLab\backup\lab-log.trn' WITH CHECKSUM;
RESTORE FILELISTONLY FROM DISK=N'C:\DRLab\backup\lab-full.bak';
GO
```

Execution contract: run the preparation file with sqlcmd 18 using integrated authentication and -b, for example sqlcmd -S localhost -E -b -i sql-lab-prepare.sql, with a trusted server certificate. Both files require SQLCMD directives; do not run them in ordinary SSMS query mode. :ON ERROR EXIT and -b terminate execution on SQL errors across GO batches. Capture RecoveryTarget, edit the NULL value in the second file, review it, then execute that file once with the same runner. Keep the original target timestamp; rerunning preparation is not the restore workflow.

```sql
:ON ERROR EXIT
USE master;
GO
-- Set to RecoveryTarget returned by the preparation file.
DECLARE @RecoveryTarget datetime = NULL;
IF @RecoveryTarget IS NULL
    THROW 51001, 'Set the captured RecoveryTarget first.', 1;
IF DB_ID(N'ResilienceLab_Restore') IS NOT NULL
    THROW 51002, 'Restore database already exists; stop the exercise.', 1;
RESTORE DATABASE ResilienceLab_Restore FROM DISK=N'C:\DRLab\backup\lab-full.bak'
 WITH MOVE N'ResilienceLab' TO N'C:\DRLab\data\ResilienceLab_Restore.mdf',
 MOVE N'ResilienceLab_log' TO N'C:\DRLab\data\ResilienceLab_Restore_log.ldf',
 NORECOVERY, CHECKSUM;
RESTORE DATABASE ResilienceLab_Restore FROM DISK=N'C:\DRLab\backup\lab-diff.bak'
 WITH NORECOVERY, CHECKSUM;
RESTORE LOG ResilienceLab_Restore FROM DISK=N'C:\DRLab\backup\lab-log.trn'
 WITH STOPAT=@RecoveryTarget, RECOVERY, CHECKSUM;
GO
DBCC CHECKDB (N'ResilienceLab_Restore') WITH NO_INFOMSGS;
SELECT Id,Amount FROM ResilienceLab_Restore.dbo.Invoice ORDER BY Id;
```

Expected: backup/restore completion messages, VERIFYONLY success, CHECKDB with no reported integrity errors and three rows totalling 600.00; row 3 is 300.00, not 999.00. FILELISTONLY must match the logical names used in MOVE. Interpret failures as a stopped exercise, not permission to overwrite another database. If the target was captured incorrectly, retain the source/backups and rebuild the exercise in a fresh isolated instance. Rollback is disconnecting the test environment; cleanup is a separately reviewed lab teardown. These commands were inspected, not executed against a live SQL instance during article preparation.

VERIFYONLY checks backup readability/completeness, not the full logical correctness of its contents. CHECKSUM also has limits. A successful restore plus CHECKDB and application reconciliation provide stronger evidence. Production policy additionally needs backup encryption, independently protected certificate/private-key exports, tested key recovery, retention long enough for delayed discovery and capacity for every dependent full/differential/log file. A tail-log backup may help after an incident when the log is accessible, but its safe feasibility is a DBA decision.

[Source: point-in-time recovery and recovery-model limits](https://learn.microsoft.com/en-us/sql/relational-databases/backup-restore/restore-a-sql-server-database-to-a-point-in-time-full-recovery-model?view=sql-server-ver16)

## 7. Independent backup architecture: 3-2-1-1-0

![VMware VMs, SQL and file servers feed an isolated repository with separate offsite immutable and offline copies](/assets/img/articles/content/high-availability-backup-disaster-recovery-enterprise-guide/enterprise-backup.svg)

| Rule | Concrete design example |
| --- | --- |
| 3 | Production plus a local backup plus an offsite copy: three data copies, not three snapshots of the same volume. |
| 2 | Two independent storage/media systems, e.g. production array and backup disk with tape/object copy; evaluate common control-plane and firmware risks too. |
| 1 offsite | A copy outside the primary site’s power, flood, fire and administrator failure domain. |
| 1 offline or immutable | A physically offline rotation or enforced immutable retention under separate administration; an online immutable store is not an air gap. |
| 0 | No unresolved errors in verified recovery evidence: scheduled restore, integrity and application tests; not a promise of zero future failures. |

For VMware, SQL and file services, keep a backup control server in a restricted management zone, separate repository storage and restricted data-transfer paths. Protect the backup catalogue/configuration as well as payloads. SQL-aware processing must coordinate VSS or database-native backups and log ownership; two uncoordinated jobs can make troubleshooting and retention unsafe. A backup server sharing production domain-admin credentials and the same writable datastore has a common compromise path even if its dashboard is green.

Application-consistent backup coordinates application flushing/quiescence; crash-consistent backup resembles recovery after power loss and may need log replay or application repair. Neither label proves an entire multi-service transaction is consistent. A storage snapshot is useful for short rollback or as a supported backup capture stage. Replication supplies another current copy. Backup retains independently recoverable history. Test all required volumes, logs and external dependencies at the selected recovery point.

A sample retention policy is 30 daily, 12 weekly and 12 monthly recovery points, with annual copies only where business/legal requirements justify them. GFS (Grandfather-Father-Son) is a tiered retention pattern, not an exemption from capacity planning. Forecast full chains, changed data, immutability lock duration, object versions and restore working space. Encrypt in transit and at rest, separate keys from data, use MFA and least privilege, restrict repository management, audit deletion and test break-glass access without production identity. Offline means inaccessible to the live attack path; air-gapped means a genuine isolation boundary, with a controlled procedure for reconnecting media.

## 8. Seven organisations, seven defensible designs

### Scenario 1 — five servers

With AD, file, SQL and application services on one or two hypervisors, start with independent VM/application backups, daily offsite copying, one protected offline/immutable copy and a spare restore target sized for essential services. With one host, explicitly accept a restore-based outage; two hosts without independent storage and sufficient capacity are not automatically HA. Add a second domain controller on an independent failure domain where feasible. A hypothetical 4-hour RPO/8-hour RTO may suit ordinary files, while SQL logs need a separately agreed interval. Low licensing cost is traded for manual recovery and staff availability.

### Scenario 2 — twenty VMs

For VMware, Windows/Linux and ERP on shared storage, design three-host N+1 HA if the uptime requirement funds it; dual storage paths must terminate in a resilient storage system. Use application-aware VM backup to a different repository, SQL log protection, an offsite immutable copy and monthly isolated full-service restore. Define file-level, whole-VM and database restore paths separately. A two-hour ERP RTO must include AD/DNS and client reconnection. A single shared array remains a boundary even with three hosts.

### Scenario 3 — fifty VMs

Separate compute HA from database HA: spread SQL AG replicas across hosts, provide quorum/witness independence, monitor send/redo queues and keep SQL backups outside both replica storage systems. Assign recovery waves from the dependency map, not VM names: identity/network first, data next, then ERP/API and reporting. Keep backup administration isolated and reserve restore IOPS while backups run. Fund an on-call primary and deputy; a sophisticated platform owned by one unavailable administrator has an operational single point of failure.

### Scenario 4 — one hundred VMs

Across multiple clusters and datastores, catalogue business services with owners, recovery tiers and cross-service transaction dependencies. Stagger backups by measured change rate and source latency; prioritise log jobs and avoid simultaneous synthetic fulls exhausting repository IOPS. Model datastore growth, snapshot growth and concurrent recovery throughput. A DR site may serve only a documented critical subset, with resource reservations and a load-tested reduced mode. Quarterly application tests expose missing certificates and middleware that per-VM restore tests miss.

### Scenario 5 — two data centres

Choose active-passive when one writable service stack with a warm recovery stack meets objectives: it simplifies ownership but consumes idle capacity and requires promotion time. Active-active can serve traffic at both sites only when application state, routing, data conflicts and failure capacity are explicitly designed. Two writable copies of a legacy ERP database are not an active-active design. Prefer synchronous local HA and measured asynchronous remote DR where WAN latency makes cross-site synchronous commits unacceptable. Keep an independent offsite backup outside both replication administration paths.

### Scenario 6 — Kubernetes and stateful applications

Distribute control-plane/etcd voters and workers across failure domains; protect the API endpoint, CNI, CSI, registry, ingress and DNS. Back up etcd/API configuration, secrets with their encryption keys, manifests and application data with database-consistent procedures. PersistentVolume durability depends on its backend and topology; StatefulSet supplies identity/ordering, not database replication. Rebuilding a Pod does not restore a lost local volume. Use a separate recovery cluster and rehearse PV restoration and application reconciliation together.

### Scenario 7 — headquarters and fifty branches

For centrally hosted ERP, classify reachability independently from data-centre health. One branch losing its circuit needs dual genuinely diverse WAN paths, local DNS caching/forwarding design and an approved offline procedure; moving ERP to DR will not repair that last mile. A WAN-wide fault needs carrier diversity and route convergence tests. A DNS failure needs independent authoritative/recursive infrastructure and validated records. Central ERP failure needs application/database recovery. Whole-DC failure needs branch routes, firewall policy and name resolution to the DR service, tested from representative branches.

Local continuity may mean queueing approved orders with unique identifiers and reconciling them later, never silently creating a second financial system of record. Keep network-device configurations, VPN/SD-WAN templates and protected emergency access offsite. Test DR connectivity under failover and failback; two links through the same street duct do not provide meaningful route diversity.

## 9. Kubernetes and Docker Swarm: replacement has prerequisites

![Kubernetes replaces eligible Pods and Swarm replaces service tasks after worker failure; both depend on healthy control, storage, network and application data](/assets/img/articles/content/high-availability-backup-disaster-recovery-enterprise-guide/orchestrator-ha.svg)

Kubernetes detects node problems and reconciles eligible workloads according to controller, eviction and scheduling rules. Swarm managers reconcile service tasks when manager quorum and eligible worker capacity remain. Neither promises immediate replacement or preserves process memory. A local Docker volume stays on its node; a Kubernetes ReadWriteOnce volume may require safe detach/attach and topology-compatible capacity. During a partition, fence the old stateful writer before allowing another to take over. Force-deleting a Pod is not proof that its old process is dead.

For Swarm, three voting managers tolerate one manager loss; workers add no votes. Follow Docker’s documented manager-state backup procedure and retain the autolock key separately if enabled. For Kubernetes, etcd snapshots protect cluster state, not the SQL rows or PV payload. For both, retain approved images and configuration outside the lost site. Readiness/health checks must represent the dependency path that users need, with backoff and a controlled traffic switch.

[Source: StatefulSet guarantees and limitations](https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/)

[Source: Swarm quorum, manager backup and recovery](https://docs.docker.com/engine/swarm/admin_guide/)

## 10. Design the recovery site as a complete service path

![Independent primary and recovery data centres, branch connectivity, identity, compute, storage and application tiers with separate backup protection](/assets/img/articles/content/high-availability-backup-disaster-recovery-enterprise-guide/primary-dr.svg)

Start with a dependency graph: branch client → DNS → WAN/firewall → identity/PKI → application ingress/API → database → storage. Add NTP, secrets, licensing, queues, object storage and third-party APIs where actually used. Pre-provision the recovery network, sufficient compute and storage IOPS, monitoring and protected administration. The recovery cluster must not depend on primary vCenter, primary-only password vaults or a primary-site internet exit. Keep offline-readable runbooks and an inventory of asset owners.

| Dependency / question | Design and validation |
| --- | --- |
| AD and DNS | For domain-dependent services, maintain appropriate DR domain controllers/DNS and tested system-state/forest recovery procedures. Replicated AD can also replicate compromise; cyber recovery needs a separate trusted identity plan. |
| PKI / certificates | Protect CA database/configuration, private keys, chains and reachable revocation endpoints. Check SANs and expiry for DR names; a valid database does not fix TLS failure. |
| Firewall / routing | Pre-stage least-privilege DR rules, NAT/VPN, routes and branch reachability; activate only under change control after confirming destination ownership. |
| IP and application connections | Prefer stable service names/listeners. Decide routed DR subnets versus supported stretched addressing; test DNS TTL, caches and client retries. IPs may change and every hardcoded endpoint must be inventoried. |
| Split-brain | Use product-supported quorum, witness and fencing. Confirm the old writer is isolated before promotion; never treat loss of ping as proof it cannot write. |
| Data validation | Capture selected recovery timestamps/LSNs, DB integrity evidence, queue offsets and business balances; reconcile external transactions and obtain owner acceptance. |
| Failback | Repair and validate the old site, back up current DR data, synchronise from the authoritative DR writer, schedule a controlled switch, monitor and keep a return-to-DR path. |

Keep local synchronous database HA and optional remote asynchronous DR roles distinct. The SQL diagram’s remote replica is an extension to the baseline two-node example, not a requirement that a two-node AG somehow has three nodes. If licensing or WAN makes replication unsuitable, a backup-based recovery service can be defensible when measured restore times meet the signed targets. The independent backup copy needs a third failure/administration boundary where feasible, not merely a folder on the DR storage.

## 11. If the whole data centre goes down at 03:00, where do we start?

![Incident detection, classification, impact assessment, declaration, infrastructure and data recovery, validation and business restoration](/assets/img/articles/content/high-availability-backup-disaster-recovery-enterprise-guide/dr-workflow.svg)

This is a reference runbook for a hypothetical Windows/VMware/SQL ERP, not approval to fail over production. Maintain an approved site-specific copy with actual owners, contacts, service graph, isolation steps, product-specific promotion commands and escalation deadlines. All commands below are diagnostic; “activation” refers to the separately approved recovery procedure. Use a common incident clock, record evidence securely and stop on conflicting writer ownership or suspected compromise. The recovery order follows dependencies: network/time/identity/trust, storage/compute as required, database, middleware/application, ingress, then business validation.

### 1. Incident Detection

| Field | Runbook entry |
| --- | --- |
| Owner | NOC / on-call |
| Prerequisite | Independent monitoring and incident access |
| Operation | Correlate host, storage, WAN and external probes; record first failed user transaction and open incident. |
| Expected output / interpretation | TcpTestSucceeded reflects the tested path; cross-check a second vantage point. |
| Success | Confirmed outage scope/time, not one failed ICMP probe |
| Risk / rollback | Monitoring itself may be down; no changes to undo, preserve evidence. |

```powershell
Test-NetConnection erp.example.internal -Port 443
```

### 2. Incident Classification

| Field | Runbook entry |
| --- | --- |
| Owner | Incident lead + domain specialists |
| Prerequisite | Incident ID and independent console access |
| Operation | Separate power, network, storage, compute, security and database hypotheses using PDU/BMC, switch, array and application evidence. |
| Expected output / interpretation | Host state or an access error; a disconnected host alone does not identify the cause. |
| Success | Evidence-backed classification with security escalation if uncertain |
| Risk / rollback | Premature reboot destroys evidence; diagnostics only, stop intrusive investigation. |

```powershell
Get-VMHost | Select Name,ConnectionState,PowerState
```

### 3. Impact Assessment

| Field | Runbook entry |
| --- | --- |
| Owner | Service owners + operations |
| Prerequisite | Current dependency graph and BIA |
| Operation | Map affected user journeys and branch populations; calculate remaining RTO budget and candidate data points. |
| Expected output / interpretation | Expected service record or a DNS failure; compare independent resolver evidence. |
| Success | Critical dependency set and business impact agreed |
| Risk / rollback | Stale map misses middleware; keep assessment provisional and do not redirect users. |

```powershell
Resolve-DnsName erp.example.internal
```

### 4. Disaster Declaration

| Field | Runbook entry |
| --- | --- |
| Owner | Authorised incident commander + Business Owner |
| Prerequisite | Impact, repair estimate, RPO/RTO and security assessment |
| Operation | Record go/no-go, accepted data-loss exposure and authority; establish one writer/fencing decision before activation. |
| Expected output / interpretation | Timestamp captured in the decision record; command cannot grant authority. |
| Success | Signed/ticketed activation decision and fenced-source plan |
| Risk / rollback | Wrong declaration creates dual writers; cancel before promotion if evidence changes. |

```powershell
Get-Date -Format o
```

### 5. Communication

| Field | Runbook entry |
| --- | --- |
| Owner | Incident communications lead |
| Prerequisite | Approved contact tree outside failed site |
| Operation | Notify technical teams and owners through agreed channels; state scope, next update time and transaction restrictions. |
| Expected output / interpretation | Timestamp for status update and acknowledgements; notification itself is a human/process action. |
| Success | Acknowledged owners and an accessible status record |
| Risk / rollback | Unverified ETA misleads users; correct the record without erasing previous updates. |

```powershell
Get-Date -Format o
```

### 6. Recovery Activation

| Field | Runbook entry |
| --- | --- |
| Owner | DR coordinator + security |
| Prerequisite | Authorised decision, trusted access and fenced source |
| Operation | Load the approved runbook version, validate target capacity and credentials access; activate only its authorised recovery stages. |
| Expected output / interpretation | Expected witness/quorum model on DR management path; inspect actual membership separately. |
| Success | Single-writer protection and target readiness evidenced |
| Risk / rollback | Do not force quorum as a shortcut; stop and retain source isolation if readiness fails. |

```powershell
Get-ClusterQuorum
```

### 7. Infrastructure Recovery

| Field | Runbook entry |
| --- | --- |
| Owner | Network, identity and virtualisation owners |
| Prerequisite | Trusted recovery zone, power and admin path |
| Operation | Validate routing/firewall, NTP, AD/DNS/PKI, storage and compute in dependency order; use approved provisioning procedures. |
| Expected output / interpretation | DNS tests pass for the intended DR domain controller; inspect warnings and secure output. |
| Success | Base services reachable and trust/certificates valid |
| Risk / rollback | Replicated compromised AD can reintroduce attack; keep networks isolated, revert staged routes if approved. |

```powershell
dcdiag /test:DNS /s:dr-dc01
```

### 8. Database Recovery

| Field | Runbook entry |
| --- | --- |
| Owner | DBA + security for cyber incidents |
| Prerequisite | Valid clean recovery point, keys, storage and writer isolation |
| Operation | Use the selected approved promotion or restore path; record recovery point and consistency/application reconciliation evidence. |
| Expected output / interpretation | Required databases ONLINE after authorised recovery; ONLINE alone does not prove integrity or correct business data. |
| Success | DBA integrity acceptance and documented actual data exposure |
| Risk / rollback | Bad target can lose transactions; hold traffic and choose another isolated restore point, never overwrite evidence. |

```sql
SELECT name,state_desc FROM sys.databases;
```

### 9. Application Recovery

| Field | Runbook entry |
| --- | --- |
| Owner | Application/DevOps owner |
| Prerequisite | Accepted DB, secrets, queues, certificates and dependencies |
| Operation | Start approved versions in dependency waves, connect to DR endpoints, keep schedulers/payment workers controlled until reconciliation. |
| Expected output / interpretation | active for the actual unit; readiness and database connectivity checked separately. |
| Success | Health/readiness pass without unintended duplicate processing |
| Risk / rollback | Duplicate jobs can double-charge; stop affected workers and revert app configuration under the runbook. |

```bash
systemctl is-active erp-api
```

### 10. Validation / service restoration

| Field | Runbook entry |
| --- | --- |
| Owner | Business Owner + QA + operations |
| Prerequisite | Stable service, restricted test access and authorised synthetic account |
| Operation | Test branch login, read/write transaction, reports, totals and external integrations; release user traffic only after acceptance. |
| Expected output / interpretation | HTTP success plus documented readiness body; separately validate authorised business transaction. |
| Success | Owner signs tested journeys, load and actual RTA/RPA |
| Risk / rollback | Health 200 can hide broken finance; close traffic gate and restore previous safe routing, preserve accepted writes. |

```bash
curl --fail --silent --show-error --max-time 10 https://erp.example.internal/health/ready
```

### 11. Monitoring

| Field | Runbook entry |
| --- | --- |
| Owner | NOC + platform owners |
| Prerequisite | Service accepted and DR telemetry active |
| Operation | Watch latency/errors, queues, storage, security and backup protection; freeze unrelated changes and document deviations. |
| Expected output / interpretation | Recent logs without unexplained errors; do not publish sensitive log contents. |
| Success | Stable agreed observation window and working DR backup jobs |
| Risk / rollback | Silent backup gaps create new exposure; stop rollout, retain working DR service and escalate. |

```bash
journalctl -u erp-api --since "30 minutes ago" --no-pager
```

### 12. Failback

| Field | Runbook entry |
| --- | --- |
| Owner | Change manager + DBA + service owner |
| Prerequisite | Repaired trusted primary, current DR backup, reverse sync and approved window |
| Operation | Plan reverse data flow, reconcile deltas, validate synchronisation and switch only by the product-specific authorised procedure. |
| Expected output / interpretation | States fit the intended planned-failback design; synchronized target plus quorum/fencing and application checks still required. |
| Success | One authoritative writer, accepted service and resumed protection |
| Risk / rollback | Old primary is stale; before switch remain on DR, after new writes use planned reverse sync rather than DNS-only rollback. |

```sql
SELECT synchronization_state_desc,synchronization_health_desc FROM sys.dm_hadr_database_replica_states;
```

Command context: Test-NetConnection, Resolve-DnsName and Get-Date use Windows PowerShell 5.1/Windows Server 2022; Get-VMHost uses authenticated PowerCLI 13.x/vSphere 8; cluster commands need FailoverClusters module and read permissions; dcdiag uses AD DS tools. SQL DMVs target SQL 2022 and require appropriate server-state permissions (VIEW SERVER PERFORMANCE STATE for relevant 2022 DMVs). Linux commands target Ubuntu 24.04; erp-api and example.internal are illustrative unit/DNS names to replace from the approved inventory. Commands report evidence, not a universal pass/fail oracle.

## 12. Ransomware recovery starts with trust

![Compromised production is isolated; a clean immutable recovery point is validated in a clean environment before staged restoration](/assets/img/articles/content/high-availability-backup-disaster-recovery-enterprise-guide/ransomware.svg)

Assume Windows servers, a domain controller, VMware and SQL are compromised, with possible backup-console access. HA can restart encrypted VMs and replication can copy encrypted files or malicious logical changes. Neither knows the business intent of a write. An isolated repository with independent identity, restrictive management access and enforced retention reduces the shared attack path, but does not prove the stored operating system or credentials are clean.

First contain affected networks with the incident/security team, preserve forensic evidence and establish trusted administrative access. Build a timeline from endpoint, identity, network and backup audit evidence; the first encryption alert may be much later than initial access. Select candidate recovery points before the known compromise window, then scan and inspect them in isolation. An immutable point is protected from certain modifications, not automatically free of malware. Retain several candidates if the timeline is uncertain.

A clean recovery environment uses trusted installation media, patched hosts, independent administration, controlled DNS/NTP, restricted egress, EDR/logging and no automatic route back to compromised production. Recover identity/forest trust under a specialised approved plan; restoring a domain controller casually can restore the attacker’s access. Rotate affected secrets, tokens and certificates in a dependency-aware order, close the initial entry path, restore data/applications, validate integrity and business transactions, and reconnect in monitored stages with security and owner sign-off.

If validation fails, quarantine that recovery environment and preserve evidence; do not reconnect it while trying another restore point. Recovery may still be incomplete because of missing history, stolen data, unavailable keys or undetected persistence. Document residual risk and business decisions rather than guaranteeing full recovery.

[Reference: CISA #StopRansomware Guide (publisher blocks automated full-page access; verify the current guide before execution)](https://www.cisa.gov/stopransomware/ransomware-guide)

## 13. Monitor whether recovery remains possible

Zabbix can collect host, VMware, SQL and network signals; Prometheus collects instrumented metrics; Grafana visualises data from its sources. None replaces the backup platform’s job and restore evidence. Use an independent external probe and a recovery dashboard organised by business service: current availability, last valid backup, measured recoverable age, replication lag, DR readiness and latest restore-test result. Every actionable alert needs an owner, runbook and escalation path.

| Signal | Example threshold, not a universal default | Interpretation / action |
| --- | --- | --- |
| ESXi availability / VM restart events | Missing polls for 2 minutes; any unexpected HA restart | Confirm from another path; correlate power/network and user errors |
| Datastore space / growth | Warn below 20%, critical below 10% OR projected full before next response window | Use absolute free bytes, snapshot growth and response time; large arrays need different percentages |
| Storage latency | Sustained >20 ms for 10 min as an initial investigation trigger | Compare normal p95 and workload SLA; separate read/write and array/host views |
| AG sync / log send / redo | Unexpected NOT_HEALTHY immediately; queue age >half RPO warns, >RPO critical | Rate-aware estimate; redo lag is not identical to potential committed-data loss |
| Backup job / age | Any failed critical job; latest recoverable data age exceeds service RPO | Retry must finish before objective expires; job success without valid chain is insufficient |
| Repository / integrity | Forecast full within 14 days; any corruption or immutability-policy drift | Include locked data and full-chain dependencies; no emergency blind retention deletion |
| Restore test / DR readiness | Critical service test older than 30 days or failed; missing DR capacity | Track timestamp, tested point, load and owner acceptance, not a green backup icon |
| WAN / replication lag | Two independent probe failures; lag consumes the agreed data-loss budget | Check branch-to-DR path, route diversity and catch-up bandwidth |

Start from several weeks of normal and peak-period measurements, then tune persistence, hysteresis and maintenance windows. Avoid paging on every short latency spike. Treat missing telemetry as a separate failure. Queue KB divided by recent KB/s is only an estimate and becomes meaningless when the rate is zero or bursty. A one-minute RPO needs tighter telemetry and scheduling than a fifteen-minute polling cycle can substantiate.

### Example Zabbix 7.0 expressions

Create the numeric custom items backup.age.seconds and backup.job.failed on the illustrative host ERP-Backup first, using authenticated platform integration; do not expose a privileged script or credentials. Age must represent the newest independently restorable data point, not merely a job start time. Output is a TRUE problem condition when the expression holds. Recovery expressions/hysteresis and action routing must be configured separately.

```text
last(/ERP-Backup/backup.age.seconds)>900
last(/ERP-Backup/backup.job.failed)=1
nodata(/ERP-Backup/backup.age.seconds,10m)=1
```

[Source: Zabbix 7.0 trigger expressions](https://www.zabbix.com/documentation/7.0/en/manual/config/triggers/expression)

## 14. Read-only operational checks and their limits

Use authenticated sessions and least-privilege read access. These checks change no HA, datastore, AG or network configuration. Example names are deliberate lab identifiers. Treat permission errors as an access issue, not healthy output. Capture timestamps and inspect results with the dependency map. A diagnostic command has no configuration rollback; stop collection if it adds load or exposes sensitive output.

### VMware / PowerCLI 13.x, vSphere 8.0

```powershell
Connect-VIServer -Server vcsa.lab.example
Get-VMHost | Select-Object Name,ConnectionState,PowerState
Get-Datastore | Select-Object Name,State,FreeSpaceGB,CapacityGB
Get-VM | Get-Snapshot | Select-Object VM,Name,Created,SizeGB
Get-Cluster | Select-Object Name,HAEnabled,HAAdmissionControlEnabled,HAIsolationResponse
Get-VM | Select-Object Name,@{N='ConsolidationNeeded';E={$_.ExtensionData.Runtime.ConsolidationNeeded}}
```

Purpose/output: an interactive secure connection (do not disable certificate validation), host state, datastore capacity, snapshot inventory, HA/admission/isolation settings and consolidation flags. Expect Connected hosts, accessible datastores, policy-matching HA and an understood snapshot list; “no snapshots” is valid empty output. FreeSpaceGB must be compared with growth and array thin-provisioning capacity. HAEnabled=true alone proves neither N+1 nor VM protection. Query detailed per-VM overrides/events in the client before deciding recovery order.

### SQL Server 2022: AG health and backup history

```sql
SELECT ar.replica_server_name, drs.is_local, drs.database_id,
 drs.synchronization_state_desc, drs.synchronization_health_desc,
 drs.is_suspended, drs.log_send_queue_size, drs.log_send_rate,
 drs.redo_queue_size, drs.redo_rate, drs.last_hardened_time, drs.last_redone_time
FROM sys.dm_hadr_database_replica_states AS drs
JOIN sys.availability_replicas AS ar ON ar.replica_id=drs.replica_id;

SELECT TOP (30) database_name,type,backup_start_date,backup_finish_date,
 is_copy_only,has_backup_checksums,first_lsn,last_lsn
FROM msdb.dbo.backupset
WHERE database_name=N'ResilienceLab'
ORDER BY backup_finish_date DESC;

SELECT database_name,MAX(backup_finish_date) AS LastLogBackup
FROM msdb.dbo.backupset WHERE type='L' GROUP BY database_name;
SELECT name,state_desc,recovery_model_desc,log_reuse_wait_desc FROM sys.databases;
```

Purpose/output: one row per visible database/replica with queues in KB and rates in KB/s, recent D/I/L backups (Full/Differential/Log), latest log times and database state. Query the primary for wider AG visibility and each node’s msdb because backup history is local. An asynchronous replica can be SYNCHRONIZING without being faulty; interpret against its role. Empty history or old log timestamps need investigation; msdb records do not prove files still exist. ONLINE/FULL does not prove a valid backup chain. Permissions include relevant SQL 2022 DMV server-state access and authorised msdb access; avoid granting sysadmin solely for monitoring.

### Ubuntu 24.04 / systemd / GNU tools

```bash
df -hT
df -i
lsblk -o NAME,SIZE,FSTYPE,MOUNTPOINTS
systemctl is-active erp-api
journalctl -u erp-api --since "30 minutes ago" --no-pager
find /srv/backup -maxdepth 2 -type f -mmin -60 -printf '%TY-%Tm-%Td %TH:%TM %s %p\n'
ip route get 192.0.2.20
curl --fail --silent --show-error --max-time 10 https://erp.example.internal/health/ready
```

Purpose/output: filesystem byte/inode utilisation, mounted devices, actual service state/logs, recently modified backup files with size/time, selected route to an illustrative documentation address, and HTTPS readiness. Expect adequate bytes/inodes, intended mounts, active service, plausible files, the approved route and a valid readiness response. A fresh file is not a verified backup; calculate/compare trusted checksums or use the backup platform verifier and perform a restore. curl failure can be DNS, TLS, network or application; do not bypass TLS to make the check green. Use the real approved DR address in place of 192.0.2.20.

## 15. Six recovery exercises, increasing evidence and risk

| Test / goal | Prerequisites | Risk / boundary | Success and recorded evidence |
| --- | --- | --- | --- |
| Documentation Review: remove ambiguity | Current inventory, owners and architecture | No live changes; avoid exposing secrets in documents | Owners verify contacts, dependencies and command versions; dated findings and actions |
| Tabletop: exercise decisions | Incident scenario, facilitators and decision owners | Simulation only; assumptions can hide missing access | Timed decisions/declaration and communication log; named gaps |
| Isolated Restore: prove a recovery point | Isolated network, capacity, keys and copy of backup | Prevent routes, duplicate identities and outbound jobs | Restored data, integrity evidence and measured throughput; selected point recorded |
| Application Recovery: prove a user journey | Isolated dependency-complete stack and synthetic data/accounts | Do not send real invoices/emails/payments | Owner validates login, transaction and totals; RTA/RPA and defects recorded |
| Controlled Failover: prove ownership transfer | Lab/staging replica topology, approvals and fencing/rollback plan | Split-brain/data loss; no live production failover is authorised here | One writer, reconnection, load and failback verified; decision/evidence timeline |
| Full DR Exercise: prove service recovery | Isolated realistic recovery environment, multidisciplinary team and business script | Capacity/network collisions; production exercise requires separate executive/change authority | Critical service graph restored within signed targets, rollback/failback rehearsed; owner-signed report |

Each report should name the runbook version, backup point, environment, operators, start/end timestamps, actual data age, integrity and transaction evidence, excluded dependencies, load level, defects and follow-up owner/deadline. Repeat after material changes to identity, network, storage, database, application or backup policy. A tabletop pass is not evidence of restore throughput. Keep production exercises separate and explicitly authorised; the examples here use labs or isolated recovery environments.

## 16. Mistakes that make a green dashboard misleading

### VMware HA as backup

A restarted VM still has yesterday’s deleted file missing.

### Always On prevents all loss

A committed mistaken UPDATE reaches the secondary too.

### Backups on production storage

One array failure removes VMs and their only backups.

### No offsite copy

Building power/fire takes every local copy offline.

### No immutable/offline protection

A stolen backup admin can erase writable history.

### Never testing restore

Job success hides a missing encryption key.

### Undefined RPO/RTO

A nightly full is proposed for a service tolerating five minutes of loss.

### Insufficient DR capacity

VMs boot but ERP times out under branch load.

### No network configuration backup

Restored servers remain unreachable after firewall replacement.

### Ignoring DNS and AD

The API runs but nobody can authenticate.

### No dependency map

ERP starts before its queue and certificates are ready.

### No runbook

At 03:00 two admins promote different writers.

### No monitoring

Log backups have silently failed for a week.

### No failback plan

DNS returns users to the stale former primary.

### Trusting snapshots too much

A growing delta fills the same datastore it was supposed to protect.

## 17. Final decision matrix

| Organization Scenario | HA | Backup | DR | RPO/RTO Considerations |
| --- | --- | --- | --- | --- |
| Small business | Optional by impact | Independent + offsite | Documented restore target | Accept manual restore time explicitly |
| 20 VMs | N+1 compute if funded | VM + DB-aware | Offsite restore capacity | Measure ERP end-to-end |
| 50 VMs | Compute + DB | Isolated administration | Recovery waves | Per-service targets |
| 100 VMs | Multiple failure domains | Scheduled by IO/change rate | Tiered warm capacity | Concurrent restore throughput |
| SQL Always On | Supported quorum/sync design | Full/diff/log + keys | Remote role or restore | Separate infra and logical-loss targets |
| Two data centres | Local HA; deliberate site roles | Outside shared admin boundary | Tested failover and failback | WAN lag, fencing and DNS included |
| Kubernetes | Control + worker + dependencies | etcd/config + PV/app data | Independent recovery cluster | Pod start is not service recovery |
| 50 branches | Diverse WAN + core services | Core data + device configs | Branch-to-DR path | Local continuity and route convergence |
| Critical services | Failure-domain verified | Frequent validated points | Reserved, load-tested capacity | Business-signed measured evidence |
| Ransomware exposure | Availability only | Immutable/offline + separate identity | Clean-room workflow | Clean point discovery adds recovery time |

### If I were responsible for this infrastructure, what would I design?

Small Business: choose a maintainable hypervisor estate, independent application-aware backup, protected offsite/offline copies, a reachable spare restore target and tested identity/configuration recovery. Document the accepted outage and keep a deputy who can execute the restore. This is economical only when the business accepts manual recovery; it does not provide instant continuity or survive every simultaneous failure.

Medium Enterprise: use N+1 VMware HA with resilient storage, a justified SQL AG on separate hosts, an isolated backup plane and offsite immutable history. Assign recovery tiers, fund monitoring/on-call and a warm target for critical ERP dependencies. Test full-service recovery quarterly and selected restores more often. Costs include spare capacity, licences, repository growth and sustained DBA/platform ownership; adding an AG without those owners increases complexity faster than resilience.

Large Enterprise: design independent sites with verified WAN/identity/trust paths, local HA and explicit remote data durability, reserved critical-service capacity, a separately administered immutable/offline backup layer and a cyber clean-room option. Maintain versioned runbooks, service maps, exercises and a measured failback process. Active-active is justified only for applications that safely support it. Operational cost includes cross-team exercises, security/key lifecycle, capacity testing and 24-hour response. The final artefact is not a rack diagram: it is evidence of how much data may be lost and how an accepted business transaction returns.

## 18. Frequently asked questions

### Does VMware HA replace backup?

No. It restarts protected VMs after covered failures; independent historical copies are still required.

### Can Always On recover a deleted table?

A committed deletion normally reaches replicas. Restore a pre-deletion point in isolation and validate an authorised data repair.

### Are RPO and RTO measured results?

They are objectives. Record actual recoverable data age and elapsed business-service recovery time separately.

### Is an immutable backup necessarily clean?

No. Immutability restricts changes for a retention period; security validation of the stored recovery point is separate.

### Does a restarted Pod recover its database?

Only if its data, storage, topology and application recovery path remain usable; orchestration does not supply historical data recovery.

### How often should DR be tested?

Set a business-approved cadence and repeat after material changes. Combine frequent isolated restores with service-level exercises.

## 19. Official sources and related articles

Documentation was checked on 6 November 2026. Product documentation supports feature boundaries; sizing, scenarios and threshold values are engineering examples to validate locally. NIST SP 800-34 Rev.1 is a foundational contingency-planning publication, not a current product configuration guide. CISA full-page retrieval returned HTTP 403 and Broadcom TechDocs could not be retrieved; the Broadcom VMCP knowledge-base article was accessible. Verify blocked documents directly before operational use. No VMware/SQL/Kubernetes failure exercise or live production failover was performed during preparation.

[VMware vSphere 8 Availability: HA, FT, admission and isolation](https://techdocs.broadcom.com/us/en/vmware-cis/vsphere/vsphere/8-0/vsphere-availability-8-0.html)

[Broadcom: APD/PDL and VMCP boundaries](https://knowledge.broadcom.com/external/article/406100/best-practice-for-vsphere-high-availabil.html)

[Microsoft: AG failover modes](https://learn.microsoft.com/en-us/sql/database-engine/availability-groups/windows/failover-and-failover-modes-always-on-availability-groups?view=sql-server-ver16)

[Microsoft: SQL Server backup overview](https://learn.microsoft.com/en-us/sql/relational-databases/backup-restore/backup-overview-sql-server?view=sql-server-ver16)

[Microsoft: point-in-time restore](https://learn.microsoft.com/en-us/sql/relational-databases/backup-restore/restore-a-sql-server-database-to-a-point-in-time-full-recovery-model?view=sql-server-ver16)

[Microsoft: secondary backup support by version](https://learn.microsoft.com/en-us/sql/database-engine/availability-groups/windows/active-secondaries-backup-on-secondary-replicas-always-on-availability-groups?view=sql-server-ver16)

[Microsoft: DBCC CHECKDB and consistency](https://learn.microsoft.com/en-us/sql/t-sql/database-console-commands/dbcc-checkdb-transact-sql?view=sql-server-ver16)

[Microsoft: RESTORE VERIFYONLY limitations](https://learn.microsoft.com/en-us/sql/t-sql/statements/restore-statements-verifyonly-transact-sql?view=sql-server-ver16)

[Kubernetes: StatefulSet and storage limitations](https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/)

[Kubernetes: etcd backup/recovery](https://kubernetes.io/docs/tasks/administer-cluster/configure-upgrade-etcd/)

[Docker: Swarm quorum and recovery](https://docs.docker.com/engine/swarm/admin_guide/)

[NIST SP 800-34 Rev.1: contingency planning](https://csrc.nist.gov/pubs/sp/800/34/r1/upd1/final)

[CISA: #StopRansomware Guide](https://www.cisa.gov/stopransomware/ransomware-guide)

[Zabbix 7.0: trigger expressions](https://www.zabbix.com/documentation/7.0/en/manual/config/triggers/expression)

[Related: ESXi 8 installation and configuration](/articles/vmware-esxi-8-installation-basic-configuration)

[Related: vSphere networking](/articles/vsphere-standard-switch-vs-distributed-switch)

[Related: SQL Server backup jobs](/articles/sql-server-automatic-backup-job)

[Related: Veeam backup chains](/articles/veeam-forward-vs-reverse-incremental)

[Related: Docker Swarm and Kubernetes enterprise design](/articles/docker-swarm-vs-kubernetes-enterprise-guide)

[Related: Zabbix and agent monitoring](/articles/zabbix-server-linux-windows-agents-backup)

[Related: Grafana with Zabbix](/articles/grafana-installation-zabbix-integration)

[Download isolated SQL lab preparation (SQLCMD required)](/downloads/high-availability-backup-disaster-recovery-enterprise-guide/sql-lab-prepare.sql)

[Download isolated SQL lab restore (set captured target first)](/downloads/high-availability-backup-disaster-recovery-enterprise-guide/sql-lab-restore.sql)

[Complete Persian Markdown edition](/downloads/high-availability-backup-disaster-recovery-enterprise-guide/article.fa.md)

[Complete English Markdown edition](/downloads/high-availability-backup-disaster-recovery-enterprise-guide/article.en.md)