Why shared storage matters

Consider three HPE ProLiant DL380 hosts, each with two Xeon processors, 256 GB RAM, enterprise SSDs and multiple NICs. ESXi runs VMs on each host’s local datastore. When Host A loses power, Hosts B and C can still run their own VMs, but they cannot open the VMDKs stranded on A. Enabling vSphere HA alone cannot restart those VMs. The survivor needs access to their files, enough CPU and memory, and a correctly configured HA cluster.

Compute-only vMotion moves execution while both hosts retain access to the same datastore. Storage vMotion relocates files, and a supported combined compute-and-storage migration can move a running VM between hosts without shared storage. These are planned moves that need a working source. They cannot retrieve files from a powered-off server.

Broadcom guidance on local storage and shared-nothing migration

Meet AJ guide to ESXi 8 installation

What is VMware vSAN?

Software-defined storage moves placement and protection decisions into software. Distributed storage allocates data and redundancy across nodes. A storage cluster is the group of hosts coordinating those resources. A shared datastore is the logical storage location multiple hosts can open. In an HCI deployment, the same ESXi hosts supply compute and participate in storage. vSAN integrates that storage with vSphere and exposes it to workloads as a datastore managed through vCenter.

A local VMFS datastore is normally accessible only from its owner. An FC or iSCSI array presents shared block storage, commonly formatted as VMFS, while an NFS server exports a shared filesystem. A vSAN datastore provides shared access by using eligible devices in the host cluster and communicating over the vSAN network. An existing local datastore needs a migration plan before its disks can be repurposed.

VMware vSAN technology overview

Traditional SAN and vSAN

With external storage, three hosts reach an array over redundant FC fabrics or an iSCSI Ethernet design. An NFS appliance can serve the same shared-datastore objective through a file protocol. The array owns media protection, controller failover and storage expansion. With vSAN, certified devices inside the hosts provide capacity and storage policy places redundancy across failure domains. A host outage can remove both compute and storage capacity at once.

External storage can grow separately from compute and can serve several clusters if the array and host design support it. HCI can remove an external fabric or array from a suitable deployment, but couples lifecycle, network and storage changes more tightly. Compare the blast radius of losing an array, fabric, switch pair, host or rack, and include the migration and renewal costs before choosing.

Three hosts reach an external array on the left. On the right, the hosts contribute their own media to vSAN.
Three hosts reach an external array on the left. On the right, the hosts contribute their own media to vSAN.

Objects, components and placement

vSAN stores a VM through objects such as its namespace and virtual disks. An object is laid out as components according to policy. Depending on the layout, components hold mirrored data, data and parity fragments, or witness votes. Cluster membership tracks participating hosts, and distributed metadata helps locate components and determine object state. vCenter presents and manages the configuration while ESXi hosts carry the storage I/O.

For a simplified FTT=1 RAID-1 VMDK, A and B hold data copies while C hosts a witness component for that object. The VM can execute on C, even though its VMDK data lives on A and B. This example applies to one object. Other disks of the same VM may have different placements or policies. Resynchronization brings stale or rebuilt components back in line when connectivity returns or when a supported placement is available.

One VMDK has data copies on Hosts A and B and a witness component on C. Execution can be on any eligible host.
One VMDK has data copies on Hosts A and B and a witness component on C. Execution can be on any eligible host.

OSA versus ESA

OSA groups devices into disk groups. Each group has one flash cache device and one or more capacity devices. Hybrid OSA uses magnetic capacity drives, whereas all-flash OSA uses flash capacity devices. In all-flash OSA the cache tier buffers writes. Losing the cache device can take its whole disk group offline, so disk-group size matters to failure impact and remediation.

ESA uses a per-host storage pool of qualified NVMe devices without a dedicated cache device. Its redesigned write path and log-structured filesystem are built for modern flash and efficient erasure-coded writes. A device outage does not inherently remove an entire OSA-style group. ESA compression operates higher in the stack. OSA all-flash data reduction and ESA data services differ by release, so confirm deduplication, compression and activation scope for the target build. Recent ESA features must be checked against the actual deployment version.

Data-at-rest encryption protects stored media and introduces key-provider recovery dependencies. Data-in-transit encryption protects vSAN traffic between participants. Record the supported key provider, escrow and recovery procedure where applicable, and include CPU and latency effects in testing. Check vSAN 8 and VCF 9.x behavior independently before planning an upgrade or a change of architecture.

In ESA releases before VCF 9.1, SPBM could control compression. VMware documents compression as always on for ESA in VCF 9.1. ESA global deduplication was limited in 9.0 and became generally available in VCF 9.1 for eligible clusters. Two-node and stretched configurations are excluded from that 9.1 capability. OSA all-flash deduplication works within a disk group.

VMware ESA data services in VCF 9.1

Broadcom data-in-transit encryption behavior

Broadcom ESA data-at-rest encryption and key provider

OSA uses a flash cache device per disk group. ESA pools qualified NVMe devices without a dedicated cache tier.
OSA uses a flash cache device per disk group. ESA pools qualified NVMe devices without a dedicated cache tier.

VMware vSAN FAQ and architecture-specific capabilities

VMware data reduction and space efficiency

Storage policy and FTT

Storage Policy-Based Management (SPBM) expresses requirements per VM or virtual disk. Failures to Tolerate (FTT) specifies how many qualifying failures the layout is built to tolerate. RAID-1 mirrors data. RAID-5 uses single parity and RAID-6 uses dual parity. Standard OSA RAID-5 needs four eligible fault domains and RAID-6 needs six. ESA in vSAN 8 can apply adaptive RAID-5 to three hosts with 2+1, or 4+1 at six or more hosts under its documented behavior. VCF 9.1 Auto-RAID changes automatic selection. Validate the actual policy, topology and release before using a layout for sizing.

For a critical SQL VM, apply an FTT=1 policy to the namespace and all critical virtual disks as appropriate, then verify compliance and placement. Object Space Reservation reserves capacity relative to logical object size and competes with other reservations. Leave disk stripes per object at default unless testing and version guidance justify a change. A policy change can build new components and temporarily need extra capacity and network bandwidth. Compliance shows whether objects meet policy. SQL failover and backup recovery need separate validation.

VMware policy recommendations and VCF 9.1 Auto-RAID

What happens when a host fails?

In the three-host RAID-1 example, suppose Host A runs the SQL VM and loses power. A’s data component goes absent. For the illustrated VMDK, B’s surviving data and C’s witness still provide data and quorum in the healthy partition. If all required VM objects are accessible, HA detects and evaluates the failure and can power the VM on at an eligible host with resources. SQL still has to complete crash recovery and clients must reconnect. The VM that was executing on A experiences an interruption even if the VMDK remains accessible.

Three fault domains are enough for this layout during normal operation but two survivors cannot restore the same three-domain placement on their own. After A returns, stale data may resync. If A is lost permanently, restoring compliance requires a suitable replacement or spare domain and capacity. A transient absence may wait for the configured object repair timer, 60 minutes by default in the documented releases. This timer governs repair and need not delay an otherwise eligible HA restart. A device recorded as failed can trigger repair differently. Follow object state and resync progress to distinguish accessible with reduced protection from inaccessible.

A failed disk can affect multiple components, and on OSA a cache-device failure can remove a disk group. A network partition can make live hosts unable to reach data or votes. Accessibility requires a valid quorum and enough data components to serve the object. If B also drops before protection is restored, C’s witness cannot serve the disk. A second fault can make some objects inaccessible while other objects remain available.

Host A and its data copy are absent. B and C preserve this object’s accessibility. HA restart and SQL recovery follow if the required objects and resources are available.
Host A and its data copy are absent. B and C preserve this object’s accessibility. HA restart and SQL recovery follow if the required objects and resources are available.

VMware availability and spare-domain guidance

Broadcom repair delay and resync diagnostics

Does vSAN replace vSphere HA?

vSAN protects storage objects according to their policies. vSphere HA restarts affected VMs under its applicable failure response, placement rules and available resources. Admission control reserves failover capacity according to configuration. Application recovery follows its own process, including service startup, database recovery and client reconnection. Measure storage accessibility, guest restart and application acceptance as distinct milestones in the recovery timeline.

With vSAN enabled, HA inter-agent heartbeats use the vSAN network. A partition can therefore affect both storage and HA. Review isolation response, restart priorities and admission control, then test recovery paths rather than relying only on a green compliance indicator.

Broadcom HA heartbeats over the vSAN network

vSAN supplies accessible objects, HA starts the guest, and the application must pass its own recovery checks.
vSAN supplies accessible objects, HA starts the guest, and the application must pass its own recovery checks.

Meet AJ enterprise HA, backup and disaster recovery guide

Designing the vSAN network

In a Cisco environment, use VLAN 99 for management, 180 for guests and 190 for vSAN as an example. Enable vSAN on its own VMkernel port and verify the VLAN, trunks, IP addresses and routing. Two uplinks to independent switches protect against a path loss. Check shared power, switch stacks and inter-switch links as possible common failure domains. VLAN separation simplifies diagnosis but physical links can still be congested.

The design guide specifies 10 GbE minimum for all-flash OSA. ESA supports 10 GbE only on eligible entry profiles, and VMware recommends 25 GbE or faster for typical designs. The actual profile determines the requirement. Size for writes and resync on the surviving path. Use vDS Network I/O Control where supported. LACP requires a matching supported upstream configuration, and a single flow does not get the sum of the link rates.

Jumbo frames are optional. If used, match MTU across the entire path. Check latency, drops, CRC errors and queue congestion under load. A host can be up with working management traffic while its vSAN path is broken and objects become inaccessible.

Each ESXi host has two physical paths and a VMkernel port tagged for vSAN traffic.
Each ESXi host has two physical paths and a VMkernel port tagged for vSAN traffic.

VMware network design and ReadyNode profiles

Meet AJ guide to virtual switches

Storage performance under load

IOPS counts operations and throughput counts bytes per second. A SQL log is sensitive to synchronous write latency, Windows patching can bunch random I/O, a Linux application may flush writes periodically, and monitoring systems can spike during retention compaction. Record read and write latency separately, including tail percentiles, alongside block size, read/write mixture and queue depth.

SSD sustained writes, NVMe parallelism, CPU, network latency, RAID processing and contention during resync can all matter. Compression and supported deduplication depend on the data and release. Benchmark representative working sets and compressibility after warm-up. Repeat during resync, with a failed uplink and at planned occupancy. Record the build, policy and latency distribution so the result is reproducible.

Broadcom guidance on resync contention

Hardware qualification

HPE DL380 Gen10 and Dell PowerEdge are product families. Check the exact CPU, backplane, controller model and mode, device ID and endurance, NIC, firmware and drivers in the Broadcom Compatibility Guide and the architecture-specific ReadyNode profile. General ESXi certification does not qualify the vSAN storage stack. OSA controller passthrough or approved RAID mode depends on the compatibility entry. ESA needs a qualified platform and devices.

Desktop SSDs with similar peak throughput can have weaker sustained writes, endurance or power-loss protection. Document a supported spare and verify firmware and driver together before every lifecycle change. Check OEM support separately from software entitlement.

Broadcom Compatibility Guide

Broadcom CPU and ReadyNode lifecycle notes

Two-node, standard and stretched clusters

Two data hosts and a separately hosted Witness Appliance can serve a branch. The witness holds voting components and metadata without VM payload. If one host fails, the other needs its data and appropriate quorum. Loss of the witness alone may leave both data hosts serving I/O with reduced tolerance for a subsequent fault. Put the witness in an independent failure domain and validate its release and network thresholds.

A standard cluster starts at three data hosts. Larger deployments can group hosts by rack or power domains. Stretched clusters mirror across two data sites with a site protection policy and place a witness in a third domain. The documented data-site limit is 5 ms RTT. Witness thresholds vary by scale and topology. Verify local protection separately and size the surviving site for failover load.

Two data hosts exchange data and independently reach a witness that holds votes.
Two data hosts exchange data and independently reach a witness that holds votes.
Two sites mirror protected data. A third failure domain hosts the witness.
Two sites mirror protected data. A third failure domain hosts the witness.

Broadcom cluster deployment models

Broadcom witness latency and connectivity guidance

Capacity and rebuild headroom

As a hypothetical example, 12 TiB of claimed capacity theoretically holds 6 TiB of data with two copies or 8 TiB with 2+1 erasure coding. A 4+2 layout also yields about 8 TiB in simple arithmetic, but requires six domains and cannot fit a three-host cluster. Subtract metadata, temporary data, reservations, imbalance, growth and operational headroom. These ratios do not predict exact usable space.

Check Operations Reserve and Host Rebuild Reserve where supported. Model maintenance and permanent loss at the largest fault-domain boundary. Free space on two remaining hosts cannot meet a three-domain placement rule. Estimate how much capacity and time it takes to re-protect objects before promising recovery windows.

Each block represents a data or protection unit in mirror, 2+1 and 4+2 layouts.
Each block represents a data or protection unit in mirror, 2+1 and 4+2 layouts.

Local RAID and host resilience

Hardware RAID presents a protected logical device inside a server. The local datastore is still tied to that host and controller. vSAN policies place components across hosts or fault domains. Use the controller mode certified for the architecture and exact hardware. Treat boot device protection as a separate decision.

vSAN, SAN and NAS compared

These are common designs. Confirm the capabilities and interoperability of the specific products before using the table for procurement.

Decision area vSAN Block SAN NAS
Architecture Distributed objects Array presents LUNs Array exports files
Access vSAN datastore FC or iSCSI VMFS Shared NFS
Hardware Qualified hosts, devices and Ethernet Array, adapters and fabric Supported NFS and LAN
Scale Add devices or hosts Grow array independently Add shelves or nodes
Performance Policy, peers, CPU and resync Controllers, queues and fabric Protocol, cache and LAN
Failure boundary Host, rack, network and quorum Array, controllers and paths Head, export and Ethernet paths
Operations vSphere and shared lifecycle Zoning and multipath Exports and access controls
Licensing Bundle and TiB entitlement Array features and support Appliance and feature terms
Backup VM or application integration VM backup and optional array integration VM backup and optional NAS integration
Monitoring vCenter and cluster telemetry Fabric and array counters NFS, NIC and array metrics
Enterprise support Bundle, OEM and certified BOM Host, fabric and array vendors Host, Ethernet and NAS vendors

Is vSAN suitable for SQL Server?

Size the log for synchronous commit latency, data files for random I/O and tempdb and backups for peak concurrency. Measure SQL waits, transaction latency and guest I/O during normal operation and reduced redundancy. Always On Availability Groups can provide database-level failover with appropriate SQL and cluster design. Place replicas in separate failure domains if the recovery objective demands it. Keep application-consistent backups and a tested log chain for point-in-time restores. Storage redundancy alone gives no historical recovery point.

Microsoft Always On Availability Groups

Microsoft SQL Server recovery models

Meet AJ SQL Server backup jobs

Backup and disaster recovery

If ransomware encrypts a guest or an administrator deletes it, distributed storage can faithfully commit the harmful change. Keep independent recovery copies with isolated credentials, immutable retention where appropriate and an offsite copy. Use supported image-level integration and add application-aware or native backups for transaction consistency. Test full VM and granular restores in isolation.

RPO is the acceptable data loss window and RTO is the acceptable recovery time. Rehearse recovery of routes, identity, key providers and dependencies after site loss. Stretched storage can protect against an eligible site failure. Historical recovery needs an independently designed backup or DR workflow.

Broadcom vSphere backup APIs

Monitoring and first-response diagnostics

In vCenter, start with Cluster, Monitor, vSAN. Review Skyline Health or the version’s health view, Virtual Objects, Resyncing Objects and Performance Service. Track device health, capacity by host and disk, object accessibility and compliance, latency, network errors and resync progress. Use supported vSAN and vCenter APIs where available. Zabbix can correlate switch and host signals and Grafana can visualize them. Verify each metric’s coverage.

Substitute the actual names and IPv4 peer address below. Look for the vSAN traffic tag and agreement on cluster membership. If small pings work but an unfragmented large ping fails, check MTU and the path in both directions. The example uses MTU 1500. For MTU 9000 use payload 8972. Correlate NIC counters, object state and resync samples with the incident.

CODE
esxcli vsan network list
esxcli vsan cluster get
vmkping -I vmk2 -d -s 1472 192.0.2.12
esxcli network nic stats get -n vmnic2
esxcli vsan debug object health summary get
esxcli vsan debug resync list

Broadcom VMkernel and MTU diagnostics

Broadcom ESXCLI commands and repair timing

Meet AJ Grafana and Zabbix guide

Subscription and total cost

Broadcom KB 313548 documents a minimum of 16 licensed physical cores per CPU, with 1 TiB of vSAN entitlement per VCF core or 0.25 TiB per VVF core, rounded up to the next TiB for VVF. Compare bundled entitlement with contributed raw capacity, excluding OSA cache, and size the add-on for any shortfall. Check the written SKU and contract. Version 9 and later use VCF Operations and Business Services for subscription licensing.

Include certified devices, network upgrades, spares, support, capacity add-ons and staff time. Compare with external storage support, controllers or shelves, fabrics and migration. For the two-socket DL380 example, count cores even if utilization is low. Model storage-led growth separately from compute-led growth.

Broadcom core and TiB counting

Broadcom version 9+ licensing workflow

Broadcom licensing capacity differences in OSA and ESA

Where vSAN fits

A. One ESXi server

A single host lacks another compute location for failover. Local RAID plus independent recovery capacity is more directly relevant until the organization funds a multi-host design.

B. Two branch hosts

Two-node vSAN is viable when each host can carry the required failover load and an independent witness is reachable. Test data-host failure and witness isolation separately.

C. Three HPE hosts and 60 VMs

Size working sets, peak I/O and compute, then qualify the exact configuration. If compliance must return during a sustained outage, model an additional fault domain. For the illustrated FTT=1 RAID-1 layout, a fourth eligible host and enough rebuild space can make reprotection possible without waiting for the failed host to return.

D. Existing high-performance Fibre Channel SAN

If current storage meets latency and recovery objectives, replacement needs a measurable benefit such as retiring an expensive fabric. Test cutover and total cost on a pilot cluster first.

E. New HCI deployment

Qualify the target ReadyNode profile before purchase. Plan rack, power, network, backups and failure capacity as one architecture. Verify maintenance and restore before admitting critical workloads.

Operational strengths and constraints

vSphere management and object policies put compute and storage in one workflow. A suitable design can retire an external array. The same team now owns compatibility, fabric contention, storage entitlement, redundancy overhead and repair placement. Plan support handoffs across the network, hypervisor and application teams.

Common production design mistakes

Unsupported media or firmware can complicate support and replacement. A single switch, undersized links or inconsistent MTU can break object access. Inadequate space or poorly chosen policies can keep objects degraded through the next outage. Audit real rack and power boundaries. Assign ownership for health alerts and capacity. Before production, test host loss at load, resync, network failover and restore from an independent backup.

Engineering verdict

vSAN turns qualified local devices into policy-governed shared storage. Pilot it when local disks strand VM files or when consolidating a new platform could meet availability and growth goals. Qualification, resilient networking, a supported topology, rebuild headroom, backup and a written subscription belong in the budget.

For the three-host example, compare both proposals with the same workload and recovery criteria. Measure service recovery with one host down and verify when objects regain protection. Keep the external array when independent scaling, proven latency or existing integrations deliver more value than consolidation.

Meet AJ enterprise virtualization comparison

Frequently asked questions

Can I reuse the disks already in my ESXi hosts?

Use only devices and configurations qualified for the target release and architecture. Move existing data elsewhere before reclaiming a device for vSAN.

Can a three-host cluster rebuild after one host is gone?

A surviving mirror and witness can serve the illustrated FTT=1 object. Full reprotection for this layout needs the failed domain to return or another eligible domain with sufficient capacity.

Will vSAN keep a SQL VM running during host power loss?

When a host loses power its running VM stops. If its objects stay accessible and HA can place it, the VM starts elsewhere. Database recovery and client reconnection follow.

Does a vSAN witness add usable datastore capacity?

A witness supplies voting components and related metadata for quorum. It contributes no space for VM payload.

Can I recover a deleted VM from a vSAN mirror?

A mirror protects the current storage state from eligible failures. Restore a deleted VM from an independent retained recovery copy and confirm the application is consistent.

Official references

VMware vSAN technology overview

VMware vSAN design guide

VMware vSAN availability technologies

VMware vSAN operations guide

Broadcom Compatibility Guide

Broadcom core and vSAN capacity licensing

Share

Meet AJInfrastructure & DevOps
Loading…