Install VMware ESXi inside VMware Workstation

Introduction

Running ESXi as a virtual machine in VMware Workstation is a Nested Virtualization lab scenario. Broadcom documents that nested ESXi is not supported for production on Workstation, but it can be used for learning, demonstrations, and testing.

Prerequisites

• A current VMware Workstation version compatible with the intended ESXi release; Broadcom lists Workstation 16.0 or later for ESXi 8.0 lab nesting.
• A supported 64-bit CPU with Intel VT-x/EPT or AMD-V/RVI enabled in firmware.
• Official ESXi ISO, sufficient RAM/disk for the host and lab workloads, and a planned management network.

01 — Prepare Nested Virtualization

1. Enable Intel VT-x or AMD-V in the physical host BIOS/UEFI, then fully restart the host if the firmware requires it.
2. In the Workstation VM settings, enable the processor virtualization option named Virtualize Intel VT-x/EPT or AMD-V/RVI.
3. If Workstation cannot obtain hardware virtualization, inspect host Hyper-V/VBS conflicts and follow the supported Broadcom guidance for that host configuration.

02 — Create the ESXi VM

1. Create a new virtual machine and select the matching VMware ESXi version as the guest type when available.
2. Attach the official ESXi ISO to the virtual CD/DVD drive.
3. Assign CPU, RAM, disk, and NIC resources for the intended lab. Start small for a basic host, but reserve enough capacity for the nested test VM that will run inside ESXi.
4. Choose the Workstation NAT, Bridged, or isolated network intentionally; the selected NIC must reach the management client according to the lab design.

03 — Install ESXi

Power on the VM and boot from the ISO. In the installer, accept the license agreement, select the virtual disk, choose the keyboard layout, set a strong root password, and confirm installation. Remove or disconnect the ISO after installation and reboot into ESXi.

04 — Configure Management Network

At the Direct Console User Interface (DCUI), press F2 and sign in as root. Configure the address information under this path:

F2 > Configure Management Network

Set the correct NIC, Static IP address, subnet mask, Default Gateway, DNS servers, and hostname for the lab. Apply the change and confirm the management network restart message.

05 — Test the Host Client and lab resources

1. Ping the ESXi management IP from the designated management client; then confirm gateway and DNS reachability from the DCUI test tools when available.
2. Open the Host Client using the management IP address:

https://ESXI-IP/

3. In the Host Client, verify the datastore is visible, create a small test VM, attach an appropriate ISO, and power it on to validate nested compute and network behavior.

Troubleshooting

• ESXi VM does not start: check BIOS/UEFI VT-x or AMD-V, Workstation processor virtualization settings, and host Hyper-V/VBS conflicts.
• Host Client is unreachable: verify the ESXi management IP, subnet, gateway, Workstation network mode, NIC connection state, and local firewall path.
• Test VM cannot run: check resource allocation on both the physical host and nested ESXi host, datastore free space, and the selected virtual-network connection.

Best Practices

• Keep the setup isolated from production and document the physical-to-Workstation-to-ESXi network mapping.
• Use a dedicated management subnet where practical, strong root credentials, and restricted management access.
• Treat snapshots as a lab convenience, not as a backup strategy; retain tested backups of important lab configurations.

Frequently Asked Questions

Conclusion

A nested ESXi host in Workstation is valuable for a controlled lab: enable virtualization support, create the VM deliberately, install from the official ISO, configure the management network, then validate Host Client, datastore, and a small test VM. Keep it outside production.

Official references

• Broadcom Knowledge Base — Nested ESXi support
• Broadcom Knowledge Base — Workstation nested virtualization
• Broadcom Knowledge Base — Enable virtualization technology