Install Nested ESXi in Workstation and Diagnose VMCISr

What nested ESXi is useful for

Without a spare ESXi host, you can run ESXi inside Workstation; this is nested ESXi. This tutorial prepares a lab for virtual-network and API practice and checks that an inner VM can boot. Success does not establish production hardware support or performance. A VMCISr failure must be investigated in that VM's own log.

CPU, host and compatible builds

Use ESXi 8.x and Workstation compatible with the host OS. Record exact versions and the full error. Check applicability before borrowing another build’s workaround.

The example host has 32 GiB RAM, SSD and VT-x/EPT or AMD-V/RVI. Allocate four vCPUs, 8 GiB RAM and suitable boot/datastore disks to ESXi. These allocations are not speed guarantees.

Enable virtualization in host firmware and expose the required guest features through Workstation. Keep administrator/console access and a valid image.

Nested support limitations

This workflow is for a lab. Check Broadcom’s nested-support conditions. VMCISr names the failing module rather than one universal cause across versions.

Two virtualization layers

There are two layers: Workstation exposes host CPU features to ESXi, and ESXi uses them for its own VMs. Opening the ESXi interface is therefore not the final check. Boot a small inner VM to establish whether nested virtualization is actually available to it.

Example: rehearse a vSwitch change

Build one ESXi host with two small nested VMs. Keep management host-only and provide required outbound access through NAT. This keeps test traffic out of the company LAN.

Host security during workarounds

Use lab-specific credentials and keep management off the Internet. When a KB calls for Hyper-V, VBS or Secure Boot changes, check its build scope and security impact. Disabling host controls is not a general nested-error fix.

Build the ESXi VM and test Host Client

Enable host-firmware virtualization and select Virtualize Intel VT-x/EPT or AMD-V/RVI in VM CPU settings. Record Workstation and host-OS versions.

Select the ESXi guest type and image-compatible adapter. Attach a valid ISO, verify the dedicated target disk, then install ESXi.

After installation, press F2 in the DCUI and open Configure Management Network to set the IP, DNS and hostname. The address 10.99.0.11/24 is a host-only example and must match your VMnet. Run Test Management Network before moving to the browser.

Test TCP/443 from the workstation with this command.

POWERSHELL

Test-NetConnection -ComputerName 10.99.0.11 -Port 443

TcpTestSucceeded: True establishes only TCP connectivity. Open the Host Client, compare the lab certificate fingerprint through the console, and boot a small VM inside ESXi. Check that VM's networking and storage as well; these results go beyond an open port 443.

Host and guest resource pressure

Inspect host RAM, ESXi swapping and disk latency. Slow lab guests might reflect host oversubscription. Record results with the allocated resources.

Investigate VMCISr and VT-x failures

Follow the reported failure. For Module VMCISr power on failed, inspect the VM's vmware.log, VMCI setting and the KB for that build; removing vmci0 is not a universal fix. For VT-x unavailable, check firmware, the nested option and the host virtualization engine. A missing NIC requires checking adapter/image compatibility, not injecting an unknown driver into the ISO.

Collect the exact power-on failure on the Windows host

Replace the path with this powered-off VM’s actual folder. Read its log before editing VMX. The block is Windows PowerShell, runs on the Workstation host and makes no VM configuration change. A missing vmware.log calls for checking the actual VM folder and most recent power-on attempt.

POWERSHELL

$VmFolder = 'D:\LabVMs\NestedESXi'
Get-ChildItem -LiteralPath $VmFolder -Filter 'vmware*.log' | Select-Object Name,Length,LastWriteTime
Get-Content -LiteralPath (Join-Path $VmFolder 'vmware.log') -Tail 100
Select-String -LiteralPath (Join-Path $VmFolder 'vmware.log') -Pattern 'VMCISr','VMCI','VT-x','AMD-V','MonitorMode'

Expected: timestamped context around the actual failed module or CPU virtualization limitation. No match is not proof of support. Match the exact host, Workstation and ESXi builds to the vendor KB; preserve a powered-off VMX copy before any documented workaround and undo only that change if it fails. Never blindly delete VMCI entries or disable host security controls.

Restore VMX and lab data

Keep a copy of the powered-off VM and VMX. Undo VMX edits only while powered off, using the recorded copy. Application data inside nested guests needs in-guest backups too. Lab snapshots do not replace them.

Record reproducible failures

Record the error, host/Workstation/ESXi builds and CPU settings together. Apply each workaround independently and repeat power-on and nested-guest boot tests.

Best Practices

Version-control configuration and review changes.
Use least privilege and explicit allow-lists.
Automate repeatable checks and monitor the expected state.
Test upgrades and restores before production rollout.

Conclusion

A reliable implementation of Install Nested ESXi in Workstation and Diagnose VMCISr is more than a successful first run. Keep the configuration documented, observable, recoverable, and aligned with the team's change-management process.

Install Nested ESXi in Workstation and Diagnose VMCISr

What nested ESXi is useful for