VCF 9.0.x Ultimate Deployment Guide

In order to install ESXi you’ll need the ESXi ISO from Broadcom, you can find this under VMware Cloud Foundation

Expanding VMware Cloud Foundation 9 and select the release you want to deploy, I am deploying 9.0.0.0 so I clicked that release

Click View Group on VMware ESX

Accept the T&Cs, you need to click the links, then the box can be checked

And Download the ISO Image

Mount this to the server, by using rufus to create a bootable USB, or by mounting it to your servers virtual CD ROM in the IPMI, iDRAC for Dell and iLO for HPE

Once the server has booted ESXi, you’ll have this screen, select enter to continue

Accept the EULA with F11

It will then scan for bootable devices, for a production system this should be something in RAID 1
Examples are Dell’s BOSS card
For HPE G11 you should have the NS204i-U, or for G10 systems the NS204i-P, which is a PCIe card

As this is a lab, I have a virtual disk, and will be using the 128GB one by making sure its highlighted in Yellow and clicking Enter to Continue

Select your keyboard layout and hit Enter

Set a root password, use something easy to use, we can set a secure random one later

You may get a CPU warning depending on your hardware, I have a 3rd Gen Epyc CPU which flagged this, it is supported but may not be in a future release
Press Enter

Then click F11 to install

Once thats done, reboot the server when prompted and unmount your media

When the host boots, it should look like this, press F2 to login

Enter the root credentials and press enter

Scroll to Configure Management Network and press enter

Press Enter on Network Adapters and ensure that a connected Nic is selected, these should all be configured the same on the switch

In my case VMNIC0 is connected, and I will be using this for management, so I will press Escape and leave it as it is

Press enter on VLAN

And enter your ESXi management VLAN, I am using VLAN 1023
This is only needed if you have your VLANs trunked down, if your management VLAN is the native VLAN you can ignore this, as all my VLANs are trunked down, I am entering mine

On IPv4

Use the space bar to select the third option to set a static IP and add your management IP details in and press enter to Save

For IPv6, select disable on the first option, unless you are specifically using it, and press Enter

Add your DNS servers and the hostname for this server and press Enter

Add your domain under DNS Suffixes and press Enter

Now press Escape and enter Y to apply changes and reboot the host

Then, login on the WebUI at
https://fqdn
And login with the root credentials

Navigate to Manage/Security & Users/Users on the left, and click the root account and click Edit User to change the root password to something more secure

Then add the password
This needs to be 15 characters with the only allowed special characters being !@#$%^&*

Then we need to enable SSH, click Manage/Services then click TSM-SSH and click Start

Now we need to SSH into the host with the root credentials, using something like Putty and run the following to set the hostname/FQDN correctly for the certificate and renew it for the VCF deployment wizard

For my host, lab-vcf9-esx01, lets set the hostname with

esxcli system hostname set -H=<hostname>

So for my host this is

esxcli system hostname set -H=lab-vcf9-esx01

Then set the FQDN with

esxcli system hostname set -f=<fqdn>

Which for my host is

esxcli system hostname set -f=lab-vcf9-esx01.leaha.co.uk

Now renew the certificates with

/sbin/generate-certificates

And reboot the host with

reboot

When it comes back up, you will need to restart SSH for the Installer

Lastly, we need to setup NTP on all servers, you can use a windows App, DC or a docker container, for this

Head to Manager/System/Time & Date, and click Edit NTP Settings

Select the second radio button to use a NTP server, select the service to start and stop with the host, and put the IP address for your NTP server and click save

Then, under services, click ntpd, and click start

We will need to repeat this on the remaining hosts

Before we proceed we need to check we have enough space on the first host to deploy the VCF installer appliance, your boot device will typically be 512GB or larger so there should be a good size local datastore created we can use for this as we cant use our vSAN disks

We can check this under Storage/Datastores
Mine is only 128GB but should be enough

If you are doing this as a nested lab for learning purposed, I would suggest doing section 2.14.2 here, before going through the installer

We first need the appliance OVF, this can find this under VMware Cloud Foundation

Expanding VMware Cloud Foundation 9 and select the release you want to deploy, I am deploying 9.0.0.0 so I clicked that release

Click View Group on the VMware Cloud Foundation Installer

Accept the T&Cs, you need to click the links, then the box can be checked

And download the appliance

Log into the first host and click Networking
If you set a VLAN for the management VLAN and the management components are going on the same VLAN, which I recommend, we will need to edit the VM Network to set this VLAN, click the VM Network, then Edit Settings

Set the VLAN tag and click Save

Now click Virtual Machines/Create / Register VM

Click the second option to deploy a virtual machine from an OVF or OVA file and click Next

Enter a name, this will become the SDDC Manager later so name it for that, then click the box in the middle

Find and double click the SDDC Manager appliance

Then click Next

Select the install datastore which is create when ESX is installed, this uses leftover space on the boot device, and click Next

Click I Agree and then Next

The network mapping should be on VM Network, provisioning needs to be thin, and the box to power the VM on automatically should be checked, then click Next

Under Application enter a root and local user password, these need to be 15 characters with the only special characters being !@#$%^&*
For the hostname enter the FQDN and for NTP add your NTP server

For the networking section, enter the SDDC Manager IP address, subnet mask, gateway, DNS domain and search domain path and DNS servers, comma separated, then click Next

And click Finish
Do not refresh your page while this is deploying

Now we have the appliance deployed we need to download all the software binaries, log into the VCF Installer on
https://fqdn

Then click Depot Settings And Binary Management

On the Connect To The Online Depot widget, click Configure

Enter your download token, you may need to enable the proxy server if you have one in your environment, and click Authenticate

At the bottom, select your release version, I am doing 9.0.0.0, and select everything apart from the SDDC Manager and click Download

When its all done, it should look like this

We can then click Return Home at the top left to get back to the main menu

As we proceed through we will need to add various appliances, ensure all are DNS registered as you go through it before you finish the deployment

On the Deploy Widget, click Deployment Wizard/VMware Cloud Foundation

As this is a brand new install, select Deploy New VCF Fleet and click Continue
You would use the other option if you have a VCF fleet/instance, and want a separate SDDC manager to add to an Operations instance

We dont have any existing components, so click Next

Now we need to select our version, as I downloaded 9.0.0.0, thats what I selected, then enter a name for the VCF9 Instance, you can use the SDDC Manager hostname, or a different name, enter a management domain name, you can use a name, like I have, for whats being run in this domain, or the vCenter name
We can leave the advanced check box unchecked

For deployment model, simple works best for most environments, the only thing you likely want as a multi node setup is NSX, but we can set this later
Then add your domain name, DNS servers, comma separated and NTP servers, also comma separated
Password create I am opting to do manually

Then click Next

For VCF Operations, here is the sizing chart so you can line it up for your environment, Extra Small should not be used in production

We need to select our Operations node size, then add the FQDN, ensure this is registered in DNS, then add an admin and root passwords meeting the requirements at the start of this section

The fleet management appliance needs an FQDN, registered in DNS, then an admin and root password meeting the requirements

Lastly the collector for operations needs a DNS registered FQDN and a root password meeting the requirements

Then click Next in the bottom right

For VCF Automation, we need a DNS registered FQDN, an administrator password meeting the requirements
Then we needs two node IPs, these want to be additional IPs
Eg, my FQDN, lab-vcf9-vcfa.leaha.co.uk is 10.1.23.24, and my node IPs are 10.1.23.25 and 10.1.23.26
Add a node name prefix, and leave the internal cluster CIDR at the stock value and click Next

The vCenter sizing has these requirements, tiny should only be used in labs and POCs, otherwise select the size that meets your requirements

We then need a DNS registered vCenter FQDN, select the appliance size, storage size, default is generally fine here, then we need a datacenter and cluster name, vSphere domain name, vsphere.local is fine here and can be left at the default, then we need a password for the [email protected] account and root accounts meeting the requirements
Then click Next

Sizing wise, VCF has the following, this is from VCF 5.1 as VCF 9 doesnt give host numbers

For NSX we need to set the manager size, Medium is generally fine and what you likely need, we then need a cluster and node FQDN, both DNS registered, and an admin, root and audit password meeting the requirements
Then click Next

We need to select our storage type, we can do VMFS on Fibre Channel, NFS v3, and vSAN, which we will be using
Select the vSAN architecture, you likely want ESA, which requires NVMe disks and HCL compatibility, but you can use the OSA architecture with Cache and Capacity disks
Name the datastore and click Next

We then need to add the password for the ESX hosts, they should all be the same, add our hosts, I have 4, add the FQDN, then click Add for each host

This will grab the thumbprint, and we can click Confirm

Then click Next

For networks we need our ESX management network, including the VLAN, leave the default 1500 MTU, CIDR and gateway
I am using the same VM for the VM management so I checked that box, but if you wanted a separate one you can add those details
Then we need vMotion and vSAN networks, and the ports should have an MTU of 9216 for this, at the physical switch level, normally these are isolated non routable networks, but a gateway is needed, so we can just enter an IP here
We need the VLAN, CIDR, gateway, my lab has this gateway to make troubleshooting easier, but you can put a placeholder IP in, and an IP range to assign to hosts, on a /24 network I like 11-254
Then click Next

How we do the VDS depends on our NIC config, you should have 4 or 6 NICs at 10Gb all configured the same at the switch side
For a 4 NIC host setup, select Storage Traffic Separation, for a 6 NIC host setup like mine, select Storage Traffic And NSX Traffic Separation

We will then get this wizard, we can use the arrows on the VDS to edit them

We can then name the VDS, the uplinks here want to contain the vmnic used in management at the moment, if you have only 2 physical NIC cards, this is the one which should have two uplinks from a single card, while loosing vCenter, ESX Management and vMotion isnt ideal, it isnt a P1 incident compared to loosing vSAN or VM traffic with NSX

We then need to name the port groups, the remaining settings are fine at the default

We then need to repeat for the vSAN VDS naming it, adding our uplinks, if you have 2 or more cards these uplinks should also span both physical cards, and name the port group

We then need to edit the last VDS and name it, this has out network traffic, and should use the remaining ports, if you have only two physical NIC cards this should use ports on both cards, 1 on each

Leave this on the default

We then need the transport zone information, name the transport zone, I named it after the management domain name, add the host TEP VLAN ID, IP assignment should be IP Pool, then name the pool, add the CIDR, IP start/end range and the gateway, this gateway must exist

Then click Next

Enter the SDDC Manager admin password, this should be the same as the admin password for the VCF installer and click Next

Then click Next for the validation

You’ll need to address any issues you get, though mine ran without any and looks like this, when you are ready, click Deploy

We’ll then get a different menu that looks like this where we can track our progress

When thats done it will look like this, and we can start deploying the remaining components

We can follow the link on the right to open VCF Operations for the next Steps

The one system we really do want HA on is the NSX manager cluster, we need to expand this with the SDDC Manager API, as of 9.0.x

Thankfully we dont need to do anything complex with this, as APIs can be very confusing if you are new, Operations has an API explorer with a nice template we can use to easily do this

Log into VCF Operations and click Developer Center/APIs & SDKs then click API Explorer on the SDDC Manager API widget

Search for NSX and the bit we want to expand is the GET request for getting our clusters, this will give us our cluster ID which we need for the scale out operation

Expand the request and click Execute, you dont need to fill anything out

We can see the cluster object, you can click the link to expand it

Now we can see our single node, and we have the ID we can copy for later
In my case my ID is d9d1b556-db33-4d8e-88e7-19daccfff4d9

Now we need the POST request to scale out the cluster

We have two main parameters we need, the cluster ID and the body, for the cluster ID, enter your ID

We then need to add the body, the template for it is this

{
  "nsxManagerSpecs": [
    {
      "name": "",
      "networkDetailsSpec": {
        "dnsName": "",
        "gateway": "",
        "subnetMask": ""
      }
    },
    {
      "name": "",
      "networkDetailsSpec": {
        "dnsName": "",
        "gateway": "",
        "subnetMask": ""
      }
    }
  ]
}

We then need to fill our the variables like below, for each manager

• Name – Hostname
• dnsName – FQDN
• Gateway – Network Gateway
• subnetMask – Subnet mask

This is what I did for my managers

{
  "nsxManagerSpecs": [
    {
      "name": "lab-vcf9-mgmt-nsx02",
      "networkDetailsSpec": {
        "dnsName": "lab-vcf9-mgmt-nsx02.leaha.co.uk",
        "gateway": "10.1.23.1",
        "subnetMask": "255.255.255.0"
      }
    },
    {
      "name": "lab-vcf9-mgmt-nsx03",
      "networkDetailsSpec": {
        "dnsName": "lab-vcf9-mgmt-nsx03.leaha.co.uk",
        "gateway": "10.1.23.1",
        "subnetMask": "255.255.255.0"
      }
    }
  ]
}

Then click Execute

Its worth noting this will execute the command on your environment, if you are happy, click Continue

We can view the status from Fleet Management/Tasks and then select our VCF instance

Now we have the extra appliances deployed we need some Edge VMs to do the network transport
We need two DNS registered FQDNs for this

Its here that we need our Uplink VLANs for BGP
I would recommend having ToR 1 owning Uplink 1 as the BGP neighbor on this subnet, and ToR 2 owning Uplink 2
In my lab, I only have 1 OPNsense router, so it my case it will own both Uplink VLANs

In vCenter, click the vCenter its self then Networks/Network Connectivity and click Configure Network Connectivity

Select Centralized Connectivity and click Next

Check the Select All Box, reviewing the prerequisites, and click Continue

Give the Edge cluster a name, select the Large form factor and click Add

Enter the node FQDN, select the cluster, optionally add a resource pool, leave host affinity on No, select a datastore
Then for the management IP select Static, then add the VM Management port group, enter the management IP in CIDR address and add the gateway

For the Active PNIC, select the first uplink on the VDS used for NSX traffic

This will correctly auto populate the rest

Enter the Edge TEP VLAN, select IP pool, then click the three dots and click Create New

Give it a name and click Set under Subnets

Click Add Subnet/IP Ranges

Add an IP range for TEPs, as the whole VLAN is dedicated for this I used almost all the IPs, then add the network in CIDR notification, gateway, DNS servers and DNS suffix then click Add

Then click Apply

And click Save

Then click Apply

And repeat for the second Edge Node

Once thats done it should look like this, then Click Next

We then need a name for the, enter a name for the T0 gateway, keep HA on Active/Standby, this is very difficult to change later, and Active/Active isnt supported for the supervisor with VPC, routing needs to be BGP, and we then need a local AS number, this must be unique on your network
My lab router has ASN 65535 and thats all I have
But you might want something like ToR1 on 65534 and ToR2 on 65535
I used 65530 for the edge cluster

For gateway uplinks click Set

Now we need to enter the details for the Uplink 1 VLAN
Enter the VLAN ID, interface CIDR, this is the UP the Edge will have and much be unique, gateway IP the ToR has, and enter the ASN number configured on ToR1, then click Next

Repeat for Uplink 2 and click Apply

The same config should be applied to the other Edge node
Uplink 1

Uplink 2

When its done it should look like this

We then need our VPC connectivity, we need two large subnets, I recommend /16s, that can be split out as needed within VPCs, these blocks should not overlap anywhere else on your network
I opted for 10.100.0.0/16 and 10.101.0.0/16
Then click Next

Review the config and when you are happy, click Deploy

You will need to update your BGP config on the ToRs with the addresses the Edges have on each uplink VLAN so BGP is then communicating properly

To get access to K8S in vSphere, or the new All Apps organisation type in VCF Automation, we need the supervisor deploying

Ib vSphere, click the three lines in the top left and click Supervisor Management

Then click Get Started

Make sure you have selected VCF Networking With VPC and click Next

Click the Cluster Deployment tab, then enter a name for the supervisor, make sure the toggle is selected for control plane HA, select the cluster, and optionally provide a zone name, I recommend the cluster name, it must be all lower case, if you dont enter one, the system will generate one and it cannot be changed
Then click Next

Select a control plane storage policy, I recommend the vSAN default for a 4 node or less, or you can use the ESA default, likely RAID 5 on a 5 node cluster or larger, then click Next
If you are using VMFS you will need to create your own and this must be a thick provisioning policy, thin can be used for deployments within namespaces however
Then click Next

For the control plane networking, set the mode to static, then select the VM Management port group all our VMs like VCF Ops and vCenter are on, then enter a block of 5 IPs, add the subnet mask, gateway and DNS search domain, DNS/NTP should be pre populated, if not add them, comma separated, then click Next

The NSX project and VPC connectivity profile should be automatically populated with the External and Private Transit gateway IP blocks
We then need private VPC blocks for the workload, this can overlap with any other network, I recommend a /16, I used 10.1.0.0/16, the service CIDR can be left at the default, then add your DNS/NTP servers, comma separated and click Next

For the control plane size, small should be fine for most environments, we can add a DNS registered FQDN for accessing the API, we will need at a later point during the Supervisor configuration guide, for now ensure this isnt bound to any FQDN, then click Next

Then review and when you are happy click Finish

First, we need to get the service files from the Broadcom portal, head to My Downloads and click the HERE button for Free Software Downloads

Search and click vSphere Supervisor Services

Expand Local Consumption Interface and click on the latest release, eg 9.0.1

Click the Terms And Conditions like to enable the check box, then click the download icon on the right for the YAML file

In vSphere click the three lines in the top left and click Supervisor Management

Then click Services and on the Add New Service widget, click Add

Then click Add

Double click the YAML file

Then click Finish

We can then click on the Local Consumption Interface Widget and click Actions/Manage Service

Select our supervisor and click Next

Wait for the pre check to run and click Finish

This will generate a new namespace and we can wait for the pods to deploy

VCF 9 doesnt need vCLS services and it will be removed in the future, to put it into retreat mode, click the cluster and head to Configure/vSphere Cluster Services/general and click Edit vCLS Mode on the right

Select Retreat Mode and click Next

The NSX passwords will expire after 90 days, and with SDDC managing them, and there being a total of 7 appliances, with three accounts each, password expiry is going to be a problem, give we have random 15+ character passwords, I would suggest disabling password expiring for the NSX appliances, but this is entirely optional

Sadly, SSH is disabled, so this will have to manually done via the console, when you are logged in run

set user admin password-expiration 9999
set user audit password-expiration 9999
set user root password-expiration 9999

You can confirm this with

get user admin password-expiration

I ran out of VCF 9 licenses in the portal so I couldnt license this lab, so the below steps are from another environment so hostname will be different but the steps are identical

Licensing is no longer handled by Keys in vCenter, rather through VCF Ops and the Broadcom Business Services

If you have an active subscription then youre licenses should show up here
Under License Management/Licenses

To set this up click License Management/Registration
Now we will assume you have internet connectivity and will do a connected registration, on the Connected Widget, click Start Registration

This will prompt you to log into the Broadcom portal, you will need licensing permissions in your Broadcom portal for your organisation

On the default loaded page, enter a display name for the license and click Save And Next in the bottom right

Select your license, I am selecting my VCF license and vSAN, then click Save And Next

Then click Next

Click Copy next to the activation code

In VCF Ops click Enter Activation Code

Then click Activate

Now we can head to License Management/Licenses, click our vCenter in VCF Ops, mine is called VCF_istlab-vcf and click Assign Primary License
If yours doesnt show up here, check out the bottom in section 1.12 and changing the SDDC manager integration as this will be needed

Select our license and click Assign

Wait for it to be applied

We need to also click Assign Add On License for vSAN

Select the vSAN license and click Assign

We can then see our cluster is fully licensed and we can see the usage

Its worth noting the vDefend licenses arent here and need to be applied directly to NSX, you can get the keys from your entitlements like with VCF 5.2 licenses

Then in NSX, from System/Settings/Licenses, we can click Add License and copy the keys over

The best practices method to backup a vCenter is to use the config backups in VAMI
To access VAMI go to the following link substituting fqdn for your vCenters FQDN
https://fqdn:5480

You can log in here with the local root account, or an SSO admin login

Now head to the backups tab at the bottom on the left, from here you can click ‘Configure’ on the right to setup a schedule

You’ll need a valid backup location to store them, an SMB, NFS or FTP server work best but you can also use HTTPS and FTPS
The backup schedule will give you a format for the backup location

We want to setup our location, here I am using an SMB server, but for NFS/SFTP the process is the same you just change the protocol at the start to NFS or SFTP respectively
We can also add in an account with read/write permissions to the share, I recommend a service account with a password that wont expire, as if it expires and you forget, the backups will stop working
You can encrypt the backup, however you must not loose the password else you cant restore it
You’ll want it to run daily, ideally if you need to restore you dont want a backup older than 24 hours
Retain the last 7 backups, this will remove older backups and maintain its self
And check all boxes at the bottom to back up everything
Then hit create

To test this works, run a manual backup by clicking backup now on the right

Click use backup location and username at the top of the pop up, this will pull the settings from the schedule, you’ll just need to enter the account password
Then click start

That will create a manual backup task

If all is working, this should complete with no errors

Now your vCenter is backed up and will automatically back its self up everyday for you, so if something goes wrong you have a way to restore it

We then need to configure the SDDC Manager backups, in VCF Operations, click Administration/SDDC Manager and click your VCF Instance

Click Backup Settings/Site Settings
Then enter your SFTP server fqdn, the port will be 22, protocol is SFTP, there are no other options, enter the username/password for the SFTP server, I recommend a service account
Then add the backup directory for where you want to store the backups, click Confirm Fingerprint, enter a passphrase for the backups and click Save
The passphrase should be 15 characters, the only allowed special characters are !@#$%^&*
Mine did also error on the FQDN which was odd, you can just enter the IP instead which solved that

Then click Confirm

We need to wait for this configuration to complete

Then click SDDC Manager Configurations and click Edit under Backup Schedule

Click the toggle to enable the schedule, set the frequency to weekly and select all days, for daily backups, add a time, you can optionally do backups on a state change, then set the retention policy, which I need 7 days, then click Save

Then click Backup Now to make sure it works

And make sure its successful

Lastly we need some backups for the fleet management servers, click Fleet Management/Lifecycle/VCF Management

Click Settings/SFTP Settings, then add your backup server IP, I tried the FQDN but got resolution issues on internal K8S pods so I recommend using the IP, port 22, SFTP protocol, username, I recommend a service account, and click the + to add a password

Add an alias, a friendly name, then the password and click Add

Click Select Password

And select the password

Enter the directory to save the backups on the SFTP server, then click the + for a passphrase

Enter an alias, the password, and click Add
The passphrase should be 15 characters, the only allowed special characters are !@#$%^&*

Then click Select Passphrase

And select the password

Then click Fetch Fingerprint

And click Save

This will trigger an SFTP server update

We can then configure a schedule for the automation appliance

In Settings/Backup Settings click Edit on the Management Node

Set a time for a daily backup and enable the retention policy, 7 days is plenty and click Save

It should look like this

Unfortunately the config backups for fleet management are stored locally and oddly dont follow the SFTP setup, so I recommend also backing it up with your backup provider with the other VMs below

For VCF Automation to test that works, head to Components and click the automation component

Click the three dots, then Backup And Restore/Backup

Check the box and click Confirm

For VMs to backup with your backup software that arent covered by these, you’ll need to add

• VCF Ops For Networks
• Fleet Management
• VCF Ops
• VCF Ops Collector
• VCF Ops For Logs

Open the management domain vCenter and click the three lines in the top left, then click Global Inventory Lists

On the left, click Hosts

We first need a network pool, if you are expanding a cluster thats already been created, there will already be a pool that can be used, in that case you can skip this part, but if you are adding a workload domain a new pool will be required, for a new cluster you may or may not need a new pool
Pools must not have overlapping IP ranges
Click the Network Pools section and click Create Network Pool

This part will depend on what you are deploying storage wise, but you’ll pool for vMotion and one for your storage, for this cluster I am using vSAN, it also needs a name
Hosts will also need to be setup like in the ESX section
When you have your networks enter the VLAN, MTU, which likely is 9000, but this will match what you did in the deployment, then add the network, subnet mask, gateway, its worth noting here, neither of my networks actually have a gateway, then enter an IP range for hosts
When you have the address ranges, you’ll need to click Add

When you are done it should look like this, when you are happy, click Save

Then we need to commission a host, click Unassigned Hosts and click Commission Host

Check and prerequisites, you will need to select all before continuing
When you are happy click Proceed

Its worth noting 10GbE NICs are required for hosts in VCF 9
There is a workaround WilliamLam posted, which allows slower NICs, but should only be applied to lab environments, his article can be found here
But all you need to do is SSH into the SDDC Manager, switch to the root account with, su root, and run

echo "enable.speed.of.physical.nics.validation=false" >> /etc/vmware/vcf/operationsmanager/application.properties

systemctl restart operationsmanager

Add your host FQDN, select the storage type, select the network pool we created earlier, enter the root credentials and click Add
If your hostname is over 15 characters click Acknowledge, this can be ignored as hosts shouldnt be domain joined
Repeat for all hosts to commission, a non vSAN cluster requires at least two hosts, while vSAN requires at least three hosts

At the bottom, click the toggle to confirm the fingerprint and click Validate All

Then click Validate All

Once thats validated, click Next

Then click Commission

When its done it should look like this

This will need to be done via VCF Operations, click Inventory, by default you will be on the simplified view, so click Details View

Expand VCF Instances and select your VCF instance, then click Add Workload Domain/Create New

Check the prerequisites and click Proceed

Give the workload domain a name, we can disable the supervisor for now, it can be manually configured later like in section 7
Provide an SSO domain name for the vCenter, I would stick with the default vsphere.local, I then unchecked the Password Creation box so we can manually set passwords
When you are happy click Next

Add the vCenter FQDN and root password, then click Next
This will default to a large vCenter with 8vCPU and 30GB RAM

Give the cluster a name and click Next

Select our new cluster image and click Next

We then need to setup our NSX instance, I would always recommend HA in production, then select the size, I am using Medium

We then need to add the FQDNs of the three target NSX manager nodes, the IP addresses will be populated automatically from a DNS lookup

Next enter the NSX instance VIP address FQDN

Next add a password for the admin and auditor accounts, and select the VPC network connectivity topology, generally, you’ll always want to go with Centralized, this uses NSX Edges and gives you full functionality
Then click Next

We then need to choose our storage type, I have planned for vSAN ESA, so I select that and clicked Next

The default vSAN HCI is what you likely want, if you are unsure use this option, if you know you need a vSAN storage cluster select that and click Next

We then need to select our hosts, I am using all four I added earlier, then click Next

We then need to select our networking topology, I recommend 6 NICs using the storage and NSX traffic separation, but if you have only 4 NICs, what I would consider to be the minimum, use storage separation

Now, while we have the defaults, this doesnt actually let you edit the settings, and the issue there is the NSX TEP addresses will be using DHCP when we want an IP pool like the management domain during the deployment, so click Create Custom Switch Configuration at the bottom, and we will manually set up the topology

Click Create Distributed Switch

Add a VDS name and set the MTU, this should be 9000, but in line with what was set for the management domain, we need 2 uplinks, select the vmnics you want to use then click Configure Network Traffic Type/Management
One of these uplinks should be bound to the vSwitch on the default ESX install

Give the port group a name and select the load balancing policy of route based on physical NIC load then click Save Configuration

Click Configure Network Traffic/vMotion

Give the port group a name and select the load balancing policy of route based on physical NIC load then click Save Configuration

Scroll to the bottom and click Create Distributed Switch

Click Create Distributed Switch

Give the VDS a name, set the MTU the same, which should be 9000, select the two uplinks you want to bind for storage and click Configure Network Traffic/vSAN
If you chose NFS configure that here

Give the port group a name and set the same load balancing option then click Save Configuration

Then scroll to the bottom and click Create Distributed Switch

Click Create Distributed Switch one last time

Give the VDS a name, set the MTU to the same, at 9000, then add the remaining uplinks, then click Configure Network Traffic/NSX

Leave the default boxes checked, all three should be, give the overlay transport zone a name, add the host TEP VLAN as the Transport VLAN, set the IP Allocation to Static IP Pool and create a new pool
Then give the pool a name, set the CIDR, IP range and gateway

Give the VLAN transport zone a name, the uplink number should match the number on the VDS, in our case, 2, and set the NSX and VDS uplinks to match

Give the NSX uplink profile a name and set the Teaming Policy to Load Balance Source and click Save Configuration

Scroll down and click Create Distributed Switch

Then click Next

Review all the info and when you are happy click Finish

We can click View SDDC Manager Tasks on the pop up to track the progress

The default workload domain workflow doesnt deploy a collector as part of it, you can either use the collector in the management domain or you can deploy one to the cluster, I would recommend each workload domain has its own collector so thats what we will setup now

We can do this from within VCF Operations, head to Fleet Management/Lifecycle/VCF Management/Components and click the operations components

Click Add Node

Then click

When thats done click back to the Components view

Click the operations component again

Then click Add Nodes

Now click Proceed

This should all be pre populated, click Next

Click Next again, this should all populated again

Click Add Password

Give the password a name, this will be for the root password, and add a new password, then click Add

Scroll down to Components and click the + then click Cloud Proxy

Enter a VM name, FQDN IP address and set the size, we want a unified proxy, small is fine for up to 16k VMs, standard does up to 80k, then click the setting icon, next to the trash can icon, on the right

Scroll down to Root Password and click Select Root Password

Select our new password

Then click Save

Then click Next

Then click Run Precheck

When thats passed, click Next

Review the config and click Submit

By Default, the workload domain should get integrated after about an hour or so, but lets properly configure this with our new collector and Ops For Logs

In VCF Operations click Administrator/Integrations, expand VMware Cloud Foundation, then click the three dots on your VCF Instance and click Edit

Make sure System Managed Credentials is checked, select our new workload domain collector, and make sure Operational Actions is enabled, then check Activate Log Collection and click the vSAN tab

Ensure its enabled and SMART data is being collected then click the NSX tab

Ensure NSX is activated, if you have Operations For Networks, also enable that, I didnt redeploy it when I rebuilt my lab, so I left it, then click Save in the bottom left

We need to log into the SDDC Manager UI on
https://fqdn
We will then get redirected to login with the management domain vSphere SSO accounts

Click Inventory/Workload Domains and select the workload domain which is having its cluster expanded, I will be doing this on the vcf9-wld01 domain

Click Actions/Add Cluster

Select our storage topology, in my case this is vSAN, then click Begin

Enter a cluster name, and select the existing datacenter then click Next

Select our image, I will be using the same as my first workload domain cluster, then click Next

If you didnt select vSAN storage you will not have this option so skip ahead

For vSAN select the vSAN type, this will typically be vSAN HCI unless you know you need a vSAN storage/Compute cluster, then click Next

Select our hosts and click Next

We will be opting for a 6 NIC config and we want to be manually specifying a topology else we cant set an IP pool for NSX, 4 NICs will also work, just ensure storage is on its own VDS
Click Create Custom Switch Configuration

Click Create Distributed Switch

Add a VDS name and set the MTU, this should be 9000, but in line with what was set for the management domain, we need 2 uplinks, select the vmnics you want to use then click Configure Network Traffic Type/Management
One of these uplinks should be bound to the vSwitch on the default ESX install

Give the port group a name and select the load balancing policy of route based on physical NIC load then click Save Configuration

Click Configure Network Traffic/vMotion

Give the port group a name and select the load balancing policy of route based on physical NIC load then click Save Configuration

Scroll to the bottom and click Create Distributed Switch

Click Create Distributed Switch

Give the VDS a name, set the MTU the same, which should be 9000, select the two uplinks you want to bind for storage and click Configure Network Traffic/vSAN
If you chose NFS configure that here

Give the port group a name and set the same load balancing option then click Save Configuration

Then scroll to the bottom and click Create Distributed Switch

Click Create Distributed Switch one last time

Give the VDS a name, set the MTU to the same, at 9000, then add the remaining uplinks, then click Configure Network Traffic/NSX

Leave the default boxes checked, all three should be, give the overlay transport zone a name, add the host TEP VLAN as the Transport VLAN, set the IP Allocation to Static IP Pool and we can create a new pool or reuse the pool for the first workload domain cluster, I am re using the pool

Select the VLAN transport zone from the drop down and set the NSX uplinks to match the VDS uplinks

Give the NSX uplink profile a name and set the Teaming Policy to Load Balance Source and click Save Configuration

Scroll down and click Create Distributed Switch

Then click Next

Review the config and click Finish

Automation can only be scaled out into a cluster, we first need to ensure we have the VCF Automation Binary matching the current version you are running, in my case 9.0.2, select the binary and click Download

To scale out Operations, click Components and click the automation component

Then click the three dots and click Scale

Select a target size of Medium, Large is basically same config, then click Next

The default single node setup will have two IP addresses in a pool, we need a total of four IPs for a HA cluster, click Add Cluster Node IP Pool

Enter a range of two IPs on the same subnet as Automation and click Add

Then click Next

Click Run Precheck

Then click Next

And click Submit