VCF 9.1.x Ultimate Deployment Guide

Mount this to the server, by using rufus to create a bootable USB, or by mounting it to your servers virtual CD ROM in the IPMI, iDRAC for Dell and iLO for HPE

Once the server has booted ESX, you’ll have this screen, select enter to continue

Accept the EULA with F11

It will then scan for bootable devices, for a production system this should be something in RAID 1
Examples are Dell’s BOSS card
For HPE G11 you should have the NS204i-U, or for G10 systems the NS204i-P, which is a PCIe card

As this is a lab, I have a virtual disk, and will be using the 400GB one by making sure its highlighted in Yellow and clicking Enter to Continue

Select your keyboard layout and hit Enter

Set a root password, use something easy to use, we can set a secure random one later

You may get a CPU warning depending on your hardware
Press Enter

Then click F11 to install

Once thats done, reboot the server when prompted and unmount your media

Once thats done, reboot the server when prompted and unmount your media

When the host boots, it should look like this, press F2 to login

Enter the root credentials and press enter

Scroll to Configure Management Network and press enter

Press Enter on Network Adapters and ensure that a connected Nic is selected, these should all be configured the same on the switch

In my case VMNIC0 is connected, and I will be using this for management, so I will press Escape and leave it as it is

Press enter on VLAN

And enter your ESXi management VLAN, I am using VLAN 1023
This is only needed if you have your VLANs trunked down, if your management VLAN is the native VLAN you can ignore this, as all my VLANs are trunked down, I am entering mine

On IPv4

Use the space bar to select the third option to set a static IP and add your management IP details in and press enter to Save

For IPv6, select disable on the first option, unless you are specifically using it, and press Enter

Add your DNS servers and the hostname for this server and press Enter

Add your domain under DNS Suffixes and press Enter

Now press Escape and enter Y to apply changes and reboot the host

Then, login on the WebUI at
https://fqdn
And login with the root credentials

Navigate to Host/Configure/Security & Users/Users, and click the root account and click Edit User to change the root password to something more secure

Then add the password
This needs to be 15 characters with the only allowed special characters being !@#$%^&*
Then click Save

Then we need to enable SSH, click Host/Configure/System/Services select SSH and click Start

Now we need to SSH into the host with the root credentials, using something like Putty and run the following to set the hostname/FQDN correctly for the certificate and renew it for the VCF deployment wizard

For my host, lab-vcf91-esx01, lets set the hostname with
esxcli system hostname set -H=<hostname>
So for my host this is

esxcli system hostname set -H=lab-vcf91-esx01

Then set the FQDN with

esxcli system hostname set -f=<fqdn>

Which for my host is

esxcli system hostname set -f=lab-vcf91-esx01.leaha.co.uk

Now renew the certificates with

/sbin/generate-certificates

And reboot the host with

reboot

When it comes back up, you will need to restart SSH for the cloud builder

Lastly, we need to setup NTP on all servers, you can use a windows App, DC or a docker container, for this

Head to Host/Comfigure/System/Time Configuration and click Edit under Network Time Protocol

Check the box to enable it, enter your NTP server and click ok

Then, under services, click NTP Daemon and click Edit Startup Policy

Check the radio button for Start And Stop With Host and click ok

We will need to repeat this on the remaining hosts

Before we proceed we need to check we have enough space on the first host to deploy the VCF installer appliance, your boot device will typically be 512GB or larger so there should be a good size local datastore created we can use for this as we cant use our vSAN disks

We can check this under Storage/Datastores
Mine is only 271GB but should be enough

This only seems to happen for virtual ESX hosts used in labs, when the build option tries to migrate the management vmk to a VDS it will fail

This does not need to be done on physical hardware

We need SSH and the ESX Shell, to enable this, click Host/Configure/System/Services, select ESXi Shell and click Start

Then open up the ESX console and press Alt + F1 to access the shell and login

Then run

esxcli network ip interface list | less

We should see vmk0, the management interface

We need to note the portgroup, which should be ‘Management Network’
We can press ‘q’ to exit this

Now remove the interface with

esxcli network ip interface remove –-interface-name=vmk0

Then recreate it with

esxcli network ip interface add -–interface-name=vmk0 -p "Management Network"

We can press Alt + F2 to switch back to the DCUI, press F2 and login as root

Press Enter on Configure Management Network

Press Enter on IPv4 Configuration

Use the third option to set a static IP and enter the details setup at the ESX deployment stage and press Enter

Then press Escape

And press Y here

Now back in the host UI, click Host/Configure/System/Services, select ESXi Shell and click Disable

Then repeat for the remaining hosts
 

Log into the first host and click Networking
If you set a VLAN for the management VLAN and the management components are going on the same VLAN, which I recommend, we will need to edit the VM Network to set this VLAN, click Networking, then click the three dots on the VM Network entry and click Edit Settings

Set the VLAN tag and click ok

Now click Host/Actions/Deploy OVF Template

Give the VM a name and click Upload Files

Double click the SDDC Manager OVA

Then click Next

Click Next

Accept the EULA

Select the default datastore and make sure Thin Provision is enabled then click Next

Make sure the VM Network is selected and click Next

Under Application enter a root and local user password, these need to be 15 characters with the only special characters being !@#$%^&*
For the hostname enter the FQDN and for NTP add your NTP server

For the networking section, select the IP version, likely IPv4, enter the SDDC Manager IP address, subnet mask, gateway, DNS domain and search domain path and DNS servers, comma separated, then click Next

And click Finish
Do not refresh your page while this is deploying

When thats uploaded, click Virtual Machines, right click the VM and click Power/Power On

Now we have the appliance deployed we need to download all the software binaries, log into the VCF Installer on
https://fqdn

Then click Depot Settings And Binary Management

On the Connect To The Online Depot widget, click Configure

Copy the service ID and then head to vcf.broadcom.com

Then click Software Depot Registration and click New Registration

Paste the Service ID from the VCF Installer and give the depot a name, then click Register

Copy the activation code and click Finish

Paste this into the Activation Code section in the VCF Installer and click Authenticate

At the bottom, select your release version, I am doing 9.1.0.0, and select everything apart from the SDDC Manager and click Download

When its all done, it should look like this

We can then click Return Home at the top left to get back to the main menu

As we proceed through we will need to add various appliances, ensure all are DNS registered as you go through it before you finish the deployment

On the Deploy Widget, click Deployment Wizard/VMware Cloud Foundatio

Click Next

As this is a brand new deployment from scratch we want to make sure we have Deploy A New VCF Fleet selected and click Next

Review the config and click Next when you are happy

This will kick off the validation

If anything fails it must be addressed, the only warning I got was over capacity, it assumes a 1:1 ratio on pCPU to vCPU which is a little overkill, but you are unlikely to hit this in production and my storage is a little lower, with thin provisioning it should be fine, but ensure you have enough storage on a production system

When you are happy, click Deploy

We can then watch it go through all of the stages
We can also click Review Passwords for all the generated credentials, ensure you save these

You can copy this as a JSON or CSV

When its done it will look like this and we can proceed to the VCF Operations UI

The one system we really do want HA on is the NSX manager cluster, we need to expand this with the SDDC Manager API, as of 9.0.x

Thankfully we dont need to do anything complex with this, as APIs can be very confusing if you are new, Operations has an API explorer with a nice template we can use to easily do this

Log into VCF Operations and click Build/Developer Center/APIs & SDKs then click API Explorer on the SDDC Manager API widget

Search for NSX and the bit we want to expand is the GET request for getting our clusters, this will give us our cluster ID which we need for the scale out operation

Expand the request and click Execute, you dont need to fill anything out

We can see the cluster object, you can click the link to expand it

Now we can see our single node, and we have the ID we can copy for later
In my case my ID is 8f4739c3-f002-493b-ae7c-731c1247344a

Now we need the POST request to scale out the cluster

We have two main parameters we need, the cluster ID and the body, for the cluster ID, enter your ID

We then need to add the body, the template for it is this
Remove the IPv6 entries, like mine below, if you arent using it

{
  "nsxManagerSpecs": [
    {
      "name": "",
      "networkDetailsSpec": {
        "dnsName": "",
        "gateway": "",
        "ipAddress": "",
        "ipv6Gateway": "",
        "ipv6PrefixLength": 0,
        "subnetMask": ""
      }
    },
    {
      "name": "",
      "networkDetailsSpec": {
        "dnsName": "",
        "gateway": "",
        "ipAddress": "",
        "ipv6Gateway": "",
        "ipv6PrefixLength": 0,
        "subnetMask": ""
      }
    }
  ]
}

We then need to fill our the variables like below, for each manager

• name – Hostname
• dnsName – FQDN
• ipAddress – IPv4 Addres
• gateway – Network Gateway
• subnetMask – Subnet Mask

This is what I did for my managers

{
  "nsxManagerSpecs": [
    {
      "name": "lab-vcf91-nsx02",
      "networkDetailsSpec": {
        "dnsName": "lab-vcf91-nsx02.leaha.co.uk",
        "gateway": "10.1.23.1",
        "ipAddress": "10.1.23.162",
        "subnetMask": "255.255.255.0"
      }
    },
    {
      "name": "lab-vcf91-nsx03",
      "networkDetailsSpec": {
        "dnsName": "lab-vcf91-nsx03.leaha.co.uk",
        "gateway": "10.1.23.1",
        "ipAddress": "10.1.23.163",
        "subnetMask": "255.255.255.0"
      }
    }
  ]
}

Then click Execute

We’ll see it showing as in progress now

We can view the status from Build/Tasks and then by clicking our VCF instance

Now we have the extra appliances deployed we need some Edge VMs to do the network transport
We need two DNS registered FQDNs for this

Its here that we need our Uplink VLANs for BGP
I would recommend having ToR 1 owning Uplink 1 as the BGP neighbor on this subnet, and ToR 2 owning Uplink 2
In my lab, I only have 1 OPNsense router, so it my case it will own both Uplink VLANs

In vCenter, click the vCenter its self then Networks/Transit Gateways and click Setup Network Connectivity

Select the Span to be the default, we can use spans to limit connectivity between certain vCenters within an NSX instance, but the default will set it for all, then set the Connection to Centralized Connection and click Next

Check the Select All Box, reviewing the prerequisites, and click Continue

Give the Edge cluster a name, select the Large form factor, this will be needed for the Supervisor, and click Add

Enter the node FQDN, select the cluster, optionally add a resource pool, leave host affinity on No, we can configure this later, select a datastore
Then for the management IP select IPv4 only and click Static for the assignment, enter the management IP in CIDR address, add the gateway, then add the VM Management port group we set during the deployment, if you need to check its the port group vCenter is on

For the uplinks, uncheck the box to use the host overlay network the active/standby pNICs should alternate like this and will be populated by default

Enter the Edge TEP VLAN, for the IP Type, select IPv4, select IP pool, then click the three dots and click Create New

Give it a name and click Set under Subnets

Click Add Subnet/IP Ranges

Add an IP range for TEPs, then add the network in CIDR notification, gateway, DNS servers and DNS suffix then click Add

Then click Apply

Add a description and click Save

Now click Run Check to check the VLAN MTU

Then click Apply

And repeat for the second Edge Node, we wont need a new IP pool as we can select the new one from the drop down

Once thats done it should look like this

At the bottom we can remove the toggle to set our own passwords if we want to, when you are happy, click Next

We then need a name for the gateway, enter a name for the T0 gateway, keep HA on Active/Standby, this is very difficult to change later, and Active/Active isnt supported for the supervisor with VPC, routing needs to be BGP, and we then need a local AS number, this must be unique on your network

My lab router has ASN 65535 and thats all I have
But you might want something like ToR1 on 65534 and ToR2 on 65535
I used 65532 for the edge cluster

For gateway uplinks click Set

Now we need to enter the details for the Uplink 1 VLAN
Enter the VLAN ID, interface CIDR, this is the UP the Edge will have and much be unique, gateway IP the ToR has, and enter the ASN number configured on ToR1, then click Next

Repeat for Uplink 2 and click Apply

The same config should be applied to the other Edge node
Uplink 1

Uplink 2

When its done it should look like this

We then need our VPC connectivity, we need two large subnets, I recommend /16s, these must not overlap anywhere else on your datacenter, that can be split out as needed within VPCs, these blocks should not overlap anywhere else on your network
For VPC External IP Blocks, click the three dots and click Create New

Enter a name and add the CIDR, I opted for 10.102.0.0/16 and click Save

Repeat for the transit gateway blocks clicking the three dots and click Create New

Add a name and the IP block, I opted for 10.103.0.0/16, then click Save

Then click Next

Review the config and when you are happy, click Deploy

You will need to update your BGP config on the ToRs with the addresses the Edges have on each uplink VLAN so BGP is then communicating properly

We can click View Details to see the deployment

This takes us to Configure/Networking/Edge Clusters

This K8S cluster hosting a large sum of components can be scaled if you did a simple deployment when building the VCF instance, the key difference is the K8S control plane consists of a single server, 4vCPU/10GB, and can be scaled to a HA 3 node cluster

I would recommend scaling this to HA on the control plane in a production environment

Its also worth noting, during this deployment I did notice one additional worker node, the 12vCPU/24GB VMs, get deployed so scaling this can have larger requirements than just the control plane cluster

Also, after scaling this to medium, the worker nodes were slowly replaced

I would recommend scaling this if you are planning to scale components running on this platform beyond the minimum, this includes

• Log Management
• Real Time Analytics
• Identity Broker
• Salt
• Software Depot

To begin, log into VCF Operations and head to Build/Lifecycle/VCF Management and click VCF Services Runtime

Click Actions/Scale

Select the Medium Size, unless your VCF Fleet is massive Large is very overkill, check the box to acknowledge potential service interruption and click Next

We can add additional IPs if needed, as we deployed this with an IP pool of 30, meant for the larger scales, we can just click Next as we have plenty of IPs

And click Finish

Now that we have the environment and Log Management deployed, we need to onboard our domains so they log to the Log Management cluster

Head to Operate/Administration/Integrations, expand VMware Cloud Foundation, and click the three dots on your VCF instance and click Edit

Click Domains, here we will then see all our domains, you will be on the vCenter by default, under Log Operations, click Activate Log Collection, and select the radio button to bring them directly into the cluster

On vSAN, if you have it, check the box for Enable SMART Data Collection

And for NSX do the same as vCenter and click Save

Now that we have the environment and Log Management deployed, we need to onboard our domains so they log to the VCF Operations For Networks cluster

Head to Operate/Administration/Integrations, expand VMware Cloud Foundation, and click the three dots on your VCF instance and click Edit

Click Domains, here we will then see all our domains, you will be on the vCenter by default, under Log Operations, click Activate Network And Flow Collection, select our collector, then click the check box to Enable NetFlow on the vCenter

Click Enable

Repeat for the Antrea IPFIX and check the box

And click Enable

Click NSX at the top, and check the box for Activate Network And Flow Collection, select the collector and check the box to enable IPFIX

Then check the bot for latency collection and click enable

Then click Save

This section was done on another environment due to licensing constraints, FQDNs will not match the rest of the guide

Its worth noting, vDefend licensing is no longer done by a key in NSX, it requires a different licensing server deployed by the Security Services Platform, requiring 6vCPU and 24GB RAM
It has been excluded from this section due to it being a paid add on and will be included in the vDefend configuration guide, though there is no ETA on this

To license our environment we need to log into VCF Operations and click Manage/Licensing/Licenses & Registration then click Continue

Then click Start

Click Connected and click Continue

Now click Start again under Registration

Log into the Broadcom portal and click Start
The screenshot below may automatically appear

We can use the VCF Operations name and click Save

Click Start on Generate Activation Code

Click Copy to copy the code, then click Finish

In VCF Operations click Start on Enter Activation Code

Paste the code in and click Activate

Now under Add Licenses To License Server, click Start

In the VCF portal, under Add Licenses, click Start

Select your licences and click Confirm

In VCF Operations, click Download

You should get a pop up, when thats come through, click Close

And click Finish

In the VCF Portal, click Mark As Completed

In VCF Operations, scroll down, select your vCenter and click Assign Primary License

Select the license and click Assign

Repeat click Assign Addon License/VMware vSAN for vSAN

Select the license and click Assign

Open the management domain vCenter and click the three lines in the top left, then click Global Inventory Lists

On the left, click Hosts

We first need a network pool, if you are expanding a cluster thats already been created, there will already be a pool that can be used, in that case you can skip this part, but if you are adding a workload domain a new pool will be required, for a new cluster you may or may not need a new pool
Pools must not have overlapping IP ranges
Click the Network Pools section and click Create Network Pool

This part will depend on what you are deploying storage wise, but you’ll pool for vMotion and one for your storage, for this cluster I am using vSAN, it also needs a name
Hosts will also need to be setup like in the ESX section

When you have your networks enter the VLAN, MTU, which likely is 9000, but this will match what you did in the deployment, then add the gateway in CIDR notation, its worth noting here, neither of my networks actually have a gateway, then enter an IP range for hosts
When you have the address ranges, you’ll need to click Add

It should then look like this with the network range added

We then need to repeat for vMotion and click Save

Then we need to commission a host, in vSphere click the three lines in the top left and click Global Inventory Lists

Click Hosts/Unassigned Hosts/ click Commission Host

Check and prerequisites, you will need to select all before continuing
When you are happy click Proceed

Add your host FQDN, select the storage type, select the network pool we created earlier, enter the root credentials and click Add
If your hostname is over 15 characters click Acknowledge, this can be ignored as hosts shouldnt be domain joined

Repeat for all hosts to commission, a non vSAN cluster requires at least two hosts, while vSAN requires at least three hosts, though I recommend four

At the bottom, click the toggle to confirm the fingerprint and click Validate All

Then click Validate All

Once thats validated, click Next

Then click Commission

When its done it should look like this

This will need to be done via VCF Operations, click Operate/Inventory, by default you will be on the simplified view, so click Details View

Expand VCF Instances and select your VCF instance, then click Add Workload Domain/Create New

Check the prerequisites and click Proceed

Give the workload domain a name and select Full Deployment With Cluster

We can disable the supervisor for now, it can be manually configured later like in section 7, credentials will be automatically generated and can be extracted after the domain has been built from the Operations UI, click Next

Add the vCenter FQDN, this will need to be on the same network as the management domain vCenter, enter an SSO domain fort he vCenter, the default vsphere.lcoal wil be fine then click Next
This will default to a large vCenter with 8vCPU and 30GB RAM

Give the cluster a name and click Next

Select our new cluster image and click Next

We then need to setup our NSX instance, I would always recommend HA in production

We get a couple of options, in VCF 9.1 we no longer require a dedicated NSX instance for the first workload domain and we now have the option of joining this to the management domain NSX instance

There are a few of things to note here

Firstly, you likely deployed the medium size for NSX during the deployment, this only supports two vCenters and I cant find the correct way, if possible, to scale this up

Secondly, everything in NSX will generally be available across domains, some people prefer separation, in which case we would want a dedicated NSX instance, though we can use the new Spans features to localize VPCs to particular domains

Thirdly, if you reuse a given NSX instance, the hosts will need access to the TEP VLAN the NSX is configured to use

If you are only deploying a single workload domain, this easily fits into the management domain NSX instance, so I would use this, it will save a lot of resources, if you are planning many workload domains, deploying a dedicated NSX instance here makes more sense, and then using the large size to accommodate up to 16 vCenters 

If you deploy a new NSX instance you will need a VIP FQDN plus three more, one for each node and these will need to be on the same VLAN as the management domain vCenter

I will be adding my instance to my existing NSX manager as I have a single workload domain and this is more resource efficient, if I need to isolate VPCs I can use the Spans feature

Select the Join Existing NSX Manager Instance and select the NSX instance, then click Next

We then need to choose our storage type, I have planned for vSAN ESA, so I select that and clicked Next

For vSAN, the default vSAN HCI is what you likely want, if you are unsure use this option, if you know you need a vSAN storage cluster select that and click Next

We then need to select our hosts, I am using all four I added earlier, then click Next

We then need to select our networking topology, I recommend 6 NICs using the storage and NSX traffic separation, but if you have only 4 NICs, what I would consider to be the minimum, use storage separation

Now, while we have the defaults, this doesnt actually let you edit the settings, and the issue there is the NSX TEP addresses will be using DHCP when we want an IP pool like the management domain during the deployment, so click Create Custom Switch Configuration at the bottom, and we will manually set up the topology

Click Create Distributed Switch

Add a VDS name and set the MTU, this should be 9000, but in line with what was set for the management domain, for Type, I would use VDS Uplinks, we need 2 uplinks, select the vmnics you want to use then click Configure Network Traffic Type/Management
One of these uplinks should be bound to the vSwitch on the default ESX install

Give the port group a name and select the load balancing policy of route based on physical NIC load then click Save Configuration

Click Configure Network Traffic/vMotion

Give the port group a name and select the load balancing policy of route based on physical NIC load then click Save Configuration

Scroll to the bottom and click Create Distributed Switch

Click Create Distributed Switch

Give the VDS a name, set the MTU the same, which should be 9000, for the Type I recommend VDS Uplink, select the two uplinks you want to bind for storage and click Configure Network Traffic/vSAN
If you chose NFS configure that here

Give the port group a name and set the same load balancing option then click Save Configuration

Then scroll to the bottom and click Create Distributed Switch

Click Create Distributed Switch one last time

Give the VDS a name, set the MTU to the same, at 9000, for the Type I recommend VDS Uplinks, then add the remaining uplinks, then click Configure Network Traffic/NSX

Leave the default boxes checked, all three should be, give the overlay transport zone a name, add the host TEP VLAN as the Transport VLAN, set the IP Allocation to Static IP Pool and create a new pool
Then give the pool a name, set the CIDR, IP range and gateway

Give the VLAN transport zone a name, the uplink number should match the number on the VDS, in our case, 2, and set the NSX and VDS uplinks to match

Give the NSX uplink profile a name and set the Teaming Policy to Load Balance Source and click Save Configuration

Scroll down and click Create Distributed Switch

Then click Next

Review all the info and when you are happy click Finish

Once the validation has passed click Finish

We can click View SDDC Manager Tasks on the pop up to track the progress

Ideally, you want one collector/cloud proxy per domain, this isnt a hard rule, and it can be more efficient to reuse existing cloud proxies for data collection rather than deploying more increasing resource requirements

Here we will go through deploying a new proxy, if you wanted one, and in the next sub section, setting up the workload domain to use it

NSX and vCenter appliances are deployed into the management domain, and thats the bulk of the data we are collecting, and as such, the default workflow will also deploy the collector into the management domain, on the same network, so its as close to the data sources as possible#

Log into VCF Operations and click Build/Lifecycle/VCF Management/Components and click VCF Operations

Click Actions/Add Cloud Proxy

Enter an FQDN, this must be registered in DNS, select the size, Small will be fine for most scenarios, enter the VCF Operations password and click Re-Generate

You can use the eye to see the password and save it, you wont be shown it again

Select our VCF instance and click Add

By Default, the workload domain should get integrated after about an hour or so, but lets properly configure this with our new collector, Ops For Logs and Networks

In VCF Operations click Operate/Administration/Integrations, expand VMware Cloud Foundation, then click the three dots on your VCF Instance and click Edit

Make sure System Managed Credentials is checked, select our new workload domain collector, and make sure Operational Actions is enabled, then check Activate Log Collection and click the vSAN tab

Ensure its enabled and SMART data is being collected then click the NSX tab

Ensure NSX is activated, if you have Operations For Networks, also enable that, I didnt redeploy it when I rebuilt my lab, so I left it, then click Save in the bottom left

Before we can start, we need a new Edge cluster in the workload domain to keep all its networking local

We will also need two FQDNs registering on DNS, for the network, I will be using the same network as my ESX management, in a similar style to the management domain, but you can choose any port group in the workload domain, my ESX management VLAN in the workload domain is different from that of my management domain

In the workload domain vCenter, click the vCenter object on the left, and click Configure/Networking/Edge Clusters/Add Cluster

Give the cluster a name and set the form factor to Large then click Add

Enter the node FQDN, select the cluster, optionally add a resource pool, leave host affinity on No, we can configure this later, select a datastore

Then for the management IP select IPv4 only and click Static for the assignment, enter the management IP in CIDR address, add the gateway, then select the port group you want for management, you can use the ESX management one, but I recommend a dedicated port group on for VM Management, this should be ephemeral

For the uplinks, uncheck the box to use the host overlay network the active/standby pNICs should alternate like this and will be populated by default

Enter the Edge TEP VLAN, for the IP Type, select IPv4, select IP pool, then click the three dots and click Create New

Give it a name and click Set under Subnets

Click Add Subnet/IP Ranges

Add an IP range for TEPs, then add the network in CIDR notification, gateway, DNS servers and DNS suffix then click Add

Then click Apply

Add a description and click Save

Now click Run Check to check the VLAN MTU

Then click Apply

And repeat for the second Edge Node, we wont need a new IP pool as we can select the new one from the drop down

Once thats done it should look like this

At the bottom we can remove the toggle to set our own passwords if we want to, when you are happy, click Save

When its done it should look like this

Before we can get our T0 gateway configured for external connectivity at the vCenter we need the uplink segments, and we need 4 to match the existing ones but for the uplink VLANs for our new workload domain 

Open the NSX GUI at the VIP address and log in with admin account

Head to Networking/Connectivity/Segment Connectivity/Segments and click Add Segment

Give it a name, and select the same transport zone as the other segments, the nsx-system-vlan-transport-zone, add the VLAN tag for uplink 1 and select the pre created teaming-1 Uplink Teaming Policy and click Save

Click No here

Repeat for three additional segments, 1 per edge per uplink VLAN
For the second VLAN, in my case 1047, use Teaming-2 like this

When your done it should look like this

Now we have our cluster, we need a new T0 gateway, if we reuse the gateway from our initial deployment, then all traffic leaving NSX will go via our other domain, which we dont want

To do this, open the NSX GUI, on the VIP FQDN and log in with the admin account

Click Networking/Connectivity/Tier-0 Gateways/Add Gateway/Tier-0

Enter a name, select the HA mode as Active/Standby, else NAT wont work causing issues with the Supervisor and K8S, select our new edge cluster and click Save

When you see this popup, click Yes, as we need to edit a few things

Expand Interfaces & GRE Tunnels and click Set

Click Add Interface

The first interface will be for the first edge on the first uplink VLAN, give it a name, add an IP in CIDR notation for that VLAN, in my case 1046, and select the segment for teaming-1 on that VLAN for edge01, then select the Edge Node, and click Save

We then need to repeat for all Edge nodes so that there is 1 interface per segment, one per VLAN per Edge

You should get something like this, click Close

Now for BGP, like before, this depends on how you have it setup, but we will be using the same concept as the management domain with a local AS number unique to this cluster, going over two BGP VLANs, which each of your ToR switches would own one each

This will not cover the BGP side outside of NSX

Expand BGP and set the local AS number, and by BGP neighbors on the right, click Set

Click Add BGP Neighbor

Add the BGP peer IP and remote AS number, then add the source addresses from the drop down that match the subnet and click Save

Then repeat for the other VLAN, your remote AS will likely be different for these peers, as mine is a single device for my lab, they ahev the same AS number

When you are done it should look like this, click Close

Now expand Route Redistribution and click Set

Click Add Route Redistribution

Click Set

Check all boxes and click Apply

Then click Add

Click Apply

Click Save

Then click Close Editing

Now we have an edge cluster, we can create a new transit gateway and connectivity profile to utilize at our workload domain

On the vSphere networking tab, right click Virtual private Clouds and click New Transit Gateway

Give the gateway a new name, leave the span on default, select Centralized Connection, for the HA mode, use Active Standby, else the Supervisor and K8S workloads will not work properly with NAT, for the Edge cluster, select our newly deployed cluster, and click Next

Select the radio button for Create New, then give the external connection a name, then select our new T0 gateway, then for VPC external blocks click the three dots and click Create New

Add a /16 block of IPs to be globally accessible over your datacenter and click Save

For Private Transit Gateway blocks click the three dots and click Create New

Add another /16 that doesnt overlap anywhere and click Save

Enable Default Outbound NAT and click Save

By default all VPCs land in the default transit gateway and connectivity profile, this is fine for the initial domain, but not so much for our workload domain

To create a new VPC click the networking tab in vSphere and right click Virtual private Clouds/New VPC

Give the VPC a name, and assign a private IP space, this wants to be another /16 that doesnt overlap anywhere else on the datacenter, this /16 can, be reused over all VPCs, which I typically recommend

Expand Advanced Settings and make sure for the connectivity profile we select the one for our new transit gateway and click Save

Now when we create subnets they will be attached to the correct Edge cluster with the local IP ranges and we dont end up with lots of cross domain routing

We’ll cover more on creating networking in the NSX configuration guide

We need to log into the SDDC Manager UI on
https://fqdn
We will then get redirected to login with the management domain vSphere SSO accounts

Click Inventory/Workload Domains and select the workload domain which is having its cluster expanded, I will be doing this on the vcf9-wld01 domain

Click Actions/Add Cluster

Enter a cluster name, and select the existing datacenter then click Next

Select our image, I will be using the same as my first workload domain cluster, then click Next

Select your storage type

For vSAN select the vSAN type, this will typically be vSAN HCI unless you know you need a vSAN storage/Compute cluster, then click Next

Select our hosts and click Next

We then need to select our networking topology, I recommend 6 NICs using the storage and NSX traffic separation, but if you have only 4 NICs, what I would consider to be the minimum, use storage separation

Now, while we have the defaults, this doesnt actually let you edit the settings, and the issue there is the NSX TEP addresses will be using DHCP when we want an IP pool like the management domain during the deployment, so click Create Custom Switch Configuration at the bottom, and we will manually set up the topology

Click Create Distributed Switch

Add a VDS name and set the MTU, this should be 9000, but in line with what was set for the management domain, for Type, I would use VDS Uplinks, we need 2 uplinks, select the vmnics you want to use then click Configure Network Traffic Type/Management
One of these uplinks should be bound to the vSwitch on the default ESX install

Give the port group a name and select the load balancing policy of route based on physical NIC load then click Save Configuration

Click Configure Network Traffic/vMotion

Give the port group a name and select the load balancing policy of route based on physical NIC load then click Save Configuration

Scroll to the bottom and click Create Distributed Switch

Click Create Distributed Switch

Give the VDS a name, set the MTU the same, which should be 9000, for the Type I recommend VDS Uplink, select the two uplinks you want to bind for storage and click Configure Network Traffic/vSAN
If you chose NFS configure that here

Give the port group a name and set the same load balancing option then click Save Configuration

Then scroll to the bottom and click Create Distributed Switch

Click Create Distributed Switch one last time

Give the VDS a name, set the MTU to the same, at 9000, for the Type I recommend VDS Uplinks, then add the remaining uplinks, then click Configure Network Traffic/NSX

Leave the default boxes checked, all three should be set the transport VLAN for NSX, and for IP Allocation, select Static IP Pool, then click Re-Use An Existing Pool and select the pool used for the first cluster in our workload domain

You can optionally create a new pool with a different VLAN if required

Give the VLAN transport zone a name, the uplink number should match the number on the VDS, in our case, 2, and set the NSX and VDS uplinks to match

Give the NSX uplink profile a name and set the Teaming Policy to Load Balance Source and click Save Configuration

Scroll down and click Create Distributed Switch

Then click Next

Review all the info and when you are happy click Next

Once the validation has passed click Finish

To scale VCF Automation out from a single node deployment to a three node medium cluster, in VCF Operations, click Build/Lifecycle/VCF Management/VCF Automation

Click Actions/Scale

Select the target size, Medium, check the box to acknowledge a potential service outage and click Scale

We can scale the Identity Broker out to a three node cluster from VCF Operations

To change this, click Build/Lifecycle/VCF Management/Identity Broker

Click Actions/Scale

Select the size as Medium, check the box to acknowledge a potential service interruption and click Scale