VCF 9 Management Services Lab Downsize

The VCF Management Services introduced with 9.1 and newer versions consolidates existing and new components into a Kubernetes cluster, and while this does bring some benefits to the table, like significantly easier scaling and availability, it has brought the unfortunately side effect of massively increased resource usage

For production environments, this is less of an issue, but if you are running labs, or even a POC, these requirements cripple the environment, and in some cases, mean you cant run VCF 9.1

Now, this is also just Kubernetes, which means, unlike normal VMware appliances, we can downsize this manually without too much effort

The control plane node, the 4vCPU/10GB VM, I couldnt reduce the size of sadly, but the worker nodes, the 12vCPU/24GB VMs, can be, and the target size for those will be 8vCPU/16GB

The one thing worth noting, with the guide, to fit in 3 smaller worker nodes, you will have to chose between having the Identity Broker, or Log Management, I opted for Log Management, so this article will cover the uninstallation of the Identity Broker

I have not covered Real-Time metrics, yet, as my lab currently has resource constraints, but I will expand some information to cover that, but it will require a 4th worker node, the good news is that you should be able get Identity Broker in with it

It should go without saying, but this is NOT even remotely supported by Broadcom, and should, under no circumstances, be applied to a production environment

I have also not tested upgrades from this, so there is no guarantee, when 9.1.1 is released, that it can be easily upgraded, in theory it should be fine, but as its not released, I dont know

1 – System Overview
1.1 – System Architecture

Under the hood, this is a K8S cluster running on an auto scaler, the more resources that are needed, the more workers that are deployed, so you dont need to worry about sizing the cluster, it will automatically handle it

The control plane size is 4vCPU/10GB

For the workers, the sizes I have seen are

  • Small – 12vCPU/24GB
  • Medium – 24vCPU/48GB

When you reach ~7 workers, the cluster will replace them with the next size up

Key services running on this are

  • VCF Service Runtime
  • Fleet LCM
  • SDDC LCM
  • Depot
  • Identity Broker
  • Salt RaaS
  • Salt Master
  • Telemetry

Additional services are

  • Log Management
  • Real-Time Metrics
1.2 – Accessing The Cluster With SSH

To access the cluster you’ll need to SSH into it, you will need the password you set during the upgrade/deployment, the [email protected] password

To then access it, in vSphere, find the control plane node, the 4vCPU/10GB VM, SSH to its IP address as the vmware-system-user, and enter the password prompted

Then switch to root with

sudo -i
1.3 – Pod List

Based on my cluster running the Log Management service only as an extra, the following pods were running and using these resources

Bare in mind this is a point in time snapshot, and doesnt factor in requests or limits by the pods, just their current usage

NamespaceNameCPU (Milicores)Memory(MiB)
kube-systemantrea-agent-7968f25m239Mi
kube-systemantrea-agent-7bxpd49m240Mi
kube-systemantrea-agent-bs5zw34m147Mi
kube-systemantrea-agent-sd8hw58m242Mi
kube-systemantrea-controller-78975945b4-95rl511m65Mi
kube-systemcoredns-5496b74c4-p9lrh8m22Mi
kube-systemcoredns-5496b74c4-rnqgt6m19Mi
kube-systemetcd-aerosphere-94pj5119m346Mi
kube-systemkube-apiserver-aerosphere-94pj5277m2311Mi
kube-systemkube-controller-manager-aerosphere-94pj569m192Mi
kube-systemkube-proxy-27vx610m31Mi
kube-systemkube-proxy-bg4lk32m29Mi
kube-systemkube-proxy-q29qp24m32Mi
kube-systemkube-proxy-rsjp913m31Mi
kube-systemkube-scheduler-aerosphere-94pj511m38Mi
kube-systemkube-vip-aerosphere-94pj54m18Mi
kube-systemvsphere-cpi-mxnnd4m22Mi
kube-systemvsphere-csi-controller-c8679dbc5-shbv98m93Mi
kube-systemvsphere-csi-node-48qf78m51Mi
kube-systemvsphere-csi-node-lmlxv2m60Mi
kube-systemvsphere-csi-node-sl8f68m62Mi
kube-systemvsphere-csi-node-v9j5b3m60Mi
ops-logslog-processor-074m3459Mi
ops-logslog-store-0139m6643Mi
saltsalt-master-8674dd877b-2xbkx31m960Mi
saltsalt-minion-6c58844558-dvwj257m127Mi
salt-raaspgdatabase-020m594Mi
salt-raasraas-5857f54b4f-n6vc436m698Mi
salt-raasredis-6cccfb46f4-q56cb4m7Mi
telemetrytelemetry-acceptor-79746d45c6-78s9q2m445Mi
vcf-fleet-depotdepot-service-77cb99b56b-jpfc823m505Mi
vcf-fleet-depotdistribution-service-84574b5cd-vfs8h1m7Mi
vcf-fleet-lcmvcf-fleet-build-service-fleetbuild-7b8464ffb8-mvc2z37m1005Mi
vcf-fleet-lcmvcf-fleet-lcm-db-018m291Mi
vcf-fleet-lcmvcf-fleet-lcm-db-127m698Mi
vcf-fleet-lcmvcf-fleet-lcm-db-26m235Mi
vcf-fleet-lcmvcf-fleet-upgrade-service-fleetupgrade-799c99f9bc-lfwbf23m1190Mi
vcf-sddc-lcmvcf-sddc-build-service-sddcbuild-79886b96bb-l8hnv124m1394Mi
vcf-sddc-lcmvcf-sddc-lcm-db-067m535Mi
vcf-sddc-lcmvcf-sddc-lcm-db-110m159Mi
vcf-sddc-lcmvcf-sddc-lcm-db-26m219Mi
vcf-sddc-lcmvcf-sddc-upgrade-service-sddcupgrade-79d4584486-8v2sg73m1043Mi
vidb-externalvidb-postgres-instance-028m1495Mi
vidb-externalvidb-service-6759897d7c-vptr982m1306Mi
vmsp-platformalertmanager-kube-prometheus-stack-alertmanager-023m47Mi
vmsp-platformargo-workflows-workflow-controller-57bbc66676-m7nz77m66Mi
vmsp-platformcapi-controller-manager-d55798b89-fp9c52m35Mi
vmsp-platformcapi-controller-manager-d55798b89-t26vr5m59Mi
vmsp-platformcapi-ipam-in-cluster-controller-manager-5dc6c6b847-k5zfp1m14Mi
vmsp-platformcapi-ipam-in-cluster-controller-manager-5dc6c6b847-lq8dw3m16Mi
vmsp-platformcapi-kubeadm-bootstrap-controller-manager-5fb45864db-cb8gn3m35Mi
vmsp-platformcapi-kubeadm-bootstrap-controller-manager-5fb45864db-gjdlw2m28Mi
vmsp-platformcapi-kubeadm-control-plane-controller-manager-ffd8fc9f-85s2b4m45Mi
vmsp-platformcapi-kubeadm-control-plane-controller-manager-ffd8fc9f-r4s5h2m31Mi
vmsp-platformcapv-controller-manager-7cd6655bff-qvbfx3m30Mi
vmsp-platformcapv-controller-manager-7cd6655bff-wk5q44m72Mi
vmsp-platformcert-manager-58d87667b-dzh6l1m35Mi
vmsp-platformcert-manager-cainjector-967cc9898-5l68x1m94Mi
vmsp-platformcert-manager-webhook-68987786cf-hl6j21m19Mi
vmsp-platformcluster-autoscaler-clusterapi-cluster-autoscaler-7c7b8d4c9qx4662m51Mi
vmsp-platformenvoy-gateway-5bb66999-jk8jq3m132Mi
vmsp-platformevent-tailer-event-tailer-01m17Mi
vmsp-platformhelm-controller-577d87cc94-jf6xt23m77Mi
vmsp-platformhooks-server-synthetic-checker-854d6b5747-tbf9l2m42Mi
vmsp-platformkube-prometheus-stack-kube-state-metrics-f87d55467-wnlm49m102Mi
vmsp-platformkube-prometheus-stack-operator-776b689c68-fkzdf1m48Mi
vmsp-platformkube-prometheus-stack-prometheus-node-exporter-lgh5b2m15Mi
vmsp-platformkube-prometheus-stack-prometheus-node-exporter-skbrc6m43Mi
vmsp-platformkube-prometheus-stack-prometheus-node-exporter-xjfmn7m16Mi
vmsp-platformkube-prometheus-stack-prometheus-node-exporter-z8p8k5m15Mi
vmsp-platformkube-vip-dr69g1m17Mi
vmsp-platformkube-vip-nfj581m16Mi
vmsp-platformkube-vip-xlsj27m21Mi
vmsp-platformlogging-operator-cc6f9f696-j4qc72m79Mi
vmsp-platformlogging-operator-filelogs-fluentbit-7fh7v1m6Mi
vmsp-platformlogging-operator-filelogs-fluentbit-nccbk1m4Mi
vmsp-platformlogging-operator-filelogs-fluentbit-sx7tw1m4Mi
vmsp-platformlogging-operator-filelogs-fluentbit-xf2tn2m7Mi
vmsp-platformlogging-operator-fluentbit-2g4955m19Mi
vmsp-platformlogging-operator-fluentbit-5s4dj4m13Mi
vmsp-platformlogging-operator-fluentbit-7jnwl11m18Mi
vmsp-platformlogging-operator-fluentbit-pxrfq5m21Mi
vmsp-platformlogging-operator-fluentd-056m630Mi
vmsp-platformlogging-operator-systemlogs-fluentbit-96l5v1m5Mi
vmsp-platformlogging-operator-systemlogs-fluentbit-fg4xm20m10Mi
vmsp-platformlogging-operator-systemlogs-fluentbit-jk7q21m5Mi
vmsp-platformlogging-operator-systemlogs-fluentbit-wsnv22m5Mi
vmsp-platformlogs-tailer-host-tailer-d6bxn1m4Mi
vmsp-platformlogs-tailer-host-tailer-dflj61m3Mi
vmsp-platformlogs-tailer-host-tailer-jtz2f2m3Mi
vmsp-platformlogs-tailer-host-tailer-znbhw2m4Mi
vmsp-platformmetadata-service-gdnvc1m21Mi
vmsp-platformmetadata-service-jf5tx1m21Mi
vmsp-platformmetadata-service-jxx9m2m22Mi
vmsp-platformmetrics-server-c77f6c7f8-lk9d83m30Mi
vmsp-platformndc-controller-manager-7547fdf4bf-jbcvb1m10Mi
vmsp-platformndc-controller-manager-7547fdf4bf-x9njg2m21Mi
vmsp-platformnode-problem-detector-cbmt91m16Mi
vmsp-platformnode-problem-detector-n85vs1m12Mi
vmsp-platformnode-problem-detector-nktz71m12Mi
vmsp-platformnode-problem-detector-qwmx96m13Mi
vmsp-platformops-logs-gateway-589859bd7-zp2k929m68Mi
vmsp-platformprometheus-kube-prometheus-stack-prometheus-026m643Mi
vmsp-platformproxy-service-778f9cb5f9-dlkrv1m18Mi
vmsp-platformseaweedfs-cleanup-54787b956c-zkvnm0m17Mi
vmsp-platformseaweedfs-filer-02m160Mi
vmsp-platformseaweedfs-master-03m69Mi
vmsp-platformseaweedfs-volume-01m41Mi
vmsp-platformseaweedfs-volume-11m60Mi
vmsp-platformseaweedfs-volume-21m71Mi
vmsp-platformsource-controller-755f68b474-dhhns5m51Mi
vmsp-platformstrimzi-cluster-operator-f7bf56558-s227j4m243Mi
vmsp-platformsupport-bundle-84989d847f-kbbbz2m21Mi
vmsp-platformsupport-bundle-cleanup-nodes-2dvnt0m4Mi
vmsp-platformsupport-bundle-cleanup-nodes-6ht9p0m4Mi
vmsp-platformsupport-bundle-cleanup-nodes-dcq2g0m4Mi
vmsp-platformsupport-bundle-cleanup-nodes-qs4fj0m3Mi
vmsp-platformsupport-bundle-logcopier-2klqn18m20Mi
vmsp-platformsupport-bundle-logcopier-cdlmt12m21Mi
vmsp-platformsupport-bundle-logcopier-dzz4t5m22Mi
vmsp-platformsupport-bundle-logcopier-event-tailer-597448947b-bpq4r1m6Mi
vmsp-platformsupport-bundle-logcopier-m2nnr132m26Mi
vmsp-platformsupport-bundle-logoffloader-7v7jn4m65Mi
vmsp-platformsupport-bundle-logoffloader-gn6c70m62Mi
vmsp-platformsupport-bundle-logoffloader-p7chj11m57Mi
vmsp-platformsupport-bundle-logoffloader-xj5mv284m62Mi
vmsp-platformsynthetic-checker-6df68966df-xkfjx24m75Mi
vmsp-platformtrust-manager-5476545759-4w7fw3m42Mi
vmsp-platformvelero-6b9655b487-r5mzl72m60Mi
vmsp-platformvmsp-agent-5458874df9-5r8rs5m191Mi
vmsp-platformvmsp-agent-configure-node-6nhfg0m2Mi
vmsp-platformvmsp-agent-configure-node-lq5nb0m2Mi
vmsp-platformvmsp-agent-configure-node-wsw5q0m2Mi
vmsp-platformvmsp-etcd-011m187Mi
vmsp-platformvmsp-gateway-6f4f75cd8-pprrc12m103Mi
vmsp-platformvmsp-identity-564649bfbb-c5wmq4m42Mi
vmsp-platformvmsp-identity-564649bfbb-sjr541m27Mi
vmsp-platformvmsp-operator-784cf768d8-m8wdp13m217Mi
vmsp-platformzalando-postgres-operator-588894b94-8kw9v2m53Mi
vmsp-platformzot-1-010m61Mi
vmsp-platformzot-1-configure-node-4jnz60m8Mi
vmsp-platformzot-1-configure-node-kprhg0m8Mi
vmsp-platformzot-1-configure-node-x8lfk0m8Mi
vmsp-platformzot-1-configure-node-zst540m7Mi
vmsp-policieskyverno-admission-controller-558cd4c545-px9s54m79Mi
vmsp-policieskyverno-admission-controller-558cd4c545-sllq58m70Mi
vmsp-policieskyverno-admission-controller-558cd4c545-sm2l618m81Mi
vmsp-policieskyverno-background-controller-67966cc6d8-s9thw15m107Mi
vmsp-policieskyverno-cleanup-controller-6c946746c7-s9g4w3m49Mi
1.4 – Pod Architecture Diagram

Under the hood of my small non HA cluster, this is what pods we have, what namespaces they are in, and where replcias sit

2 – Log Management
2.1 – Log Management Optimisation

For this, we are aiming at setting the resources to 4vCPU/8GB in line with the old lab/POC deployment size

The default YAMLs call for

Log Processor – 3vCPU/6GB with no limit on the CPU
Log Store – 5vCPU/10GB

The target we want to update these too, factoring in the JVM heap on the log store is

Log Processor – 1vCPU/3GB with no limit on the CPU allowing it to burst
Log Store – 1vCPU/6GB with a 2vCPU limit for burst – Any lower and KVM would get OOM killed

While this is a lot of memory removed on the store, this will largely be the cache data that the old log appliance used to use so the performance should be fine for a small lab environment

We can edit the log processor to this by patching the YAML with

kubectl patch statefulset log-processor -n ops-logs --type='json' -p='[
  {"op": "replace", "path": "/spec/template/spec/containers/0/resources/requests/cpu", "value": "1"},
  {"op": "add", "path": "/spec/template/spec/containers/0/resources/limits/cpu", "value": "2"},
  {"op": "replace", "path": "/spec/template/spec/containers/0/resources/requests/memory", "value": "2Gi"},
  {"op": "replace", "path": "/spec/template/spec/containers/0/resources/limits/memory", "value": "3Gi"}
]'

And the log store with this
O opted for 3 cores as a limit for the store, I did have this at 2, but queries were slow, and this over provisioning should be fine since the cluster is mostly idle

kubectl patch statefulset log-store -n ops-logs --type='json' -p='[
  {"op": "replace", "path": "/spec/template/spec/containers/0/resources/requests/cpu", "value": "1"},
  {"op": "replace", "path": "/spec/template/spec/containers/0/resources/limits/cpu", "value": "3"},
  {"op": "replace", "path": "/spec/template/spec/containers/0/resources/requests/memory", "value": "4.5Gi"},
  {"op": "replace", "path": "/spec/template/spec/containers/0/resources/limits/memory", "value": "5Gi"},
  {"op": "replace", "path": "/spec/template/spec/containers/0/env/5/value", "value": "-Xms3584m -Xmx3584m -Djavax.net.ssl.trustStore=/usr/share/opensearch/config/truststore/bundle.jks -Djavax.net.ssl.trustStorePassword=changeit -Djava.util.logging.config.file=/usr/share/opensearch/config/bc.logging.properties"}
]'

This should start a restart which we can check on with

kubectl rollout status statefulset/log-processor -n ops-logs
kubectl rollout status statefulset/log-store -n ops-logs

We can also view the pods with, this will take a while to restart

kubectl top pods -n ops-logs

I did get to this point where the log processor was idle and couldnt get resources from the log-store like this, and it didnt seem to recover

We can recreate it with

kubectl delete pod log-processor-0 -n ops-logs

Which kicked it back into life now the log store is up and running, if you left it long enough the system may recover it

2.2 Original YAML Backups
2.2.1 – log-processor.yaml
apiVersion: apps/v1
kind: StatefulSet
metadata:
  annotations:
    meta.helm.sh/release-name: vcf-ops-logs
    meta.helm.sh/release-namespace: ops-logs
  creationTimestamp: "2026-05-12T16:04:45Z"
  generation: 1
  labels:
    app.kubernetes.io/instance: vcf-ops-logs
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: vcf-ops-logs
    app.kubernetes.io/version: 0.0.1574
    helm.sh/chart: vcf-ops-logs-0.0.1574
    helm.toolkit.fluxcd.io/name: vcf-ops-logs
    helm.toolkit.fluxcd.io/namespace: ops-logs
    monitoring: vcf-ops-logs-app
    ops.vmsp.vmware.com/type: log-processor-service
  name: log-processor
  namespace: ops-logs
  resourceVersion: "13212589"
  uid: 0e479d18-21a5-4753-92e6-c868482dfdc8
spec:
  persistentVolumeClaimRetentionPolicy:
    whenDeleted: Retain
    whenScaled: Retain
  podManagementPolicy: Parallel
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app.kubernetes.io/instance: vcf-ops-logs
      app.kubernetes.io/name: vcf-ops-logs
  serviceName: log-processor-svc
  template:
    metadata:
      annotations:
        identity.vmsp.vmware.com/serviceaccount: opslogs-client
      labels:
        app.kubernetes.io/instance: vcf-ops-logs
        app.kubernetes.io/name: vcf-ops-logs
        monitoring: vcf-ops-logs-app
        ops.vmsp.vmware.com/type: log-processor-resource
    spec:
      affinity:
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
          - podAffinityTerm:
              labelSelector:
                matchLabels:
                  app.kubernetes.io/instance: vcf-ops-logs
                  app.kubernetes.io/name: vcf-ops-logs
              topologyKey: kubernetes.io/hostname
            weight: 100
      containers:
      - envFrom:
        - configMapRef:
            name: ops-logs-config
        image: registry.vmsp-platform.svc.cluster.local:5000/images/ops-logs/9.1.0.0.25346055/ops-logs/vcf-ops-logs:0.0.1574
        imagePullPolicy: IfNotPresent
        livenessProbe:
          failureThreshold: 15
          httpGet:
            path: /healthz
            port: web-nonssl
            scheme: HTTP
          initialDelaySeconds: 45
          periodSeconds: 5
          successThreshold: 1
          timeoutSeconds: 1
        name: vcf-ops-logs
        ports:
        - containerPort: 9000
          name: cfapi-nonssl
          protocol: TCP
        - containerPort: 514
          name: syslog-nonssl
          protocol: TCP
        - containerPort: 9543
          name: cfapi-ssl
          protocol: TCP
        - containerPort: 1514
          name: syslog-ssl
          protocol: TCP
        - containerPort: 8080
          name: web-nonssl
          protocol: TCP
        readinessProbe:
          failureThreshold: 15
          httpGet:
            path: /readyz
            port: web-nonssl
            scheme: HTTP
          initialDelaySeconds: 45
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 5
        resources:
          limits:
            ephemeral-storage: 2Gi
            memory: 6Gi
          requests:
            cpu: "3"
            ephemeral-storage: 2Gi
            memory: 6Gi
        securityContext:
          capabilities:
            drop:
            - CAP_CHOWN
            - CAP_DAC_OVERRIDE
            - CAP_FOWNER
            - CAP_NET_RAW
            - CAP_MKNOD
        startupProbe:
          failureThreshold: 60
          httpGet:
            path: /healthz
            port: web-nonssl
            scheme: HTTP
          initialDelaySeconds: 45
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 1
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /opt/vmware/ops-logs/data
          name: opslogs-data
        - mountPath: /opt/vmware/ops-logs/debug
          name: opslogs-debug
        - mountPath: /opt/vmware/certs/opensearch
          name: opensearch-certs
        - mountPath: /opt/vmware/certs/ops-logs
          name: opslogs-certs
        - mountPath: /opt/vmware/certs/ingress
          name: ingress-certs
        - mountPath: /opt/vmware/certs/auth
          name: opslogs-auth
        - mountPath: /opt/vmware/secrets
          name: logs-secrets
          readOnly: true
        - mountPath: /opt/vmware/security/platform-trust
          name: vmsp-certs
          readOnly: true
      - image: registry.vmsp-platform.svc.cluster.local:5000/images/ops-logs/9.1.0.0.25346055/ops-logs/vcf-ops-logs-ui:0.0.1574
        imagePullPolicy: IfNotPresent
        livenessProbe:
          failureThreshold: 3
          httpGet:
            path: /nginx-health
            port: http-ui
            scheme: HTTP
          initialDelaySeconds: 10
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 5
        name: ui-server
        ports:
        - containerPort: 8081
          name: http-ui
          protocol: TCP
        - containerPort: 8444
          name: https-ui
          protocol: TCP
        readinessProbe:
          failureThreshold: 3
          httpGet:
            path: /nginx-health
            port: http-ui
            scheme: HTTP
          initialDelaySeconds: 10
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 5
        resources:
          limits:
            cpu: 200m
            memory: 256Mi
          requests:
            cpu: 100m
            memory: 128Mi
        securityContext:
          capabilities:
            drop:
            - CAP_CHOWN
            - CAP_DAC_OVERRIDE
            - CAP_FOWNER
            - CAP_NET_RAW
            - CAP_MKNOD
          runAsNonRoot: true
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /etc/nginx/conf.d
          name: ui-nginx-config
          readOnly: true
        - mountPath: /etc/nginx/certs
          name: ui-certs
          readOnly: true
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext:
        fsGroup: 3000
        runAsGroup: 3000
        runAsUser: 2000
      serviceAccount: log-processor
      serviceAccountName: log-processor
      terminationGracePeriodSeconds: 240
      volumes:
      - name: opensearch-certs
        secret:
          defaultMode: 384
          secretName: tls-for-opensearch-key-pair
      - name: opensearch-admin-certs
        secret:
          defaultMode: 384
          secretName: tls-for-opensearch-admin-key-pair
      - name: opslogs-certs
        secret:
          defaultMode: 384
          secretName: tls-for-opslogs-key-pair
      - name: ingress-certs
        secret:
          defaultMode: 384
          secretName: ingress-service-cert
      - name: opslogs-auth
        secret:
          defaultMode: 384
          secretName: auth-keypair-secret
      - name: logs-secrets
        secret:
          defaultMode: 384
          secretName: ops-logs-secrets
      - configMap:
          defaultMode: 384
          items:
          - key: bundle.pem
            path: bundle.pem
          name: platform-trust
        name: vmsp-certs
      - configMap:
          defaultMode: 384
          name: log-processor-ui-nginx-config
        name: ui-nginx-config
      - name: ui-certs
        secret:
          defaultMode: 384
          secretName: tls-for-opslogs-key-pair
  updateStrategy:
    rollingUpdate:
      partition: 0
    type: RollingUpdate
  volumeClaimTemplates:
  - apiVersion: v1
    kind: PersistentVolumeClaim
    metadata:
      labels:
        velero.io/exclude-from-backup: "true"
        volume-groups.vmsp.vmware.com/resizable: "false"
      name: opslogs-debug
    spec:
      accessModes:
      - ReadWriteOnce
      resources:
        requests:
          storage: 15Gi
      volumeMode: Filesystem
    status:
      phase: Pending
  - apiVersion: v1
    kind: PersistentVolumeClaim
    metadata:
      labels:
        velero.io/exclude-from-backup: "true"
        volume-groups.vmsp.vmware.com/resizable: "false"
      name: opslogs-data
    spec:
      accessModes:
      - ReadWriteOnce
      resources:
        requests:
          storage: 60Gi
      volumeMode: Filesystem
    status:
      phase: Pending
status:
  availableReplicas: 1
  collisionCount: 0
  currentReplicas: 1
  currentRevision: log-processor-774bcc4967
  observedGeneration: 1
  readyReplicas: 1
  replicas: 1
  updateRevision: log-processor-774bcc4967
  updatedReplicas: 1
2.2.2 – log-store.yaml
apiVersion: apps/v1
kind: StatefulSet
metadata:
  annotations:
    majorVersion: "3"
    meta.helm.sh/release-name: vcf-ops-logs
    meta.helm.sh/release-namespace: ops-logs
  creationTimestamp: "2026-05-12T16:04:45Z"
  generation: 2
  labels:
    app.kubernetes.io/component: log-store
    app.kubernetes.io/instance: vcf-ops-logs
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: opensearch
    app.kubernetes.io/version: 3.0.0
    helm.sh/chart: opensearch-3.0.0
    helm.toolkit.fluxcd.io/name: vcf-ops-logs
    helm.toolkit.fluxcd.io/namespace: ops-logs
    ops.vmsp.vmware.com/type: log-store-service
  name: log-store
  namespace: ops-logs
  resourceVersion: "13211289"
  uid: 5859f867-be7b-42cc-9cf1-0a33fa15c58d
spec:
  persistentVolumeClaimRetentionPolicy:
    whenDeleted: Retain
    whenScaled: Retain
  podManagementPolicy: Parallel
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app.kubernetes.io/instance: vcf-ops-logs
      app.kubernetes.io/name: opensearch
  serviceName: log-store-headless
  template:
    metadata:
      annotations:
        configchecksum: 88acb1d6fcb84e84077a20e3544fdf5c07fee8c04a7fcd3e5d35d28f6d76294
      labels:
        app.kubernetes.io/component: log-store
        app.kubernetes.io/instance: vcf-ops-logs
        app.kubernetes.io/managed-by: Helm
        app.kubernetes.io/name: opensearch
        app.kubernetes.io/version: 3.0.0
        helm.sh/chart: opensearch-3.0.0
        ops.vmsp.vmware.com/type: log-store-resource
      name: log-store
    spec:
      affinity:
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
          - podAffinityTerm:
              labelSelector:
                matchExpressions:
                - key: app.kubernetes.io/instance
                  operator: In
                  values:
                  - vcf-ops-logs
                - key: app.kubernetes.io/name
                  operator: In
                  values:
                  - opensearch
              topologyKey: kubernetes.io/hostname
            weight: 1
      automountServiceAccountToken: false
      containers:
      - env:
        - name: node.name
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.name
        - name: cluster.initial_master_nodes
          value: log-store-0,
        - name: discovery.seed_hosts
          value: log-store-headless
        - name: cluster.name
          value: log-store
        - name: network.host
          value: 0.0.0.0
        - name: OPENSEARCH_JAVA_OPTS
          value: -Xms5120m -Xmx5120m -Djavax.net.ssl.trustStore=/usr/share/opensearch/config/truststore/bundle.jks
            -Djavax.net.ssl.trustStorePassword=changeit -Djava.util.logging.config.file=/usr/share/opensearch/config/bc.logging.properties
        - name: node.roles
          value: master,ingest,data,remote_cluster_client,warm,
        - name: DISABLE_INSTALL_DEMO_CONFIG
          value: "true"
        - name: LOG_PROCESSOR_USER
          value: log-processor-user
        - name: LOG_PROCESSOR_ROLE
          value: log-processor-role
        - name: KEYSTORE_UTIL_SECRET
          valueFrom:
            configMapKeyRef:
              key: LOGS_OPENSEARCH_KEYSTORE_UTIL
              name: ops-logs-config
        - name: CLIENT_NAME
          valueFrom:
            configMapKeyRef:
              key: CLIENT_NAME
              name: ops-logs-config
        image: registry.vmsp-platform.svc.cluster.local:5000/images/ops-logs/9.1.0.0.25346055/ops-logs/opensearchproject/log-store:3.0.0-fips-1.8
        imagePullPolicy: IfNotPresent
        name: opensearch
        ports:
        - containerPort: 9200
          name: http
          protocol: TCP
        - containerPort: 9300
          name: transport
          protocol: TCP
        - containerPort: 9600
          name: metrics
          protocol: TCP
        readinessProbe:
          failureThreshold: 3
          periodSeconds: 5
          successThreshold: 1
          tcpSocket:
            port: 9200
          timeoutSeconds: 3
        resources:
          limits:
            cpu: "5"
            ephemeral-storage: 2Gi
            memory: 10Gi
          requests:
            cpu: "5"
            ephemeral-storage: 2Gi
            memory: 10Gi
        securityContext:
          capabilities:
            drop:
            - ALL
          runAsNonRoot: true
          runAsUser: 1000
        startupProbe:
          failureThreshold: 30
          initialDelaySeconds: 5
          periodSeconds: 10
          successThreshold: 1
          tcpSocket:
            port: 9200
          timeoutSeconds: 3
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /usr/share/opensearch/data
          name: log-store
        - mountPath: /usr/share/opensearch/config/certs
          name: tls-for-opensearch-key-pair
        - mountPath: /usr/share/opensearch/config/certs/admin
          name: tls-for-opensearch-admin-key-pair
        - mountPath: /usr/share/opensearch/config/certs/log-processor
          name: tls-for-opslogs-key-pair
        - mountPath: /usr/share/opensearch/config/ops-logs-secrets
          name: ops-logs-secrets-mount
        - mountPath: /usr/share/opensearch/config/s3-secrets
          name: s3-secret-mount
        - mountPath: /usr/share/opensearch/config/s3-backup-restore-secrets
          name: s3-backup-restore-secret-mount
        - mountPath: /usr/share/opensearch/config/opensearch.yml
          name: config-emptydir
          subPath: opensearch.yml
        - mountPath: /usr/share/opensearch/cayman-opensearch-util.sh
          name: cayman-opensearch-util
          readOnly: true
          subPath: cayman-opensearch-util.sh
        - mountPath: /usr/share/opensearch/config/bc.logging.properties
          name: cayman-opensearch-bc-logging
          readOnly: true
          subPath: bc.logging.properties
        - mountPath: /usr/share/opensearch/config/truststore
          name: truststore-volume
          readOnly: true
      dnsPolicy: ClusterFirst
      enableServiceLinks: true
      initContainers:
      - args:
        - chown -R 1000:1000 /usr/share/opensearch/data
        command:
        - sh
        - -c
        image: registry.vmsp-platform.svc.cluster.local:5000/images/ops-logs/9.1.0.0.25346055/ops-logs/busybox:latest
        imagePullPolicy: IfNotPresent
        name: fsgroup-volume
        resources:
          limits:
            cpu: 100m
            memory: 128Mi
          requests:
            cpu: 100m
            memory: 128Mi
        securityContext:
          runAsUser: 0
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /usr/share/opensearch/data
          name: log-store
      - command:
        - sh
        - -c
        - |
          #!/usr/bin/env bash
          cp -r /tmp/configfolder/*  /tmp/config/
        image: registry.vmsp-platform.svc.cluster.local:5000/images/ops-logs/9.1.0.0.25346055/ops-logs/opensearchproject/log-store:3.0.0-fips-1.8
        imagePullPolicy: IfNotPresent
        name: configfile
        resources:
          limits:
            cpu: 100m
            memory: 128Mi
          requests:
            cpu: 100m
            memory: 128Mi
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /tmp/config/
          name: config-emptydir
        - mountPath: /tmp/configfolder/opensearch.yml
          name: config
          subPath: opensearch.yml
      priorityClassName: high
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext:
        fsGroup: 1000
        runAsUser: 1000
      serviceAccount: log-store-sa
      serviceAccountName: log-store-sa
      terminationGracePeriodSeconds: 240
      volumes:
      - configMap:
          defaultMode: 420
          name: log-store-config
        name: config
      - emptyDir: {}
        name: config-emptydir
      - name: tls-for-opensearch-key-pair
        secret:
          defaultMode: 384
          secretName: tls-for-opensearch-key-pair
      - name: tls-for-opensearch-admin-key-pair
        secret:
          defaultMode: 384
          secretName: tls-for-opensearch-admin-key-pair
      - name: tls-for-opslogs-key-pair
        secret:
          defaultMode: 384
          secretName: tls-for-opslogs-key-pair
      - name: ops-logs-secrets-mount
        secret:
          defaultMode: 384
          secretName: ops-logs-secrets
      - name: s3-secret-mount
        secret:
          defaultMode: 384
          secretName: s3-secret
      - name: s3-backup-restore-secret-mount
        secret:
          defaultMode: 384
          secretName: ops-logs-bucket-credentials
      - configMap:
          defaultMode: 384
          name: cayman-opensearch-util-script
        name: cayman-opensearch-util
      - configMap:
          defaultMode: 384
          name: cayman-opensearch-bc-logging
        name: cayman-opensearch-bc-logging
      - configMap:
          defaultMode: 384
          items:
          - key: bundle.jks
            path: bundle.jks
          name: platform-trust-no-default
        name: truststore-volume
  updateStrategy:
    type: RollingUpdate
  volumeClaimTemplates:
  - apiVersion: v1
    kind: PersistentVolumeClaim
    metadata:
      labels:
        app.kubernetes.io/component: log-store
        app.kubernetes.io/instance: vcf-ops-logs
        app.kubernetes.io/managed-by: Helm
        app.kubernetes.io/name: opensearch
        app.kubernetes.io/version: 3.0.0
        helm.sh/chart: opensearch-3.0.0
        vcf.vmware.com/keep-during-restore: "true"
        velero.io/exclude-from-backup: "true"
      name: log-store
    spec:
      accessModes:
      - ReadWriteOnce
      resources:
        requests:
          storage: 500Gi
      volumeMode: Filesystem
    status:
      phase: Pending
status:
  availableReplicas: 1
  collisionCount: 0
  currentReplicas: 1
  currentRevision: log-store-95968949d
  observedGeneration: 2
  readyReplicas: 1
  replicas: 1
  updateRevision: log-store-95968949d
  updatedReplicas: 1
3 – Ops Log Gateway
3.1 – Ops Log Gateway Optimisation

This service was the best place to start reducing vCPU, its got 4 cores assigned but is actively using less than 100m, memory wise it was under 100MiB with an assigned 2GiB

The log gateway is managed by the envoy gateway, so we need to edit it there

These resources are managed by a helm package, so while this change will effect the system, it will revert on an update

I opted for 400m CPU as I tried 250m, but it sat at that for a while, I assume doing background processing, which may make sense that it can use a lot given it had 4000m assigned before

kubectl patch envoyproxy ops-logs-gateway-config -n vmsp-platform --type='json' -p='[
  {"op": "replace", "path": "/spec/provider/kubernetes/envoyDeployment/patch/value/spec/template/spec/containers/0/resources/requests/cpu", "value": "400m"},
  {"op": "replace", "path": "/spec/provider/kubernetes/envoyDeployment/patch/value/spec/template/spec/containers/0/resources/requests/memory", "value": "256Mi"},
  {"op": "replace", "path": "/spec/provider/kubernetes/envoyDeployment/patch/value/spec/template/spec/containers/0/resources/limits/memory", "value": "512Mi"}
]'

The pods will be auto recreated

3.2 – Original YAML Backups
3.2.1 – envoy-proxy.yaml
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: EnvoyProxy
metadata:
  annotations:
    meta.helm.sh/release-name: ops-logs-gateway
    meta.helm.sh/release-namespace: vmsp-platform
  creationTimestamp: "2026-05-12T16:04:50Z"
  generation: 1
  labels:
    app.kubernetes.io/managed-by: Helm
    helm.toolkit.fluxcd.io/name: ops-logs-gateway
    helm.toolkit.fluxcd.io/namespace: vmsp-platform
  name: ops-logs-gateway-config
  namespace: vmsp-platform
  resourceVersion: "176946"
  uid: 1f3c50f1-4e24-4b87-9578-076a35b941a8
spec:
  logging:
    level:
      default: warn
  provider:
    kubernetes:
      envoyDeployment:
        container:
          volumeMounts:
          - mountPath: /tmp
            name: shutdown-manager
        name: ops-logs-gateway
        patch:
          type: StrategicMerge
          value:
            metadata:
              labels:
                ops.vmsp.vmware.com/type: envoygatewayproxy-service
            spec:
              template:
                metadata:
                  labels:
                    ops.vmsp.vmware.com/type: envoygatewayproxy-service
                spec:
                  affinity:
                    podAntiAffinity:
                      preferredDuringSchedulingIgnoredDuringExecution:
                      - podAffinityTerm:
                          labelSelector:
                            matchLabels:
                              gateway.envoyproxy.io/owning-gateway-name: ops-logs-gateway
                          topologyKey: kubernetes.io/hostname
                        weight: 100
                  containers:
                  - image: registry.vmsp-platform.svc.cluster.local:5000/images/envoy:v1.34.1
                    name: envoy
                    resources:
                      limits:
                        memory: 2Gi
                      requests:
                        cpu: 4000m
                        memory: 2Gi
                    volumeMounts:
                    - mountPath: /etc/ssl/certs
                      name: config-volume
                  - image: registry.vmsp-platform.svc.cluster.local:5000/images/envoyproxy-gateway:v1.5.0
                    name: shutdown-manager
                    resources:
                      limits:
                        cpu: 50m
                        memory: 256Mi
                      requests:
                        cpu: 5m
                        memory: 32Mi
                    volumeMounts:
                    - mountPath: /tmp
                      name: shutdown-manager
                  priorityClassName: vmsp-high
        pod:
          volumes:
          - emptyDir:
              medium: Memory
            name: shutdown-manager
          - configMap:
              items:
              - key: bundle.pem
                path: ca-certificates.crt
              name: platform-trust
            name: config-volume
        replicas: 1
      envoyService:
        allocateLoadBalancerNodePorts: false
        externalTrafficPolicy: Local
        loadBalancerClass: envoy
        name: ops-logs-gateway
        type: LoadBalancer
    type: Kubernetes
  telemetry:
    accessLog:
      disable: true
4 – Identity Broker Uninstallation

Unfortunately, we cant get the resources low enough with the log management server installed to downsize the worker nodes, so if you want to drop them, you’ll need to uninstall the Identity Broker, or Log Management

Broadcom have a python script that can be used to cleanup components which lists the Identity Broker, so we will be using that for this, which can be found here

Download the script that the bottom

I am running this on Ubuntu, you’ll need to be in the directory of the script and run

chmod +x ./cleanup_component.py

Then to check components, we can run

python3 cleanup_component.py list vsp-component --fleet-fqdn <fleet-lcm-fqdn> --vcf-services-runtime-fqdn <vcf-services-runtime-fqdn> --vcf-services-runtime-username <vcf-services-runtime-username>

For my system, which has the service runtime of aerosphere.leaha.co.uk, and a fleet service fqdn of, skyveil.leaha.co.uk, I ran the following

python3 cleanup_component.py delete vsp-component --component-id aceb5c33-d3ab-49cc-b4c4-9aecfbfe15d9 --fleet-fqdn skyveil.leaha.co.uk --vcf-services-runtime-fqdn aerosphere.leaha.co.uk --vcf-services-runtime-username admin

You’ll be asked for a password, which from the deployment/upgrade should be the same as the vmware-system-user, and you’ll see an output like this

Enter password for user 'admin': 

Authenticating with VSP platform (aerosphere.leaha.co.uk)...
Authentication successful.
Fetching VSP components from Fleet...

Available VSP Components:
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
ID                                   | Type                        | Deployment | Version          | FQDN                                 | VCF Services Runtime FQDN
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
aceb5c33-d3ab-49cc-b4c4-9aecfbfe15d9 | Identity broker             | VSP        | 9.1.0.0.25368698 | nimbus.leaha.co.uk                   | aerosphere.leaha.co.uk
95e53a26-3f09-4284-a53f-3c64ca7c48fe | Log management              | VSP        | 9.1.0.0.25346055 | mistral.leaha.co.uk                  | aerosphere.leaha.co.uk
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Here we can see the Identity Broker component

The generic command to remove this is

python3 cleanup_component.py delete vsp-component --component-id <vsp-component-id> --fleet-fqdn <fleet-lcm-fqdn> --vcf-services-runtime-fqdn <vcf-services-runtime-fqdn> --vcf-services-runtime-username <vcf-services-runtime-username>

For my environment it was

python3 cleanup_component.py delete vsp-component --component-id aceb5c33-d3ab-49cc-b4c4-9aecfbfe15d9 --fleet-fqdn skyveil.leaha.co.uk --vcf-services-runtime-fqdn aerosphere.leaha.co.uk --vcf-services-runtime-username admin

You’ll be asked to press Y to confirm

5 – LCM Postgres
5.1 – LCM Postgres Optimisation

There is both the Fleet and SDDC lifecycle that have a three node HA DB on them, now the memory seems fine on them, but the CPU is 500m for both, with three instances each totaling 3000m, meanwhile the usage is in the single or double digits, so aiming for 200m brings the total down to 1200m

We can set the CPU requests to 200m for both SDDC lifecycle

kubectl patch postgresql vcf-sddc-lcm-db -n vcf-sddc-lcm --type='json' -p='[
  {"op": "replace", "path": "/spec/resources/requests/cpu", "value": "200m"}
]'

And fleet lifecycle

kubectl patch postgresql vcf-fleet-lcm-db -n vcf-fleet-lcm --type='json' -p='[
  {"op": "replace", "path": "/spec/resources/requests/cpu", "value": "200m"}
]'
5.2 – Original YAML Backups
5.2.1 – vcf-fleet-lcm-db.yaml
apiVersion: acid.zalan.do/v1
kind: postgresql
metadata:
  annotations:
    vmsp.vmware.com/delete-protection: "true"
  creationTimestamp: "2026-05-12T13:50:10Z"
  generation: 2
  labels:
    ops.vmsp.vmware.com/type: pgdatabase
    velero.io/exclude-from-backup: "true"
  name: vcf-fleet-lcm-db
  namespace: vcf-fleet-lcm
  resourceVersion: "13213812"
  uid: b1cae9df-2a4d-4ddd-9f42-e5846ca2565a
spec:
  additionalVolumes:
  - mountPath: /usr/local/share/ca-certificates/client/
    name: seaweedfs-client-cert
    targetContainers:
    - postgres
    - walg
    volumeSource:
      secret:
        optional: false
        secretName: seaweedfs-client-cert
  - mountPath: /usr/local/share/ca-certificates/tls/
    name: pgbackup-sidecar-tls-cert
    targetContainers:
    - walg
    volumeSource:
      secret:
        optional: false
        secretName: vcf-fleet-lcm-db-postgres-cert-secret
  allowedSourceRanges: null
  enableConnectionPooler: false
  enableReplicaConnectionPooler: false
  env:
  - name: AWS_ACCESS_KEY_ID
    valueFrom:
      secretKeyRef:
        key: access_key
        name: vcf-fleet-lcm.db.vcf-fleet-lcm-db.wal-backup-credentials
  - name: AWS_SECRET_ACCESS_KEY
    valueFrom:
      secretKeyRef:
        key: secret_key
        name: vcf-fleet-lcm.db.vcf-fleet-lcm-db.wal-backup-credentials
  - name: AWS_ENDPOINT
    value: https://seaweedfs-s3.vmsp-platform.svc.cluster.local:8443
  - name: WALG_S3_CA_CERT_FILE
    value: /usr/local/share/ca-certificates/client/ca.crt
  - name: USE_WALG_BACKUP
    value: "true"
  - name: USE_WALG_RESTORE
    value: "true"
  - name: CLONE_USE_WALG_RESTORE
    value: "true"
  - name: CLONE_WAL_BUCKET_SCOPE_PREFIX
  - name: BACKUP_NUM_TO_RETAIN
    value: "1"
  - name: WAL_S3_BUCKET
    value: vcf-fleet-lcm.db.vcf-fleet-lcm-db.wal-backup
  - name: AWS_S3_FORCE_PATH_STYLE
    value: "true"
  - name: BACKUP_SCHEDULE
    value: '#0 25 31 2 * '
  nodeAffinity:
    preferredDuringSchedulingIgnoredDuringExecution:
    - preference:
        matchExpressions:
        - key: node.kubernetes.io/instance-type
          operator: In
          values:
          - vsphere-vm.cpu-12.mem-24gb.os-photon
      weight: 1
  numberOfInstances: 3
  patroni:
    failsafe_mode: true
    retry_timeout: 20
    synchronous_mode: true
    synchronous_mode_strict: true
    synchronous_node_count: 1
    ttl: 50
  postgresql:
    parameters:
      archive_cleanup_command: curl -sf --connect-timeout 1 --max-time 5 http://127.0.0.1:8008/replica
        > /dev/null && envdir "/run/etc/wal-e.d/env" pg_archivecleanup "/home/postgres/pgdata/pgroot/data/pg_wal"
        "%r" || exit 0
      archive_command: (curl -sf --connect-timeout 1 --max-time 5 http://127.0.0.1:8008/primary
        > /dev/null || exit 0) && envdir "/run/etc/wal-e.d/env" wal-g wal-push "%p"
      archive_timeout: 900s
      client_min_messages: error
      log_destination: stderr
      log_line_prefix: '%m %c %x %d %u %r %p %l'
      log_parameter_max_length: "0"
      log_parameter_max_length_on_error: "0"
      logging_collector: "off"
      max_connections: "500"
      password_encryption: scram-sha-256
      pgaudit.log: none
      pgaudit.log_catalog: "off"
      pgaudit.log_parameter: "off"
      restore_command: 'envdir "/run/etc/wal-e.d/env" /bin/bash -c "echo $(date -u
        +%%Y-%%m-%%d_%%H:%%M:%%S_UTC) rc: start %f >&2; if ! timeout 1s bash -c ''exec
        3<>/dev/tcp/seaweedfs-s3.vmsp-platform.svc.cluster.local/8443''; then echo
        $(date -u +%%Y-%%m-%%d_%%H:%%M:%%S_UTC) rc: preflight-fail https://seaweedfs-s3.vmsp-platform.svc.cluster.local:8443
        >&2; exit 1; fi; if timeout 30s /scripts/restore_command.sh \"%f\" \"%p\";
        then if [ -s \"%p\" ]; then echo $(date -u +%%Y-%%m-%%d_%%H:%%M:%%S_UTC) rc:
        success %f >&2; exit 0; else echo $(date -u +%%Y-%%m-%%d_%%H:%%M:%%S_UTC)
        rc: empty-file %f >&2; exit 1; fi; else rc=$?; if [ \"${rc:-0}\" -eq 124 ];
        then echo $(date -u +%%Y-%%m-%%d_%%H:%%M:%%S_UTC) rc: fetch-timeout %f >&2;
        else echo $(date -u +%%Y-%%m-%%d_%%H:%%M:%%S_UTC) rc: fetch-fail %f rc=${rc:-?}
        >&2; fi; exit 1; fi"'
      shared_preload_libraries: bg_mon,pg_stat_statements,pgextwlist,pg_auth_mon,set_user,timescaledb,pg_cron,pg_stat_kcache
      statement_timeout: "30000"
      synchronous_commit: "on"
    version: "16"
  preparedDatabases:
    vcffleetlcmdb:
      defaultUsers: true
      schemas:
        public: {}
  resources:
    limits:
      cpu: "1"
      memory: 3048Mi
    requests:
      cpu: 500m
      memory: 512Mi
  sidecars:
  - env:
    - name: DATA_SOURCE_URI
      value: localhost:5432/postgres?sslmode=disable
    - name: DATA_SOURCE_USER
      value: $(POSTGRES_USER)
    - name: DATA_SOURCE_PASS
      value: $(POSTGRES_PASSWORD)
    image: registry.vmsp-platform.svc.cluster.local:5000/images/prometheuscommunity-postgres-exporter:v0.18.1
    name: metrics-exporter
    ports:
    - containerPort: 9187
      name: http-metrics
      protocol: TCP
  - env:
    - name: PGPORT
      value: "5432"
    - name: PGHOST
      value: vcf-fleet-lcm-db.vcf-fleet-lcm.svc.cluster.local
    - name: WALE_S3_PREFIX
      value: s3://vcf-fleet-lcm.db.vcf-fleet-lcm-db.wal-backup/spilo/vcf-fleet-lcm-vcf-fleet-lcm-db/wal/16/
    - name: AWS_ACCESS_KEY_ID
      valueFrom:
        secretKeyRef:
          key: access_key
          name: vcf-fleet-lcm.db.vcf-fleet-lcm-db.wal-backup-credentials
    - name: AWS_SECRET_ACCESS_KEY
      valueFrom:
        secretKeyRef:
          key: secret_key
          name: vcf-fleet-lcm.db.vcf-fleet-lcm-db.wal-backup-credentials
    - name: AWS_ENDPOINT
      value: https://seaweedfs-s3.vmsp-platform.svc.cluster.local:8443
    - name: WALG_S3_CA_CERT_FILE
      value: /usr/local/share/ca-certificates/client/ca.crt
    - name: USE_WALG_BACKUP
      value: "true"
    - name: USE_WALG_RESTORE
      value: "true"
    - name: CLONE_USE_WALG_RESTORE
      value: "true"
    - name: CLONE_WAL_BUCKET_SCOPE_PREFIX
    - name: BACKUP_NUM_TO_RETAIN
      value: "1"
    - name: WAL_S3_BUCKET
      value: vcf-fleet-lcm.db.vcf-fleet-lcm-db.wal-backup
    - name: AWS_S3_FORCE_PATH_STYLE
      value: "true"
    - name: BACKUP_SCHEDULE
      value: '#0 25 31 2 * '
    image: registry.vmsp-platform.svc.cluster.local:5000/images/pgbackup:v1.0.0
    name: walg
    ports:
    - containerPort: 9100
      name: walg
      protocol: TCP
    resources:
      limits:
        cpu: 1000m
        memory: 768Mi
      requests:
        cpu: 100m
        memory: 100Mi
  teamId: vcf-fleet-lcm-db
  tls:
    caFile: ca.crt
    secretName: vcf-fleet-lcm-db-postgres-cert-secret
  users:
    postgres:
    - superuser
    - createdb
  volume:
    size: 30Gi
status:
  PostgresClusterStatus: Running
5.2.2 – vcf-sddc-lcm-db.yaml
apiVersion: acid.zalan.do/v1
kind: postgresql
metadata:
  annotations:
    vmsp.vmware.com/delete-protection: "true"
  creationTimestamp: "2026-05-12T13:51:09Z"
  generation: 2
  labels:
    ops.vmsp.vmware.com/type: pgdatabase
    velero.io/exclude-from-backup: "true"
  name: vcf-sddc-lcm-db
  namespace: vcf-sddc-lcm
  resourceVersion: "13213769"
  uid: 107f3f96-c4eb-47b8-8f34-9ab0b4d18e8b
spec:
  additionalVolumes:
  - mountPath: /usr/local/share/ca-certificates/client/
    name: seaweedfs-client-cert
    targetContainers:
    - postgres
    - walg
    volumeSource:
      secret:
        optional: false
        secretName: seaweedfs-client-cert
  - mountPath: /usr/local/share/ca-certificates/tls/
    name: pgbackup-sidecar-tls-cert
    targetContainers:
    - walg
    volumeSource:
      secret:
        optional: false
        secretName: vcf-sddc-lcm-db-postgres-cert-secret
  allowedSourceRanges: null
  enableConnectionPooler: false
  enableReplicaConnectionPooler: false
  env:
  - name: AWS_ACCESS_KEY_ID
    valueFrom:
      secretKeyRef:
        key: access_key
        name: vcf-sddc-lcm.db.vcf-sddc-lcm-db.wal-backup-credentials
  - name: AWS_SECRET_ACCESS_KEY
    valueFrom:
      secretKeyRef:
        key: secret_key
        name: vcf-sddc-lcm.db.vcf-sddc-lcm-db.wal-backup-credentials
  - name: AWS_ENDPOINT
    value: https://seaweedfs-s3.vmsp-platform.svc.cluster.local:8443
  - name: WALG_S3_CA_CERT_FILE
    value: /usr/local/share/ca-certificates/client/ca.crt
  - name: USE_WALG_BACKUP
    value: "true"
  - name: USE_WALG_RESTORE
    value: "true"
  - name: CLONE_USE_WALG_RESTORE
    value: "true"
  - name: CLONE_WAL_BUCKET_SCOPE_PREFIX
  - name: BACKUP_NUM_TO_RETAIN
    value: "1"
  - name: WAL_S3_BUCKET
    value: vcf-sddc-lcm.db.vcf-sddc-lcm-db.wal-backup
  - name: AWS_S3_FORCE_PATH_STYLE
    value: "true"
  - name: BACKUP_SCHEDULE
    value: '#0 25 31 2 * '
  nodeAffinity:
    preferredDuringSchedulingIgnoredDuringExecution:
    - preference:
        matchExpressions:
        - key: node.kubernetes.io/instance-type
          operator: In
          values:
          - vsphere-vm.cpu-12.mem-24gb.os-photon
      weight: 1
  numberOfInstances: 3
  patroni:
    failsafe_mode: true
    retry_timeout: 20
    synchronous_mode: true
    synchronous_mode_strict: true
    synchronous_node_count: 1
    ttl: 50
  postgresql:
    parameters:
      archive_cleanup_command: curl -sf --connect-timeout 1 --max-time 5 http://127.0.0.1:8008/replica
        > /dev/null && envdir "/run/etc/wal-e.d/env" pg_archivecleanup "/home/postgres/pgdata/pgroot/data/pg_wal"
        "%r" || exit 0
      archive_command: (curl -sf --connect-timeout 1 --max-time 5 http://127.0.0.1:8008/primary
        > /dev/null || exit 0) && envdir "/run/etc/wal-e.d/env" wal-g wal-push "%p"
      archive_timeout: 900s
      client_min_messages: error
      log_destination: stderr
      log_line_prefix: '%m %c %x %d %u %r %p %l'
      log_parameter_max_length: "0"
      log_parameter_max_length_on_error: "0"
      logging_collector: "off"
      max_connections: "500"
      password_encryption: scram-sha-256
      pgaudit.log: none
      pgaudit.log_catalog: "off"
      pgaudit.log_parameter: "off"
      restore_command: 'envdir "/run/etc/wal-e.d/env" /bin/bash -c "echo $(date -u
        +%%Y-%%m-%%d_%%H:%%M:%%S_UTC) rc: start %f >&2; if ! timeout 1s bash -c ''exec
        3<>/dev/tcp/seaweedfs-s3.vmsp-platform.svc.cluster.local/8443''; then echo
        $(date -u +%%Y-%%m-%%d_%%H:%%M:%%S_UTC) rc: preflight-fail https://seaweedfs-s3.vmsp-platform.svc.cluster.local:8443
        >&2; exit 1; fi; if timeout 30s /scripts/restore_command.sh \"%f\" \"%p\";
        then if [ -s \"%p\" ]; then echo $(date -u +%%Y-%%m-%%d_%%H:%%M:%%S_UTC) rc:
        success %f >&2; exit 0; else echo $(date -u +%%Y-%%m-%%d_%%H:%%M:%%S_UTC)
        rc: empty-file %f >&2; exit 1; fi; else rc=$?; if [ \"${rc:-0}\" -eq 124 ];
        then echo $(date -u +%%Y-%%m-%%d_%%H:%%M:%%S_UTC) rc: fetch-timeout %f >&2;
        else echo $(date -u +%%Y-%%m-%%d_%%H:%%M:%%S_UTC) rc: fetch-fail %f rc=${rc:-?}
        >&2; fi; exit 1; fi"'
      shared_preload_libraries: bg_mon,pg_stat_statements,pgextwlist,pg_auth_mon,set_user,timescaledb,pg_cron,pg_stat_kcache
      statement_timeout: "30000"
      synchronous_commit: "on"
    version: "16"
  preparedDatabases:
    vcfsddclcmdb:
      defaultUsers: true
      schemas:
        public: {}
  resources:
    limits:
      cpu: "1"
      memory: 3048Mi
    requests:
      cpu: 500m
      memory: 512Mi
  sidecars:
  - env:
    - name: DATA_SOURCE_URI
      value: localhost:5432/postgres?sslmode=disable
    - name: DATA_SOURCE_USER
      value: $(POSTGRES_USER)
    - name: DATA_SOURCE_PASS
      value: $(POSTGRES_PASSWORD)
    image: registry.vmsp-platform.svc.cluster.local:5000/images/prometheuscommunity-postgres-exporter:v0.18.1
    name: metrics-exporter
    ports:
    - containerPort: 9187
      name: http-metrics
      protocol: TCP
  - env:
    - name: PGPORT
      value: "5432"
    - name: PGHOST
      value: vcf-sddc-lcm-db.vcf-sddc-lcm.svc.cluster.local
    - name: WALE_S3_PREFIX
      value: s3://vcf-sddc-lcm.db.vcf-sddc-lcm-db.wal-backup/spilo/vcf-sddc-lcm-vcf-sddc-lcm-db/wal/16/
    - name: AWS_ACCESS_KEY_ID
      valueFrom:
        secretKeyRef:
          key: access_key
          name: vcf-sddc-lcm.db.vcf-sddc-lcm-db.wal-backup-credentials
    - name: AWS_SECRET_ACCESS_KEY
      valueFrom:
        secretKeyRef:
          key: secret_key
          name: vcf-sddc-lcm.db.vcf-sddc-lcm-db.wal-backup-credentials
    - name: AWS_ENDPOINT
      value: https://seaweedfs-s3.vmsp-platform.svc.cluster.local:8443
    - name: WALG_S3_CA_CERT_FILE
      value: /usr/local/share/ca-certificates/client/ca.crt
    - name: USE_WALG_BACKUP
      value: "true"
    - name: USE_WALG_RESTORE
      value: "true"
    - name: CLONE_USE_WALG_RESTORE
      value: "true"
    - name: CLONE_WAL_BUCKET_SCOPE_PREFIX
    - name: BACKUP_NUM_TO_RETAIN
      value: "1"
    - name: WAL_S3_BUCKET
      value: vcf-sddc-lcm.db.vcf-sddc-lcm-db.wal-backup
    - name: AWS_S3_FORCE_PATH_STYLE
      value: "true"
    - name: BACKUP_SCHEDULE
      value: '#0 25 31 2 * '
    image: registry.vmsp-platform.svc.cluster.local:5000/images/pgbackup:v1.0.0
    name: walg
    ports:
    - containerPort: 9100
      name: walg
      protocol: TCP
    resources:
      limits:
        cpu: 1000m
        memory: 768Mi
      requests:
        cpu: 100m
        memory: 100Mi
  teamId: vcf-sddc-lcm-db
  tls:
    caFile: ca.crt
    secretName: vcf-sddc-lcm-db-postgres-cert-secret
  users:
    postgres:
    - superuser
    - createdb
  volume:
    size: 30Gi
status:
  PostgresClusterStatus: Running
6 – Salt
6.1 – Salt RaaS
6.1.1 – Salt RaaS Optimisations

The RaaS instance is quite over provisioned

An example of what mine was running at

NAME                     CPU(cores)   MEMORY(bytes)   
pgdatabase-0             43m          790Mi           
raas-5857f54b4f-l4zb4    20m          716Mi           
redis-6cccfb46f4-gp7p6   6m           7Mi  

The current requests are

RaaS – 1000m/1536MiB
Redis – 500m/500MiB
Postgres – 1200m/2248MiB

So these are very over provisioned, we can set them to the following, Request and Limits as
Raas – 250m/1024MiB
Redis – 100m/128MiB
Postgres – 250m/768MiB – 500m/1536MiB

Raas

kubectl patch deployment raas -n salt-raas --type='json' -p='[
  {"op": "replace", "path": "/spec/template/spec/containers/0/resources/requests/cpu", "value": "250m"},
  {"op": "replace", "path": "/spec/template/spec/containers/0/resources/requests/memory", "value": "1Gi"},
  {"op": "replace", "path": "/spec/template/spec/containers/0/resources/limits/memory", "value": "1Gi"}
]'

Redis

kubectl patch deployment redis -n salt-raas --type='json' -p='[
  {"op": "replace", "path": "/spec/template/spec/containers/0/resources/requests/cpu", "value": "100m"},
  {"op": "replace", "path": "/spec/template/spec/containers/0/resources/requests/memory", "value": "128Mi"},
  {"op": "replace", "path": "/spec/template/spec/containers/0/resources/limits/memory", "value": "128Mi"}
]'

Postgres

kubectl patch postgresinstance pgdatabase -n salt-raas --type='json' -p='[
  {"op": "replace", "path": "/spec/resources/requests/cpu", "value": "250m"},
  {"op": "replace", "path": "/spec/resources/requests/memory", "value": "768Mi"},
  {"op": "replace", "path": "/spec/resources/limits/cpu", "value": "500m"},
  {"op": "replace", "path": "/spec/resources/limits/memory", "value": "1536Mi"}
]'

We then need to delete the pod to force the changes to be picked up with

kubectl delete pod pgdatabase-0 -n salt-raas
6.1.2 – Original YAML Backups
6.1.2.1 – raas.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
    deployment.kubernetes.io/revision: "1"
    meta.helm.sh/release-name: raas
    meta.helm.sh/release-namespace: salt-raas
    reloader.vmsp.vmware.com/auto: "true"
  creationTimestamp: "2026-05-12T15:01:21Z"
  generation: 1
  labels:
    app: raas
    app.kubernetes.io/managed-by: Helm
    helm.toolkit.fluxcd.io/name: raas
    helm.toolkit.fluxcd.io/namespace: salt-raas
    ops.vmsp.vmware.com/type: salt-service-fleet
  name: raas
  namespace: salt-raas
  resourceVersion: "13208867"
  uid: efbe38c7-0ef5-47bd-943c-746cf3c6fff2
spec:
  progressDeadlineSeconds: 600
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app: raas
  strategy:
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 25%
    type: RollingUpdate
  template:
    metadata:
      labels:
        app: raas
        app.kubernetes.io/name: raas
        ops.vmsp.vmware.com/type: salt-resource-fleet
    spec:
      affinity:
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
          - podAffinityTerm:
              labelSelector: {}
              namespaces:
              - salt-raas
              - salt
              topologyKey: kubernetes.io/hostname
            weight: 100
      containers:
      - env:
        - name: RAAS_LICENSES
          valueFrom:
            secretKeyRef:
              key: license
              name: raas-secret
        - name: DATABASE_URL
          valueFrom:
            secretKeyRef:
              key: uri
              name: raasdb-owner-user
        - name: REDIS_URL
          value: rediss://redis-service:6379
        - name: SSE_LOG_LEVEL
          value: error
        - name: ENABLE_METRICS_SERVER
          value: "false"
        - name: RAAS_UI_INDEX
          value: raas-index.html
        - name: RAAS_UI_ENABLED
          value: "false"
        - name: RAAS_PATH_PREFIX
          value: /raas
        - name: RAAS_HEALTH_PORT
          value: "8444"
        - name: DISABLE_SECOPS_TASKS
          value: "1"
        - name: DISABLE_VMAN_TASKS
          value: "1"
        - name: DISABLE_LDAP_TASKS
          value: "1"
        - name: DISABLE_CELERY_TASK_ROUTES
          value: "1"
        image: registry.vmsp-platform.svc.cluster.local:5000/images/raas:9.1.0-25346036
        imagePullPolicy: IfNotPresent
        livenessProbe:
          failureThreshold: 5
          httpGet:
            path: /livez
            port: 8444
            scheme: HTTPS
          initialDelaySeconds: 20
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 55
        name: raas
        ports:
        - containerPort: 8443
          protocol: TCP
        readinessProbe:
          failureThreshold: 4
          httpGet:
            path: /readyz
            port: 8444
            scheme: HTTPS
          initialDelaySeconds: 10
          periodSeconds: 5
          successThreshold: 1
          timeoutSeconds: 30
        resources:
          limits:
            cpu: "1"
            memory: 1536Mi
          requests:
            cpu: "1"
            memory: 1536Mi
        securityContext:
          allowPrivilegeEscalation: false
          capabilities:
            drop:
            - ALL
          readOnlyRootFilesystem: false
          runAsGroup: 999
          runAsNonRoot: true
          runAsUser: 999
          seccompProfile:
            type: RuntimeDefault
        startupProbe:
          failureThreshold: 30
          httpGet:
            path: /readyz
            port: 8444
            scheme: HTTPS
          initialDelaySeconds: 30
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 40
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /config
          name: raas-config-vol
        - mountPath: /etc/raas/pki
          name: raas-keys-vol
        - mountPath: /tls
          name: raas-tls
        - mountPath: /postgresql-tls
          name: raas-postgresql-tls
        - mountPath: /redis-tls
          name: raas-redis-tls
      dnsPolicy: ClusterFirst
      hostAliases:
      - hostnames:
        - skyveil.leaha.co.uk
        ip: 192.168.1.91
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext:
        fsGroup: 999
        runAsGroup: 999
        runAsNonRoot: true
        runAsUser: 999
      serviceAccount: raas-account
      serviceAccountName: raas-account
      terminationGracePeriodSeconds: 30
      volumes:
      - configMap:
          defaultMode: 420
          items:
          - key: raas_config
            path: raas
          name: raas-configmap
        name: raas-config-vol
      - name: raas-keys-vol
        secret:
          defaultMode: 420
          items:
          - key: db_key
            path: .raas.key
          - key: instance_id
            path: .instance_id
          secretName: raas-secret
      - name: raas-tls
        secret:
          defaultMode: 420
          items:
          - key: tls.crt
            path: tls.crt
          - key: tls.key
            path: tls.key
          - key: ca.crt
            path: ca.crt
          secretName: raas-instance-cert-secret
      - name: raas-postgresql-tls
        secret:
          defaultMode: 420
          items:
          - key: tls.crt
            path: tls.crt
          - key: tls.key
            path: tls.key
          - key: ca.crt
            path: ca.crt
          secretName: pgdatabase-postgres-cert-secret
      - name: raas-redis-tls
        secret:
          defaultMode: 420
          items:
          - key: tls.crt
            path: tls.crt
          - key: tls.key
            path: tls.key
          - key: ca.crt
            path: ca.crt
          secretName: redis-cert-secret
status:
  availableReplicas: 1
  conditions:
  - lastTransitionTime: "2026-05-12T15:01:21Z"
    lastUpdateTime: "2026-05-12T15:03:03Z"
    message: ReplicaSet "raas-5857f54b4f" has successfully progressed.
    reason: NewReplicaSetAvailable
    status: "True"
    type: Progressing
  - lastTransitionTime: "2026-05-27T06:12:01Z"
    lastUpdateTime: "2026-05-27T06:12:01Z"
    message: Deployment has minimum availability.
    reason: MinimumReplicasAvailable
    status: "True"
    type: Available
  observedGeneration: 1
  readyReplicas: 1
  replicas: 1
  updatedReplicas: 1
6.1.2.2 – redis.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
annotations:
deployment.kubernetes.io/revision: “1”
meta.helm.sh/release-name: raas
meta.helm.sh/release-namespace: salt-raas
creationTimestamp: “2026-05-12T15:01:21Z”
generation: 1
labels:
app: redis
app.kubernetes.io/managed-by: Helm
helm.toolkit.fluxcd.io/name: raas
helm.toolkit.fluxcd.io/namespace: salt-raas
ops.vmsp.vmware.com/type: salt-service-fleet
name: redis
namespace: salt-raas
resourceVersion: “13190086”
uid: a9757a2a-6741-40c0-a6cd-08eb49d3eb9d
spec:
progressDeadlineSeconds: 600
replicas: 1
revisionHistoryLimit: 10
selector:
matchLabels:
app: redis
strategy:
rollingUpdate:
maxSurge: 25%
maxUnavailable: 25%
type: RollingUpdate
template:
metadata:
labels:
app: redis
app.kubernetes.io/name: redis
ops.vmsp.vmware.com/type: salt-resource-fleet
spec:
affinity:
podAntiAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
– podAffinityTerm:
labelSelector: {}
namespaces:
– salt-raas
– salt
topologyKey: kubernetes.io/hostname
weight: 100
containers:
– args:
– –port 0
– –tls-port 6379
– –tls-cert-file /redis-tls/tls.crt
– –tls-key-file /redis-tls/tls.key
– –tls-ca-cert-file /redis-tls/ca.crt
– –tls-auth-clients yes
– –protected-mode no
command:
– redis-server
image: registry.vmsp-platform.svc.cluster.local:5000/images/redis:9.1.0-25346036
imagePullPolicy: IfNotPresent
name: redis
ports:
– containerPort: 6379
protocol: TCP
resources:
limits:
cpu: 500m
memory: 500Mi
requests:
cpu: 500m
memory: 500Mi
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
– mountPath: /redis-tls
name: raas-redis-tls
dnsPolicy: ClusterFirst
restartPolicy: Always
schedulerName: default-scheduler
securityContext: {}
terminationGracePeriodSeconds: 30
volumes:
– name: raas-redis-tls
secret:
defaultMode: 420
items:
– key: tls.crt
path: tls.crt
– key: tls.key
path: tls.key
– key: ca.crt
path: ca.crt
secretName: redis-cert-secret
status:
availableReplicas: 1
conditions:

  • lastTransitionTime: “2026-05-12T15:01:21Z”
    lastUpdateTime: “2026-05-12T15:01:28Z”
    message: ReplicaSet “redis-6cccfb46f4” has successfully progressed.
    reason: NewReplicaSetAvailable
    status: “True”
    type: Progressing
  • lastTransitionTime: “2026-05-26T07:06:38Z”
    lastUpdateTime: “2026-05-26T07:06:38Z”
    message: Deployment has minimum availability.
    reason: MinimumReplicasAvailable
    status: “True”
    type: Available
    observedGeneration: 1
    readyReplicas: 1
    replicas: 1
    updatedReplicas: 1
6.2.2.3 – raas-postgres.yaml
apiVersion: v1
items:
- apiVersion: acid.zalan.do/v1
  kind: postgresql
  metadata:
    annotations:
      vmsp.vmware.com/delete-protection: "true"
    creationTimestamp: "2026-05-12T15:01:28Z"
    generation: 2
    labels:
      ops.vmsp.vmware.com/type: pgdatabase
      velero.io/exclude-from-backup: "true"
    name: pgdatabase
    namespace: salt-raas
    resourceVersion: "13213752"
    uid: d71adbfd-9c4f-4ece-9e9f-d98ea6e92fca
  spec:
    additionalVolumes:
    - mountPath: /usr/local/share/ca-certificates/client/
      name: seaweedfs-client-cert
      targetContainers:
      - postgres
      - walg
      volumeSource:
        secret:
          optional: false
          secretName: seaweedfs-client-cert
    - mountPath: /usr/local/share/ca-certificates/tls/
      name: pgbackup-sidecar-tls-cert
      targetContainers:
      - walg
      volumeSource:
        secret:
          optional: false
          secretName: pgdatabase-postgres-cert-secret
    allowedSourceRanges: null
    enableConnectionPooler: false
    enableReplicaConnectionPooler: false
    env:
    - name: AWS_ACCESS_KEY_ID
      valueFrom:
        secretKeyRef:
          key: access_key
          name: salt-raas.db.pgdatabase.wal-backup-credentials
    - name: AWS_SECRET_ACCESS_KEY
      valueFrom:
        secretKeyRef:
          key: secret_key
          name: salt-raas.db.pgdatabase.wal-backup-credentials
    - name: AWS_ENDPOINT
      value: https://seaweedfs-s3.vmsp-platform.svc.cluster.local:8443
    - name: WALG_S3_CA_CERT_FILE
      value: /usr/local/share/ca-certificates/client/ca.crt
    - name: USE_WALG_BACKUP
      value: "true"
    - name: USE_WALG_RESTORE
      value: "true"
    - name: CLONE_USE_WALG_RESTORE
      value: "true"
    - name: CLONE_WAL_BUCKET_SCOPE_PREFIX
    - name: BACKUP_NUM_TO_RETAIN
      value: "1"
    - name: WAL_S3_BUCKET
      value: salt-raas.db.pgdatabase.wal-backup
    - name: AWS_S3_FORCE_PATH_STYLE
      value: "true"
    - name: BACKUP_SCHEDULE
      value: '#0 25 31 2 * '
    nodeAffinity:
      preferredDuringSchedulingIgnoredDuringExecution:
      - preference:
          matchExpressions:
          - key: node.kubernetes.io/instance-type
            operator: In
            values:
            - vsphere-vm.cpu-12.mem-24gb.os-photon
        weight: 1
    numberOfInstances: 1
    patroni:
      failsafe_mode: true
    podAnnotations:
      database.vmsp.vmware.com/affinity: required
    postgresql:
      parameters:
        archive_cleanup_command: curl -sf --connect-timeout 1 --max-time 5 http://127.0.0.1:8008/replica
          > /dev/null && envdir "/run/etc/wal-e.d/env" pg_archivecleanup "/home/postgres/pgdata/pgroot/data/pg_wal"
          "%r" || exit 0
        archive_command: (curl -sf --connect-timeout 1 --max-time 5 http://127.0.0.1:8008/primary
          > /dev/null || exit 0) && envdir "/run/etc/wal-e.d/env" wal-g wal-push "%p"
        archive_timeout: 900s
        client_min_messages: error
        log_destination: stderr
        log_line_prefix: '%m %c %x %d %u %r %p %l'
        logging_collector: "off"
        max_connections: "500"
        password_encryption: scram-sha-256
        pgaudit.log: none
        pgaudit.log_catalog: "off"
        pgaudit.log_parameter: "off"
        restore_command: 'envdir "/run/etc/wal-e.d/env" /bin/bash -c "echo $(date
          -u +%%Y-%%m-%%d_%%H:%%M:%%S_UTC) rc: start %f >&2; if ! timeout 1s bash
          -c ''exec 3<>/dev/tcp/seaweedfs-s3.vmsp-platform.svc.cluster.local/8443'';
          then echo $(date -u +%%Y-%%m-%%d_%%H:%%M:%%S_UTC) rc: preflight-fail https://seaweedfs-s3.vmsp-platform.svc.cluster.local:8443
          >&2; exit 1; fi; if timeout 30s /scripts/restore_command.sh \"%f\" \"%p\";
          then if [ -s \"%p\" ]; then echo $(date -u +%%Y-%%m-%%d_%%H:%%M:%%S_UTC)
          rc: success %f >&2; exit 0; else echo $(date -u +%%Y-%%m-%%d_%%H:%%M:%%S_UTC)
          rc: empty-file %f >&2; exit 1; fi; else rc=$?; if [ \"${rc:-0}\" -eq 124
          ]; then echo $(date -u +%%Y-%%m-%%d_%%H:%%M:%%S_UTC) rc: fetch-timeout %f
          >&2; else echo $(date -u +%%Y-%%m-%%d_%%H:%%M:%%S_UTC) rc: fetch-fail %f
          rc=${rc:-?} >&2; fi; exit 1; fi"'
        shared_preload_libraries: bg_mon,pg_stat_statements,pgextwlist,pg_auth_mon,set_user,timescaledb,pg_cron,pg_stat_kcache
        statement_timeout: "30000"
        synchronous_commit: "on"
      version: "15"
    preparedDatabases:
      raasdb:
        defaultUsers: true
        schemas:
          public: {}
    resources:
      limits:
        cpu: "1"
        memory: 2Gi
      requests:
        cpu: "1"
        memory: 2Gi
    sidecars:
    - env:
      - name: DATA_SOURCE_URI
        value: localhost:5432/postgres?sslmode=disable
      - name: DATA_SOURCE_USER
        value: $(POSTGRES_USER)
      - name: DATA_SOURCE_PASS
        value: $(POSTGRES_PASSWORD)
      image: registry.vmsp-platform.svc.cluster.local:5000/images/prometheuscommunity-postgres-exporter:v0.18.1
      name: metrics-exporter
      ports:
      - containerPort: 9187
        name: http-metrics
        protocol: TCP
    - env:
      - name: PGPORT
        value: "5432"
      - name: PGHOST
        value: pgdatabase.salt-raas.svc.cluster.local
      - name: WALE_S3_PREFIX
        value: s3://salt-raas.db.pgdatabase.wal-backup/spilo/salt-raas-pgdatabase/wal/15/
      - name: AWS_ACCESS_KEY_ID
        valueFrom:
          secretKeyRef:
            key: access_key
            name: salt-raas.db.pgdatabase.wal-backup-credentials
      - name: AWS_SECRET_ACCESS_KEY
        valueFrom:
          secretKeyRef:
            key: secret_key
            name: salt-raas.db.pgdatabase.wal-backup-credentials
      - name: AWS_ENDPOINT
        value: https://seaweedfs-s3.vmsp-platform.svc.cluster.local:8443
      - name: WALG_S3_CA_CERT_FILE
        value: /usr/local/share/ca-certificates/client/ca.crt
      - name: USE_WALG_BACKUP
        value: "true"
      - name: USE_WALG_RESTORE
        value: "true"
      - name: CLONE_USE_WALG_RESTORE
        value: "true"
      - name: CLONE_WAL_BUCKET_SCOPE_PREFIX
      - name: BACKUP_NUM_TO_RETAIN
        value: "1"
      - name: WAL_S3_BUCKET
        value: salt-raas.db.pgdatabase.wal-backup
      - name: AWS_S3_FORCE_PATH_STYLE
        value: "true"
      - name: BACKUP_SCHEDULE
        value: '#0 25 31 2 * '
      image: registry.vmsp-platform.svc.cluster.local:5000/images/pgbackup:v1.0.0
      name: walg
      ports:
      - containerPort: 9100
        name: walg
        protocol: TCP
      resources:
        limits:
          cpu: 1000m
          memory: 768Mi
        requests:
          cpu: 100m
          memory: 100Mi
    teamId: pgdatabase
    tls:
      caFile: ca.crt
      secretName: pgdatabase-postgres-cert-secret
    users:
      postgres:
      - superuser
      - createdb
    volume:
      size: 100Gi
  status:
    PostgresClusterStatus: Running
kind: List
metadata:
  resourceVersion: ""
6.2 – Salt Master
6.2.1 – Salt Master Optimisations

The Master instance is quite over provisioned

An example of what mine was running at

NAME                           CPU(cores)   MEMORY(bytes)   
salt-master-8674dd877b-8j6q2   16m          912Mi           
salt-minion-6c58844558-zkxqf   3m           91Mi

The current requests are

Master – 500m/1024MiB
Minion – 500m/500MiB

So these are very over provisioned, we can set them to the following, Request and Limits as
Master – 250m/1024MiB – 500m
Minion – 100m/256MiB

Master

kubectl patch deployment salt-master -n salt --type='json' -p='[
  {"op": "replace", "path": "/spec/template/spec/containers/0/resources/requests/cpu", "value": "250m"},
  {"op": "replace", "path": "/spec/template/spec/containers/0/resources/limits/cpu", "value": "500m"}
]'

Minion

kubectl patch deployment salt-minion -n salt --type='json' -p='[
  {"op": "replace", "path": "/spec/template/spec/containers/0/resources/requests/cpu", "value": "100m"},
  {"op": "replace", "path": "/spec/template/spec/containers/0/resources/requests/memory", "value": "256Mi"},
  {"op": "replace", "path": "/spec/template/spec/containers/0/resources/limits/memory", "value": "256Mi"}
]'
6.2.2 – Original YAML Backups
6.2.2.1 – master.yaml
cat salt-master-original.yaml 
apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
    deployment.kubernetes.io/revision: "2"
    meta.helm.sh/release-name: salt-master
    meta.helm.sh/release-namespace: salt
    reloader.vmsp.vmware.com/auto: "true"
  creationTimestamp: "2026-05-12T15:16:45Z"
  generation: 3
  labels:
    app: salt-master
    app.kubernetes.io/managed-by: Helm
    helm.toolkit.fluxcd.io/name: salt-master
    helm.toolkit.fluxcd.io/namespace: salt
    ops.vmsp.vmware.com/type: salt-service
  name: salt-master
  namespace: salt
  resourceVersion: "13208502"
  uid: f35451ab-5d66-4313-8099-2fd5ccbab367
spec:
  progressDeadlineSeconds: 600
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app: salt-master
  strategy:
    type: Recreate
  template:
    metadata:
      annotations:
        vmsp.vmware.com/triggerrestart: "138727"
      labels:
        app: salt-master
        app.kubernetes.io/name: salt-master
        ops.vmsp.vmware.com/type: salt-resource
    spec:
      affinity:
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
          - podAffinityTerm:
              labelSelector: {}
              namespaces:
              - salt
              - salt-raas
              topologyKey: kubernetes.io/hostname
            weight: 100
      containers:
      - env:
        - name: OPS_IDENTIFIER
          value: f4f174ed-d5d9-4990-8cb8-aaf9e5fb913e
        - name: MASTER_FQDN
          value: windspire.leaha.co.uk
        - name: SSL_CERT_FILE
          value: /etc/pki/tls/certs/ca-bundle.crt
        image: registry.vmsp-platform.svc.cluster.local:5000/images/salt-master:9.1.0-25346036
        imagePullPolicy: IfNotPresent
        name: salt-master
        ports:
        - containerPort: 4505
          protocol: TCP
        - containerPort: 4506
          protocol: TCP
        resources:
          limits:
            cpu: 500m
            memory: 1Gi
          requests:
            cpu: 500m
            memory: 1Gi
        securityContext:
          allowPrivilegeEscalation: false
          capabilities:
            drop:
            - ALL
          readOnlyRootFilesystem: false
          runAsGroup: 999
          runAsNonRoot: true
          runAsUser: 999
          seccompProfile:
            type: RuntimeDefault
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /etc/salt/master.d
          name: salt-master-config-vol
        - mountPath: /var/cache/salt
          name: salt-master-volume
          subPath: salt
        - mountPath: /etc/salt/pki/master/minions
          name: salt-master-volume
          subPath: minions
        - mountPath: /etc/pki/tls/certs/
          name: ca-bundle
        - mountPath: /etc/salt/pki/master/master.pem
          name: salt-master-keys-vol
          subPath: master.pem
        - mountPath: /etc/salt/pki/master/master.pub
          name: salt-master-keys-vol
          subPath: master.pub
        - mountPath: /etc/salt/pki/master/sseapi_key.pem
          name: salt-master-keys-vol
          subPath: sseapi_key.pem
        - mountPath: /etc/salt/pki/master/sseapi_key.pub
          name: salt-master-keys-vol
          subPath: sseapi_key.pub
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext:
        fsGroup: 999
        runAsGroup: 999
        runAsNonRoot: true
        runAsUser: 999
      serviceAccount: salt-master-account
      serviceAccountName: salt-master-account
      terminationGracePeriodSeconds: 30
      volumes:
      - configMap:
          defaultMode: 420
          items:
          - key: master_conf
            path: master.conf
          - key: raas_conf
            path: raas.conf
          - key: raas_server_conf
            path: server.conf
          name: salt-master-configmap
        name: salt-master-config-vol
      - configMap:
          defaultMode: 420
          items:
          - key: bundle.pem
            path: ca-bundle.crt
          name: platform-trust
        name: ca-bundle
      - name: salt-master-volume
        persistentVolumeClaim:
          claimName: salt-master-volume
      - name: salt-master-keys-vol
        secret:
          defaultMode: 420
          items:
          - key: master_private_key
            path: master.pem
          - key: master_public_key
            path: master.pub
          - key: raas_private_key
            path: sseapi_key.pem
          - key: raas_public_key
            path: sseapi_key.pub
          secretName: salt-master-secret
status:
  availableReplicas: 1
  conditions:
  - lastTransitionTime: "2026-05-12T15:16:45Z"
    lastUpdateTime: "2026-05-12T15:33:26Z"
    message: ReplicaSet "salt-master-8674dd877b" has successfully progressed.
    reason: NewReplicaSetAvailable
    status: "True"
    type: Progressing
  - lastTransitionTime: "2026-05-27T06:11:57Z"
    lastUpdateTime: "2026-05-27T06:11:57Z"
    message: Deployment has minimum availability.
    reason: MinimumReplicasAvailable
    status: "True"
    type: Available
  observedGeneration: 3
  readyReplicas: 1
  replicas: 1
  updatedReplicas: 1
6.2.2.2 – minion.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
    deployment.kubernetes.io/revision: "3"
    identity.vmsp.vmware.com/serviceaccount: salt-minion
    meta.helm.sh/release-name: salt-minion
    meta.helm.sh/release-namespace: salt
    reloader.vmsp.vmware.com/auto: "true"
  creationTimestamp: "2026-05-12T15:16:44Z"
  generation: 3
  labels:
    app: salt-minion
    app.kubernetes.io/managed-by: Helm
    helm.toolkit.fluxcd.io/name: salt-minion
    helm.toolkit.fluxcd.io/namespace: salt
    ops.vmsp.vmware.com/type: salt-service
  name: salt-minion
  namespace: salt
  resourceVersion: "13190067"
  uid: 48d403a6-49ad-4480-9fa0-bd4d6411fa75
spec:
  progressDeadlineSeconds: 600
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app: salt-minion
  strategy:
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 25%
    type: RollingUpdate
  template:
    metadata:
      annotations:
        identity.vmsp.vmware.com/serviceaccount: salt-minion
        vmsp.vmware.com/triggerrestart: "138702"
      labels:
        app: salt-minion
        app.kubernetes.io/name: salt-minion
        ops.vmsp.vmware.com/type: salt-resource
    spec:
      affinity:
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
          - podAffinityTerm:
              labelSelector: {}
              namespaces:
              - salt
              - salt-raas
              topologyKey: kubernetes.io/hostname
            weight: 100
      containers:
      - env:
        - name: IMDS_FQDN
          value: 169.254.100.200
        - name: VIP_FQDN
          value: aerosphere.leaha.co.uk
        image: registry.vmsp-platform.svc.cluster.local:5000/images/salt-minion:9.1.0-25346036
        imagePullPolicy: IfNotPresent
        name: salt-minion
        resources:
          limits:
            cpu: 500m
            memory: 500Mi
          requests:
            cpu: 500m
            memory: 500Mi
        securityContext:
          allowPrivilegeEscalation: false
          capabilities:
            drop:
            - ALL
          readOnlyRootFilesystem: false
          runAsGroup: 999
          runAsNonRoot: true
          runAsUser: 999
          seccompProfile:
            type: RuntimeDefault
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /etc/salt/minion.d
          name: salt-minion-config-vol
        - mountPath: /etc/salt/pki/minion/minion.pem
          name: salt-minion-keys-vol
          subPath: minion.pem
        - mountPath: /etc/salt/pki/minion/minion.pub
          name: salt-minion-keys-vol
          subPath: minion.pub
        - mountPath: /etc/salt/pki/minion/minion_master.pub
          name: salt-minion-keys-vol
          subPath: minion_master.pub
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext:
        fsGroup: 999
        runAsGroup: 999
        runAsNonRoot: true
        runAsUser: 999
      serviceAccount: salt-minion-account
      serviceAccountName: salt-minion-account
      terminationGracePeriodSeconds: 30
      volumes:
      - configMap:
          defaultMode: 420
          items:
          - key: minion_conf
            path: minion.conf
          name: salt-minion-configmap
        name: salt-minion-config-vol
      - name: salt-minion-keys-vol
        secret:
          defaultMode: 420
          items:
          - key: minion_private_key
            path: minion.pem
          - key: minion_public_key
            path: minion.pub
          - key: master_public_key
            path: minion_master.pub
          secretName: salt-minion-secret
status:
  availableReplicas: 1
  conditions:
  - lastTransitionTime: "2026-05-12T15:16:44Z"
    lastUpdateTime: "2026-05-12T16:14:34Z"
    message: ReplicaSet "salt-minion-6c58844558" has successfully progressed.
    reason: NewReplicaSetAvailable
    status: "True"
    type: Progressing
  - lastTransitionTime: "2026-05-26T07:06:38Z"
    lastUpdateTime: "2026-05-26T07:06:38Z"
    message: Deployment has minimum availability.
    reason: MinimumReplicasAvailable
    status: "True"
    type: Available
  observedGeneration: 3
  readyReplicas: 1
  replicas: 1
  updatedReplicas: 1
7 – Zot-1
7.1 – Zot-1 Optimisations

The Zot container registry is a little over provisioned, as its mostly caching container images, it should be fine to tweak a little without breaking functionaility

The YAML doesnt have a limit, so that covers any burt requirements, but the memory is sat very low vs the default 350m/600MiB with a 700MiB limit

We can run the following to give it 180m default CPU and 400MiB memory with up to 512MiB total

kubectl patch statefulset zot-1 -n vmsp-platform --type='json' -p='[
  {"op": "replace", "path": "/spec/template/spec/containers/0/resources/requests/cpu", "value": "180m"},
  {"op": "replace", "path": "/spec/template/spec/containers/0/resources/requests/memory", "value": "400Mi"},
  {"op": "replace", "path": "/spec/template/spec/containers/0/resources/limits/memory", "value": "512Mi"}
]'
7.2 – Original YAML Backups
7.2.1 – zot-1.yaml
apiVersion: apps/v1
kind: StatefulSet
metadata:
  annotations:
    meta.helm.sh/release-name: zot-1
    meta.helm.sh/release-namespace: vmsp-platform
    reloader.vmsp.vmware.com/auto: "true"
  creationTimestamp: "2026-05-12T12:46:29Z"
  generation: 2
  labels:
    app.kubernetes.io/instance: zot-1
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: zot
    app.kubernetes.io/version: v2.1.11
    helm.sh/chart: zot-0.1.68-34
    helm.toolkit.fluxcd.io/name: zot-1
    helm.toolkit.fluxcd.io/namespace: vmsp-platform
    ops.vmsp.vmware.com/type: registry-service
  name: zot-1
  namespace: vmsp-platform
  resourceVersion: "13209559"
  uid: afa14b9d-52f7-4098-bbf0-f458d3cc51df
spec:
  minReadySeconds: 10
  persistentVolumeClaimRetentionPolicy:
    whenDeleted: Retain
    whenScaled: Retain
  podManagementPolicy: OrderedReady
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app.kubernetes.io/instance: zot-1
      app.kubernetes.io/name: zot
  serviceName: ""
  template:
    metadata:
      annotations:
        checksum/config: 0a98204231fad36afb0c28360a385ad1090e9094feff5d18e517fe5411a4042f
        vmsp.vmware.com/triggerrestart: "13967"
      labels:
        app.kubernetes.io/component: zot
        app.kubernetes.io/instance: zot-1
        app.kubernetes.io/name: zot
        ops.vmsp.vmware.com/type: registry-service
    spec:
      affinity:
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
          - podAffinityTerm:
              labelSelector:
                matchExpressions:
                - key: app.kubernetes.io/name
                  operator: In
                  values:
                  - zot
              topologyKey: kubernetes.io/hostname
            weight: 100
      containers:
      - image: registry.vmsp-platform.svc.cluster.local:5000/images/zot:v2.1.11
        imagePullPolicy: IfNotPresent
        livenessProbe:
          failureThreshold: 3
          httpGet:
            path: /livez
            port: 5000
            scheme: HTTPS
          initialDelaySeconds: 5
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 1
        name: zot
        ports:
        - containerPort: 5000
          name: zot
          protocol: TCP
        readinessProbe:
          failureThreshold: 3
          httpGet:
            path: /readyz
            port: 5000
            scheme: HTTPS
          initialDelaySeconds: 5
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 1
        resources:
          limits:
            ephemeral-storage: 1Gi
            memory: 700Mi
          requests:
            cpu: 350m
            ephemeral-storage: 512Mi
            memory: 600Mi
        securityContext:
          runAsNonRoot: true
        startupProbe:
          failureThreshold: 3
          httpGet:
            path: /startupz
            port: 5000
            scheme: HTTPS
          initialDelaySeconds: 5
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 1
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /var/lib/registry
          name: zot-1-pvc
        - mountPath: /etc/zot
          name: zot-1-config
        - mountPath: /secret
          name: zot-htpasswd-secret
        - mountPath: /certs
          name: certificate
          readOnly: true
        - mountPath: /creds
          name: sync-creds
      dnsPolicy: ClusterFirst
      priorityClassName: system-cluster-critical
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext:
        fsGroup: 65534
        runAsGroup: 65534
        runAsNonRoot: true
        runAsUser: 65534
      serviceAccount: zot-1
      serviceAccountName: zot-1
      terminationGracePeriodSeconds: 30
      volumes:
      - configMap:
          defaultMode: 420
          name: zot-1-config
        name: zot-1-config
      - name: zot-htpasswd-secret
        secret:
          defaultMode: 420
          secretName: zot-htpasswd-secret
      - name: certificate
        secret:
          defaultMode: 420
          secretName: registry-certificate
      - configMap:
          defaultMode: 420
          name: zot-sync-credentials
        name: sync-creds
  updateStrategy:
    rollingUpdate:
      partition: 0
    type: RollingUpdate
  volumeClaimTemplates:
  - apiVersion: v1
    kind: PersistentVolumeClaim
    metadata:
      name: zot-1-pvc
    spec:
      accessModes:
      - ReadWriteOnce
      resources:
        requests:
          storage: 75Gi
      storageClassName: vmsp-default
      volumeMode: Filesystem
    status:
      phase: Pending
status:
  availableReplicas: 1
  collisionCount: 0
  currentReplicas: 1
  currentRevision: zot-1-64cf7c8845
  observedGeneration: 2
  readyReplicas: 1
  replicas: 1
  updateRevision: zot-1-64cf7c8845
  updatedReplicas: 1
8 – Depot
8.1 – Depot Optimisations

The depot and distribution service has a pretty high request on CPU but has no limits, so we can lower this a fair bit without compromising performance when its needed

Memory on the depot we wont touch, while it could be lowered, we can see the pod usage, but its has three containers so I dont want to over reduce one of them

For the depot we can run

kubectl patch deployment depot-service -n vcf-fleet-depot --type='json' -p='[
  {"op": "replace", "path": "/spec/template/spec/containers/0/resources/requests/cpu", "value": "100m"},
  {"op": "replace", "path": "/spec/template/spec/containers/0/resources/requests/memory", "value": "384Mi"},
  {"op": "replace", "path": "/spec/template/spec/containers/1/resources/requests/cpu", "value": "100m"},
  {"op": "replace", "path": "/spec/template/spec/containers/1/resources/requests/memory", "value": "384Mi"},
  {"op": "replace", "path": "/spec/template/spec/containers/2/resources/requests/cpu", "value": "100m"},
  {"op": "replace", "path": "/spec/template/spec/containers/2/resources/requests/memory", "value": "192Mi"}
]'

And for the distribution service we can run

# distribution-service
kubectl patch deployment distribution-service -n vcf-fleet-depot --type='json' -p='[
  {"op": "replace", "path": "/spec/template/spec/containers/0/resources/requests/cpu", "value": "250m"},
  {"op": "replace", "path": "/spec/template/spec/containers/0/resources/requests/memory", "value": "350Mi"}
]'
8.2 – Original YAML Backups
8.2.1 – depot.service.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
    deployment.kubernetes.io/revision: "2"
    meta.helm.sh/release-name: depot-service
    meta.helm.sh/release-namespace: vcf-fleet-depot
    reloader.vmsp.vmware.com/auto: "true"
    reloader.vmsp.vmware.com/exclusion.secret: depot-service-secret
  creationTimestamp: "2026-05-12T13:39:13Z"
  generation: 2
  labels:
    app: depot-service
    app.kubernetes.io/instance: depot-service
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: depot-service
    app.kubernetes.io/version: 9.1.0
    component.vmsp.vmware.com/type: vcf-fleet-depot
    helm.sh/chart: depot-service-9.1.0-0_25371105
    helm.toolkit.fluxcd.io/name: depot-service
    helm.toolkit.fluxcd.io/namespace: vcf-fleet-depot
    ops.vmsp.vmware.com/type: vcf-depot-service
  name: depot-service
  namespace: vcf-fleet-depot
  resourceVersion: "13212166"
  uid: 8b151d0c-0877-4355-99da-e73673a5cfb6
spec:
  progressDeadlineSeconds: 600
  replicas: 1
  revisionHistoryLimit: 5
  selector:
    matchLabels:
      app: depot-service
      app.kubernetes.io/instance: depot-service
      app.kubernetes.io/name: depot-service
      component.vmsp.vmware.com/type: vcf-fleet-depot
  strategy:
    type: Recreate
  template:
    metadata:
      annotations:
        vmsp.vmware.com/triggerrestart: "50227"
      labels:
        app: depot-service
        app.kubernetes.io/instance: depot-service
        app.kubernetes.io/name: depot-service
        component.vmsp.vmware.com/type: vcf-fleet-depot
        ops.vmsp.vmware.com/type: vcf-depot-service-instance
    spec:
      containers:
      - env:
        - name: DEPOT_TYPE
          valueFrom:
            secretKeyRef:
              key: depotType
              name: depot-service-secret
        - name: OFFLINE_DEPOT_ORIGIN
          valueFrom:
            secretKeyRef:
              key: offlineOrigin
              name: depot-service-secret
        - name: OFFLINE_DEPOT_BASE_PATH
          valueFrom:
            secretKeyRef:
              key: offlineBasePath
              name: depot-service-secret
        - name: OFFLINE_DEPOT_USERNAME
          valueFrom:
            secretKeyRef:
              key: offlineUsername
              name: depot-service-secret
        - name: OFFLINE_DEPOT_PASSWORD
          valueFrom:
            secretKeyRef:
              key: offlinePassword
              name: depot-service-secret
        - name: DEPOT_SERVICE_SERVICE_FQDN
          value: depot-service.vcf-fleet-depot.svc.cluster.local
        - name: DISTRIBUTION_SERVICE_HOST
          value: distribution-service.vcf-fleet-depot.svc.cluster.local
        - name: PROXY_FORWARDER_HOST
          value: 127.0.0.1
        - name: SERVER_HTTPS_PORT
          value: "7443"
        - name: DNS_RESOLVER
          value: kube-dns.kube-system.svc.cluster.local
        - name: SKIP_PROXY_FORWARDER
          value: "false"
        - name: PUBLIC_DEPOT_HOST
          value: dl.broadcom.com
        - name: PUBLIC_VVS_HOST
          value: vvs.broadcom.com
        - name: PUBLIC_VSAN_HCL_HOST
          value: eapi.broadcom.com
        - name: PUBLIC_PACKAGES_HOST
          value: projects.packages.broadcom.com
        - name: OFFLINE_OCI_WRITE_ENABLED
          value: "false"
        - name: OCI_REPO_PATH_MAPPINGS_OVERRIDE
          value: '[]'
        - name: PUBLIC_VVS_VCF_LCM_BUNDLE_PATH
          value: /v1/products/bundles/type/vcf-lcm-v2-bundle
        - name: PUBLIC_VVS_VCF_INTEROP_BUNDLE_PATH
          value: /v1/products/bundles/type/vcf-interop-bundle
        - name: PUBLIC_VVS_VLCM_INTEROP_VCG_BUNDLE_PATH
          value: /v1/products/bundles/vlcm-compatibility-data/v1/vlcm-interop-vcg-bundle
        - name: CONFIG_WORKER_CONNECTIONS
          value: "1024"
        - name: CONFIG_WORKER_RLIMIT_NOFILE
          value: "4096"
        - name: CONFIG_SEND_TIMEOUT
          value: 300s
        - name: CONFIG_OUTPUT_BUFFERS
          value: 2 512k
        - name: CONFIG_RATE_LIMIT_ZONE
          value: downloads:10m rate=100r/s
        - name: CONFIG_RATE_LIMIT_BURST
          value: "100"
        - name: DISABLE_CACHE
          value: "false"
        - name: CONFIG_CACHE_INACTIVE
          value: 365d
        - name: CONFIG_CACHE_KEYS_ZONE_SIZE
          value: 100m
        - name: CONFIG_CACHE_LOCK_AGE
          value: 3600s
        - name: CONFIG_CACHE_LOCK_TIMEOUT
          value: 60s
        - name: CONFIG_CACHE_MAX_SIZE
          value: 950g
        - name: CONFIG_CACHE_VALID_BINARIES
          value: 200 365d
        - name: CONFIG_CACHE_REVALIDATE_BINARIES
          value: "off"
        - name: CONFIG_CACHE_REVALIDATE_METADATA
          value: "on"
        - name: CONFIG_CACHE_EXCLUDE_FILE_EXTENSIONS
          value: json,yaml,xml,yml,xml.sha256,json.sha256,xml.sign,json.sign,xml.sig
        - name: CONFIG_PROXY_MAX_TEMP_FILE_SIZE
          value: "0"
        - name: CONFIG_SSL_SESSION_CACHE
          value: shared:SSL:50m
        - name: CONFIG_SSL_SESSION_TIMEOUT
          value: 1d
        - name: CONFIG_PROXY_SSL_SESSION_REUSE
          value: "on"
        - name: CONFIG_PROXY_READ_TIMEOUT
          value: "300"
        image: registry.vmsp-platform.svc.cluster.local:5000/images/file-server:9.1.0-0_25371105
        imagePullPolicy: Always
        livenessProbe:
          failureThreshold: 3
          httpGet:
            path: /internal/api/content-gateway/v1/health
            port: 7443
            scheme: HTTPS
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 1
        name: file-server
        ports:
        - containerPort: 7443
          name: fs-https
          protocol: TCP
        readinessProbe:
          failureThreshold: 3
          httpGet:
            path: /internal/api/content-gateway/v1/health
            port: 7443
            scheme: HTTPS
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 1
        resources:
          limits:
            memory: 1Gi
          requests:
            cpu: 300m
            memory: 512Mi
        securityContext:
          allowPrivilegeEscalation: false
          capabilities:
            add:
            - CHOWN
            - SETGID
            - SETUID
            drop:
            - ALL
          privileged: false
        startupProbe:
          failureThreshold: 60
          httpGet:
            path: /internal/api/content-gateway/v1/health
            port: 7443
            scheme: HTTPS
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 1
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /data
          name: data-volume
          subPath: data
        - mountPath: /etc/service/trust
          name: vcf-fleet-depot-certificate
          readOnly: true
        - mountPath: /etc/platform/trust
          name: platform-trust
          readOnly: true
        - mountPath: /etc/service/secret/depot-service-secret
          name: depot-service-secret
          readOnly: true
      - env:
        - name: PLATFORM_TRUST
          value: /etc/platform/trust
        - name: BCF_TRUSTSTORE_PATH
          value: /etc/platform/java/fips/bundle.bcfks
        - name: JAVA_TRUSTSTORE_FILE_NAME
          value: bundle.jks
        - name: SERVER_CERT_DIR
          value: /etc/service/trust
        - name: SERVER_CERT_FILE
          value: tls.crt
        - name: SERVER_KEY_FILE
          value: tls.key
        - name: BCF_SERVER_KEYSTORE_PATH
          value: /etc/platform/java/fips/tls.bcfks
        - name: PEER_PROXY_CERT_DIR
          value: /etc/platform/peer-proxy-certs
        - name: PEER_PROXY_CERT_FILENAME
          value: https_proxy.pem
        - name: SPRING_PROFILES_ACTIVE
          value: depot
        - name: SPRING_PROFILES_INCLUDE
          value: override
        - name: VCF_USE_RESOURCE_BUNDLE_LOCALIZATION
          value: "true"
        - name: KUBERNETES_CLUSTER_DOMAIN
          value: svc.cluster.local
        - name: KUBERNETES_SERVICE_ACCOUNT_PATH
          value: /etc/service/secret/depot-service-secret
        - name: DEPOT_SERVICE_NAME
          value: depot-service
        - name: DEPOT_SERVICE_SECRET_NAME
          value: depot-service-secret
        - name: DEPOT_SERVICE_SERVICE_FQDN
          value: depot-service.vcf-fleet-depot.svc.cluster.local
        envFrom:
        - configMapRef:
            name: download-service-configs
        image: registry.vmsp-platform.svc.cluster.local:5000/images/download-service:9.1.0-0_25371105
        imagePullPolicy: Always
        livenessProbe:
          failureThreshold: 3
          httpGet:
            path: /depot-service/api/depot/v1/health/livenessState
            port: 8443
            scheme: HTTPS
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 1
        name: download-service
        ports:
        - containerPort: 8080
          name: ds-http
          protocol: TCP
        - containerPort: 8443
          name: ds-https
          protocol: TCP
        readinessProbe:
          failureThreshold: 3
          httpGet:
            path: /depot-service/api/depot/v1/health/readinessState
            port: 8443
            scheme: HTTPS
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 1
        resources:
          limits:
            memory: 1Gi
          requests:
            cpu: 300m
            memory: 512Mi
        securityContext:
          allowPrivilegeEscalation: false
          capabilities:
            add:
            - CHOWN
            - SETGID
            - SETUID
            drop:
            - ALL
          privileged: false
        startupProbe:
          failureThreshold: 60
          httpGet:
            path: /depot-service/api/depot/v1/health/ping
            port: 8443
            scheme: HTTPS
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 1
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /data
          name: data-volume
          subPath: data
        - mountPath: /etc/config
          name: config-volume
        - mountPath: /etc/platform/peer-proxy-service-config
          name: peer-proxy-config
          readOnly: true
        - mountPath: /etc/platform/peer-proxy-certs
          name: proxy-peer-certificate
          readOnly: true
        - mountPath: /etc/platform/peer-proxy-credentials
          name: proxy-peer-credentials
          readOnly: true
        - mountPath: /etc/service/trust
          name: vcf-fleet-depot-certificate
          readOnly: true
        - mountPath: /etc/platform/trust
          name: platform-trust
          readOnly: true
        - mountPath: /etc/service/secret/depot-service-secret
          name: depot-service-secret
          readOnly: true
        - mountPath: /etc/vmware/feature-state-switches
          name: feature-state-switches-volume
          readOnly: true
      - env:
        - name: LISTEN_PORT
          value: "9080"
        - name: PROXY_HOST
        - name: PROXY_PORT
        - name: PROXY_SCHEME
          value: http
        - name: PROXY_CREDENTIALS_ENABLED
          value: "false"
        - name: PROXY_USERNAME
          valueFrom:
            secretKeyRef:
              key: username
              name: proxy-service-credentials
              optional: true
        - name: PROXY_PASSWORD
          valueFrom:
            secretKeyRef:
              key: password
              name: proxy-service-credentials
              optional: true
        - name: PROXY_CERTIFICATE
          valueFrom:
            secretKeyRef:
              key: https_proxy.pem
              name: proxy-service-peer-certificate
              optional: true
        - name: PLATFORM_TRUST_BUNDLE
          value: /etc/platform/trust/bundle.pem
        - name: OFFLINE_DEPOT_CERTIFICATES_PATH
          value: /etc/depot/trust/certificates.pem
        - name: DEPOT_SERVICE_SERVICE_FQDN
          value: depot-service.vcf-fleet-depot.svc.cluster.local
        - name: DISABLE_PROXY
          value: "false"
        image: registry.vmsp-platform.svc.cluster.local:5000/images/proxy-forwarder:9.1.0-0_25371105
        imagePullPolicy: Always
        livenessProbe:
          failureThreshold: 3
          httpGet:
            path: /internal/api/proxy-forwarder/v1/health
            port: 9080
            scheme: HTTP
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 1
        name: proxy-forwarder
        ports:
        - containerPort: 9080
          name: pf-http
          protocol: TCP
        readinessProbe:
          failureThreshold: 3
          httpGet:
            path: /internal/api/proxy-forwarder/v1/health
            port: 9080
            scheme: HTTP
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 1
        resources:
          limits:
            memory: 320Mi
          requests:
            cpu: 300m
            memory: 256Mi
        securityContext:
          allowPrivilegeEscalation: false
          capabilities:
            add:
            - CHOWN
            - SETGID
            - SETUID
            drop:
            - ALL
          privileged: false
        startupProbe:
          failureThreshold: 60
          httpGet:
            path: /internal/api/proxy-forwarder/v1/health
            port: 9080
            scheme: HTTP
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 1
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /etc/platform/trust
          name: platform-trust
          readOnly: true
        - mountPath: /etc/depot/trust
          name: offline-depot-certificates
          readOnly: true
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      serviceAccount: depot-service-service-account
      serviceAccountName: depot-service-service-account
      terminationGracePeriodSeconds: 30
      volumes:
      - configMap:
          defaultMode: 420
          name: peer-proxy-config
          optional: true
        name: peer-proxy-config
      - name: proxy-peer-certificate
        secret:
          defaultMode: 420
          optional: true
          secretName: proxy-service-peer-certificate
      - name: proxy-peer-credentials
        secret:
          defaultMode: 420
          optional: true
          secretName: proxy-service-credentials
      - name: data-volume
        persistentVolumeClaim:
          claimName: depotservice-data-pvc
      - configMap:
          defaultMode: 420
          name: download-service-configs
        name: config-volume
      - name: vcf-fleet-depot-certificate
        secret:
          defaultMode: 420
          secretName: depot-service-intra-cert
      - configMap:
          defaultMode: 420
          name: platform-trust
        name: platform-trust
      - name: depot-service-secret
        secret:
          defaultMode: 420
          secretName: depot-service-secret
      - name: offline-depot-certificates
        secret:
          defaultMode: 420
          items:
          - key: offlineCertificate
            path: certificates.pem
          optional: true
          secretName: depot-service-secret
      - configMap:
          defaultMode: 420
          name: feature-state-switches
        name: feature-state-switches-volume
status:
  availableReplicas: 1
  conditions:
  - lastTransitionTime: "2026-05-12T13:39:13Z"
    lastUpdateTime: "2026-05-12T14:03:31Z"
    message: ReplicaSet "depot-service-77cb99b56b" has successfully progressed.
    reason: NewReplicaSetAvailable
    status: "True"
    type: Progressing
  - lastTransitionTime: "2026-05-27T06:15:00Z"
    lastUpdateTime: "2026-05-27T06:15:00Z"
    message: Deployment has minimum availability.
    reason: MinimumReplicasAvailable
    status: "True"
    type: Available
  observedGeneration: 2
  readyReplicas: 1
  replicas: 1
  updatedReplicas: 1
8.2.2 – distribution.service.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
    deployment.kubernetes.io/revision: "2"
    meta.helm.sh/release-name: distribution-service
    meta.helm.sh/release-namespace: vcf-fleet-depot
    reloader.vmsp.vmware.com/auto: "true"
  creationTimestamp: "2026-05-12T13:38:43Z"
  generation: 2
  labels:
    app: distribution-service
    app.kubernetes.io/component: distribution-service
    app.kubernetes.io/instance: distribution-service
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: distribution-service
    app.kubernetes.io/version: 9.1.0
    helm.sh/chart: distribution-service-9.1.0-0_25371105
    helm.toolkit.fluxcd.io/name: distribution-service
    helm.toolkit.fluxcd.io/namespace: vcf-fleet-depot
    ops.vmsp.vmware.com/type: vcf-distribution-service
  name: distribution-service
  namespace: vcf-fleet-depot
  resourceVersion: "13209944"
  uid: 9b0c4088-29fa-44fc-b577-0af811ec60eb
spec:
  progressDeadlineSeconds: 600
  replicas: 1
  revisionHistoryLimit: 5
  selector:
    matchLabels:
      app.kubernetes.io/component: distribution-service
      app.kubernetes.io/instance: distribution-service
      app.kubernetes.io/name: distribution-service
  strategy:
    type: Recreate
  template:
    metadata:
      annotations:
        checksum/init-config: 9035ec91c56485b6d0bb0367988735b7afd40a05d63b49b752241482381daaca
        vmsp.vmware.com/triggerrestart: "50227"
      labels:
        app: distribution-service
        app.kubernetes.io/component: distribution-service
        app.kubernetes.io/instance: distribution-service
        app.kubernetes.io/name: distribution-service
        ops.vmsp.vmware.com/type: vcf-distribution-service-instance
    spec:
      containers:
      - image: registry.vmsp-platform.svc.cluster.local:5000/images/distribution-service:9.1.0-0_25371105
        imagePullPolicy: Always
        livenessProbe:
          failureThreshold: 3
          httpGet:
            path: /
            port: 5443
            scheme: HTTPS
          periodSeconds: 30
          successThreshold: 1
          timeoutSeconds: 5
        name: distribution-service
        ports:
        - containerPort: 18080
          name: distr-http
          protocol: TCP
        - containerPort: 18443
          name: distr-https
          protocol: TCP
        readinessProbe:
          failureThreshold: 3
          httpGet:
            path: /
            port: 5443
            scheme: HTTPS
          periodSeconds: 120
          successThreshold: 1
          timeoutSeconds: 5
        resources:
          limits:
            memory: 1Gi
          requests:
            cpu: 500m
            memory: 512Mi
        securityContext:
          allowPrivilegeEscalation: false
          capabilities:
            add:
            - CHOWN
            - SETGID
            - SETUID
            drop:
            - ALL
          privileged: false
        startupProbe:
          failureThreshold: 30
          httpGet:
            path: /
            port: 5443
            scheme: HTTPS
          periodSeconds: 15
          successThreshold: 1
          timeoutSeconds: 5
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /var/lib/registry
          name: distributionservice-data-pvc
        - mountPath: /etc/docker/registry
          name: config-dir
        - mountPath: /certs
          name: distribution-certificate
          readOnly: true
        - mountPath: /etc/platform/trust
          name: platform-trust
          readOnly: true
      dnsPolicy: ClusterFirst
      initContainers:
      - args:
        - /etc/scripts/init-config.sh
        command:
        - /bin/sh
        image: registry.vmsp-platform.svc.cluster.local:5000/images/distribution-service:9.1.0-0_25371105
        imagePullPolicy: Always
        name: distribution-service-init
        resources: {}
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /etc/docker/registry
          name: config-dir
        - mountPath: /etc/scripts
          name: init-config
        - mountPath: /certs
          name: distribution-certificate
          readOnly: true
        - mountPath: /etc/platform/trust
          name: platform-trust
          readOnly: true
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      terminationGracePeriodSeconds: 30
      volumes:
      - name: distributionservice-data-pvc
        persistentVolumeClaim:
          claimName: distributionservice-data-pvc
      - emptyDir: {}
        name: config-dir
      - configMap:
          defaultMode: 493
          name: distribution-service-init-config
        name: init-config
      - emptyDir: {}
        name: registry-workdir
      - name: distribution-certificate
        secret:
          defaultMode: 420
          secretName: distribution-intra-cert
      - configMap:
          defaultMode: 420
          name: platform-trust
        name: platform-trust
status:
  availableReplicas: 1
  conditions:
  - lastTransitionTime: "2026-05-12T13:38:43Z"
    lastUpdateTime: "2026-05-12T14:00:57Z"
    message: ReplicaSet "distribution-service-84574b5cd" has successfully progressed.
    reason: NewReplicaSetAvailable
    status: "True"
    type: Progressing
  - lastTransitionTime: "2026-05-27T06:12:21Z"
    lastUpdateTime: "2026-05-27T06:12:21Z"
    message: Deployment has minimum availability.
    reason: MinimumReplicasAvailable
    status: "True"
    type: Available
  observedGeneration: 2
  readyReplicas: 1
  replicas: 1
  updatedReplicas: 1
9 – Telemetry
9.1 – Telemetry Optimisations

The telemetry service has 500m/512MiB, the RAM its using, but the CPU not so much, this I am going to drop to 100m as this is just a phone home service and is well within the current usage

NAME                                  CPU(cores)   MEMORY(bytes)   
telemetry-acceptor-79746d45c6-7cs9g   2m           441Mi

We can change this with

kubectl patch deployment telemetry-acceptor -n telemetry --type='json' -p='[
  {"op": "replace", "path": "/spec/template/spec/containers/0/resources/requests/cpu", "value": "100m"},
  {"op": "replace", "path": "/spec/template/spec/containers/0/resources/limits/cpu", "value": "250m"}
]'
9.2 Original YAML Backups
9.2.1 – telemetry-acceptor.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
    deployment.kubernetes.io/revision: "1"
    meta.helm.sh/release-name: telemetry-acceptor
    meta.helm.sh/release-namespace: telemetry
    reloader.vmsp.vmware.com/auto: "true"
    reloader.vmsp.vmware.com/exclusion.configmap: ceip
    reloader.vmsp.vmware.com/exclusion.secret: telemetry-config
  creationTimestamp: "2026-05-12T14:30:06Z"
  generation: 1
  labels:
    app.kubernetes.io/instance: telemetry-acceptor
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: telemetry-acceptor
    app.kubernetes.io/version: 1.16.0
    helm.sh/chart: telemetry-acceptor-9.1.0-25181946
    helm.toolkit.fluxcd.io/name: telemetry-acceptor
    helm.toolkit.fluxcd.io/namespace: telemetry
    ops.vmsp.vmware.com/type: telemetry-acceptor-service
  name: telemetry-acceptor
  namespace: telemetry
  resourceVersion: "13208066"
  uid: f65038fe-8e4f-467b-b7f1-090efb8069b6
spec:
  progressDeadlineSeconds: 600
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app.kubernetes.io/instance: telemetry-acceptor
      app.kubernetes.io/name: telemetry-acceptor
  strategy:
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 50%
    type: RollingUpdate
  template:
    metadata:
      labels:
        app.kubernetes.io/instance: telemetry-acceptor
        app.kubernetes.io/managed-by: Helm
        app.kubernetes.io/name: telemetry-acceptor
        app.kubernetes.io/version: 1.16.0
        helm.sh/chart: telemetry-acceptor-9.1.0-25181946
        ops.vmsp.vmware.com/type: telemetry-acceptor-service-resource
    spec:
      affinity:
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
          - podAffinityTerm:
              labelSelector:
                matchExpressions:
                - key: app
                  operator: In
                  values:
                  - telemetry-acceptor
              topologyKey: kubernetes.io/hostname
            weight: 100
      containers:
      - env:
        - name: LOG_LEVEL
          value: INFO
        - name: SERVER_PORT
          value: "1081"
        - name: MANAGEMENT_SERVER_PORT
          value: "9091"
        - name: JAVA_TOOL_OPTIONS
          value: |
            -XX:+UseContainerSupport -XX:MaxRAMPercentage="75" -XX:InitialRAMPercentage="50"
        - name: COMPONENT_ID
          value: da545315-0aa9-49c3-b136-f17294358bbc
        - name: COMPONENT_TYPE
          value: telemetry-acceptor
        - name: COMPONENT_VERSION
          value: 9.1.0.0.25181946
        - name: POD_NAME
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.name
        - name: NAMESPACE
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.namespace
        - name: VSP_PROXY_HOST
          valueFrom:
            secretKeyRef:
              key: host
              name: vmsp-proxy-service-secret
              optional: true
        - name: VSP_PROXY_PORT
          valueFrom:
            secretKeyRef:
              key: port
              name: vmsp-proxy-service-secret
              optional: true
        - name: VSP_PROXY_USERNAME
          valueFrom:
            secretKeyRef:
              key: username
              name: vmsp-proxy-service-secret
              optional: true
        - name: VSP_PROXY_PASSWORD
          valueFrom:
            secretKeyRef:
              key: password
              name: vmsp-proxy-service-secret
              optional: true
        - name: VSP_PROXY_SCHEME
          valueFrom:
            secretKeyRef:
              key: scheme
              name: vmsp-proxy-service-secret
              optional: true
        - name: SERVER_SSL_ENABLED
          value: "true"
        - name: SERVER_SSL_CERTIFICATE
          value: file:/etc/tls/tls.crt
        - name: SERVER_SSL_CERTIFICATE_PRIVATE_KEY
          value: file:/etc/tls/tls.key
        - name: SERVER_HTTP_ENABLED
          value: "true"
        - name: SERVER_HTTP_PORT
          value: "8080"
        image: registry.vmsp-platform.svc.cluster.local:5000/images/telemetry-acceptor/9.1.0.0.25181946/telemetry-acceptor/telemetry-acceptor:9.1.0.0.25181946
        imagePullPolicy: IfNotPresent
        livenessProbe:
          failureThreshold: 30
          httpGet:
            path: /debug/health/liveness
            port: 9091
            scheme: HTTP
          periodSeconds: 5
          successThreshold: 1
          timeoutSeconds: 1
        name: telemetry-acceptor
        ports:
        - containerPort: 1081
          name: https
          protocol: TCP
        - containerPort: 8080
          name: http
          protocol: TCP
        - containerPort: 9091
          name: management
          protocol: TCP
        readinessProbe:
          failureThreshold: 3
          httpGet:
            path: /debug/health/readiness
            port: 9091
            scheme: HTTP
          periodSeconds: 5
          successThreshold: 1
          timeoutSeconds: 1
        resources:
          limits:
            cpu: "1"
            memory: 1Gi
          requests:
            cpu: 500m
            memory: 512Mi
        securityContext:
          allowPrivilegeEscalation: false
          capabilities:
            drop:
            - ALL
          readOnlyRootFilesystem: true
          runAsNonRoot: true
          runAsUser: 65532
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /etc/telemetry-config
          name: telemetry-config-volume
          readOnly: true
        - mountPath: /etc/ceip
          name: ceip-volume
          readOnly: true
        - mountPath: /etc/tls
          name: tls-cert
          readOnly: true
        - mountPath: /tmp
          name: tmp-volume
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext:
        fsGroup: 65532
        runAsGroup: 65532
        runAsNonRoot: true
        runAsUser: 65532
      terminationGracePeriodSeconds: 30
      volumes:
      - name: telemetry-config-volume
        secret:
          defaultMode: 420
          secretName: telemetry-config
      - configMap:
          defaultMode: 420
          name: ceip
        name: ceip-volume
      - name: tls-cert
        secret:
          defaultMode: 420
          secretName: telemetry-acceptor-tls
      - emptyDir: {}
        name: tmp-volume
status:
  availableReplicas: 1
  conditions:
  - lastTransitionTime: "2026-05-12T14:30:06Z"
    lastUpdateTime: "2026-05-12T14:30:28Z"
    message: ReplicaSet "telemetry-acceptor-79746d45c6" has successfully progressed.
    reason: NewReplicaSetAvailable
    status: "True"
    type: Progressing
  - lastTransitionTime: "2026-05-27T06:11:51Z"
    lastUpdateTime: "2026-05-27T06:11:51Z"
    message: Deployment has minimum availability.
    reason: MinimumReplicasAvailable
    status: "True"
    type: Available
  observedGeneration: 1
  readyReplicas: 1
  replicas: 1
  updatedReplicas: 1
10 – Node Resize
10.1 – Node Resize Optimisations

Now we have tweaked the pods and reduce the CPU/Memory requests we can tweak the node sizes now without causing as many issues

There is a few places this is defined, and we must edit it at the top level, else it will be reconciled back to the original configuration

The VMSP Platform is where this is stored, and we can get the YAML file defining node size, amoung other things, with

kubectl get packagedeployment vmsp-platform -n vmsp-platform -o yaml

In that YAML we have this spec field, which labels the node sizes as large and management.large

values:
    backups:
      destination: sftp
      full:
        enable: true
        schedule: 10 0 * * MON,TUE,WED,THU,FRI,SAT,SUN
      incremental:
        enable: false
        schedule: '*/15 * * * *'
      retention:
        maxAgeDays: 7
        maxBackups: -1
      storage:
        sftp:
          fingerprint: SHA256:LjIZbgd6sU5G1d4jIvimEJdRJpF4wrkT0xOGm8IJRDU
    ceip: {}
    cluster:
      generation: 0
      ha: "false"
      id: ae05f3e0-1b68-4a70-87c8-5a27bb70f9bc
      name: aerosphere
      nodePrefix: aerosphere
      type: management
      vmspVMMode: bootstrap
      worker:
        enabled: true
        machineType: management.large
        minReplicas: 3
        size: large

We can then patch this to the medium node size, 8vCPU/16GB with

kubectl patch packagedeployment vmsp-platform -n vmsp-platform --type='json' -p='[
  {"op": "replace", "path": "/spec/values/cluster/worker/machineType", "value": "management.medium"},
  {"op": "replace", "path": "/spec/values/cluster/worker/size", "value": "medium"}
]'

We first need to get the machine deployment

kubectl get machinedeployment -A

This returns something like this, I have removed the end bits so its fits here better

NAMESPACE       NAME                       CLUSTER      AVAILABLE   DESIRED   CURRENT   READY   AVAILABLE   
vmsp-platform   aerosphere-default-szs5k   aerosphere   True        3         3         3       3         

Whats we need here is the name, aerosphere-default-szs5k

Now we know the machine deployment name, we can get the full YAML file with

kubectl get machinedeployment <machine-deployment> -n vmsp-platform -o yaml

For my cluster, I ran

kubectl get machinedeployment aerosphere-default-szs5k -n vmsp-platform -o yaml

In that yaml, we are looking for, is the infastructureRed section with the kind: VSphereMachineTemplate, here its the name value we need, I have aerosphere-default-j72pl

In mine I have

infrastructureRef:
        apiGroup: infrastructure.cluster.x-k8s.io
        kind: VSphereMachineTemplate
        name: aerosphere-default-j72pl
      version: v1.34.2

Now we know what the name of the vSphere Machine Template, we can get its yaml config with

kubectl get vspheremachinetemplate <VSphereMachineTemplate-name> -n vmsp-platform -o yaml

For my environment I ran

kubectl get vspheremachinetemplate aerosphere-default-j72pl -n vmsp-platform -o yaml

What we are looking for is the ownerReferences part, and we want the name

ownerReferences:
- apiVersion: cluster.x-k8s.io/v1beta2
  kind: Cluster
  name: aerosphere
  uid: f2eca7f5-de56-4500-8a7c-bf295ef842d3

The correct place to edit this will be the cluster level, to check that yaml, we need to run

kubectl get cluster <name> -n vmsp-platform -o yaml

I ran

kubectl get cluster aerosphere -n vmsp-platform -o yaml

And its here we can find the name of the helm chart, we need the helm flux name, helm.toolkit.fluxcd.io/name, which for me is vmsp-cluster-aerosphere

labels:
    app.kubernetes.io/instance: vmsp-cluster-aerosphere
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: vmsp-cluster
    app.kubernetes.io/version: 9.1.1865
    cluster.x-k8s.io/cluster-name: aerosphere
    controlplane.vmsp.vmware.com/cloud-provider: vsphere
    helm.sh/chart: vmsp-cluster-9.1.1865
    helm.toolkit.fluxcd.io/name: vmsp-cluster-aerosphere
    helm.toolkit.fluxcd.io/namespace: vmsp-platform
    topology.cluster.x-k8s.io/owned: ""

We can patch this to the correct size, the reason we patch the helm release is flux will reconcile its yaml, so while we can manually change the size of the worker nodes, they will get overidden

Then next size down is medium, the default is large, and we can set this with

kubectl patch helmrelease <helm.toolkit.fluxcd.io/name> -n vmsp-platform --type='json' -p='[
  {"op": "replace", "path": "/spec/values/worker/machineType", "value": "management.medium"}
]'

Which for me was

kubectl patch helmrelease vmsp-cluster-aerosphere -n vmsp-platform --type='json' -p='[
  {"op": "replace", "path": "/spec/values/worker/machineType", "value": "management.medium"}
]'

CAPV will begin automatically rolling out new nodes of the correct size to replace the old ones

Once the roll out was completed I did have to go back and re apply the identity broker config as the DB has reset

10.2 – Original YAML Backups
10.2.1 – cluster-autoscaler.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
    deployment.kubernetes.io/revision: "1"
    meta.helm.sh/release-name: cluster-autoscaler
    meta.helm.sh/release-namespace: vmsp-platform
  creationTimestamp: "2026-05-12T12:59:50Z"
  generation: 1
  labels:
    app.kubernetes.io/instance: cluster-autoscaler
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: clusterapi-cluster-autoscaler
    helm.sh/chart: cluster-autoscaler-9.5.1
    helm.toolkit.fluxcd.io/name: cluster-autoscaler
    helm.toolkit.fluxcd.io/namespace: vmsp-platform
  name: cluster-autoscaler-clusterapi-cluster-autoscaler
  namespace: vmsp-platform
  resourceVersion: "13190340"
  uid: 9cc9caab-a879-44ea-b3a4-95ee66480edb
spec:
  progressDeadlineSeconds: 600
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app.kubernetes.io/instance: cluster-autoscaler
      app.kubernetes.io/name: clusterapi-cluster-autoscaler
  strategy:
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 25%
    type: RollingUpdate
  template:
    metadata:
      labels:
        app.kubernetes.io/instance: cluster-autoscaler
        app.kubernetes.io/name: clusterapi-cluster-autoscaler
    spec:
      containers:
      - command:
        - ./cluster-autoscaler
        - --cloud-provider=clusterapi
        - --namespace=vmsp-platform
        - --node-group-auto-discovery=clusterapi:clusterName=aerosphere
        - --enforce-node-group-min-size=true
        - --leader-elect=false
        - --logtostderr=true
        - --scan-interval=60s
        - --stderrthreshold=info
        - --v=4
        env:
        - name: POD_NAMESPACE
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.namespace
        - name: SERVICE_ACCOUNT
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: spec.serviceAccountName
        image: registry.vmsp-platform.svc.cluster.local:5000/images/cluster-autoscaler:v1.34.0
        imagePullPolicy: IfNotPresent
        livenessProbe:
          failureThreshold: 3
          httpGet:
            path: /health-check
            port: 8085
            scheme: HTTP
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 1
        name: clusterapi-cluster-autoscaler
        ports:
        - containerPort: 8085
          protocol: TCP
        resources:
          limits:
            memory: 1Gi
          requests:
            cpu: 10m
            memory: 100Mi
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
      dnsPolicy: ClusterFirst
      priorityClassName: system-cluster-critical
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      serviceAccount: cluster-autoscaler-clusterapi-cluster-autoscaler
      serviceAccountName: cluster-autoscaler-clusterapi-cluster-autoscaler
      terminationGracePeriodSeconds: 30
status:
  availableReplicas: 1
  conditions:
  - lastTransitionTime: "2026-05-12T12:59:50Z"
    lastUpdateTime: "2026-05-12T12:59:53Z"
    message: ReplicaSet "cluster-autoscaler-clusterapi-cluster-autoscaler-7c7b8d4c99"
      has successfully progressed.
    reason: NewReplicaSetAvailable
    status: "True"
    type: Progressing
  - lastTransitionTime: "2026-05-26T07:06:44Z"
    lastUpdateTime: "2026-05-26T07:06:44Z"
    message: Deployment has minimum availability.
    reason: MinimumReplicasAvailable
    status: "True"
    type: Available
  observedGeneration: 1
  readyReplicas: 1
  replicas: 1
  updatedReplicas: 1
10.2.2 – worker-machinedeployment.yaml
apiVersion: cluster.x-k8s.io/v1beta2
kind: MachineDeployment
metadata:
  annotations:
    cluster.x-k8s.io/cluster-api-autoscaler-node-group-max-size: "20"
    cluster.x-k8s.io/cluster-api-autoscaler-node-group-min-size: "3"
    machinedeployment.clusters.x-k8s.io/revision: "1"
  creationTimestamp: "2026-05-12T12:51:25Z"
  finalizers:
  - cluster.x-k8s.io/machinedeployment
  - machinedeployment.topology.cluster.x-k8s.io
  generation: 17
  labels:
    cluster.x-k8s.io/cluster-name: aerosphere
    topology.cluster.x-k8s.io/deployment-name: default
    topology.cluster.x-k8s.io/owned: ""
  name: aerosphere-default-szs5k
  namespace: vmsp-platform
  ownerReferences:
  - apiVersion: cluster.x-k8s.io/v1beta2
    kind: Cluster
    name: aerosphere
    uid: f2eca7f5-de56-4500-8a7c-bf295ef842d3
  resourceVersion: "13209883"
  uid: a37dc85c-bbfe-472a-acd2-1555fc6722de
spec:
  clusterName: aerosphere
  replicas: 3
  rollout:
    strategy:
      rollingUpdate:
        maxSurge: 1
        maxUnavailable: 0
      type: RollingUpdate
  selector:
    matchLabels:
      cluster.x-k8s.io/cluster-name: aerosphere
      topology.cluster.x-k8s.io/deployment-name: default
      topology.cluster.x-k8s.io/owned: ""
  template:
    metadata:
      annotations:
        cluster.x-k8s.io/cluster-api-autoscaler-node-group-max-size: "20"
        cluster.x-k8s.io/cluster-api-autoscaler-node-group-min-size: "3"
      labels:
        cluster.x-k8s.io/cluster-name: aerosphere
        topology.cluster.x-k8s.io/deployment-name: default
        topology.cluster.x-k8s.io/owned: ""
    spec:
      bootstrap:
        configRef:
          apiGroup: bootstrap.cluster.x-k8s.io
          kind: KubeadmConfigTemplate
          name: aerosphere-default-jg76p
      clusterName: aerosphere
      deletion:
        nodeDeletionTimeoutSeconds: 1800
        nodeDrainTimeoutSeconds: 1800
        nodeVolumeDetachTimeoutSeconds: 1800
      infrastructureRef:
        apiGroup: infrastructure.cluster.x-k8s.io
        kind: VSphereMachineTemplate
        name: aerosphere-default-j72pl
      version: v1.34.2
status:
  availableReplicas: 3
  conditions:
  - lastTransitionTime: "2026-05-25T13:19:28Z"
    message: ""
    observedGeneration: 17
    reason: Available
    status: "True"
    type: Available
  - lastTransitionTime: "2026-05-12T12:51:31Z"
    message: ""
    observedGeneration: 17
    reason: NotRollingOut
    status: "False"
    type: RollingOut
  - lastTransitionTime: "2026-05-13T06:29:16Z"
    message: ""
    observedGeneration: 17
    reason: NotRemediating
    status: "False"
    type: Remediating
  - lastTransitionTime: "2026-05-26T07:09:22Z"
    message: ""
    observedGeneration: 17
    reason: NotScalingDown
    status: "False"
    type: ScalingDown
  - lastTransitionTime: "2026-05-24T22:17:55Z"
    message: ""
    observedGeneration: 17
    reason: NotScalingUp
    status: "False"
    type: ScalingUp
  - lastTransitionTime: "2026-05-26T07:09:22Z"
    message: ""
    observedGeneration: 17
    reason: Ready
    status: "True"
    type: MachinesReady
  - lastTransitionTime: "2026-05-12T12:51:31Z"
    message: ""
    observedGeneration: 17
    reason: UpToDate
    status: "True"
    type: MachinesUpToDate
  - lastTransitionTime: "2026-05-12T12:51:31Z"
    message: ""
    observedGeneration: 17
    reason: NotPaused
    status: "False"
    type: Paused
  - lastTransitionTime: "2026-05-12T12:51:31Z"
    message: ""
    observedGeneration: 17
    reason: NotDeleting
    status: "False"
    type: Deleting
  deprecated:
    v1beta1:
      availableReplicas: 3
      conditions:
      - lastTransitionTime: "2026-05-27T06:12:18Z"
        status: "True"
        type: Ready
      - lastTransitionTime: "2026-05-27T06:12:18Z"
        status: "True"
        type: Available
      - lastTransitionTime: "2026-05-26T07:09:22Z"
        status: "True"
        type: MachineSetReady
      readyReplicas: 3
      unavailableReplicas: 0
      updatedReplicas: 3
  observedGeneration: 17
  phase: Running
  readyReplicas: 3
  replicas: 3
  selector: cluster.x-k8s.io/cluster-name=aerosphere,topology.cluster.x-k8s.io/deployment-name=default,topology.cluster.x-k8s.io/owned=
  upToDateReplicas: 3
10.2.3 – vspheremachinetemplate.yaml
apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
kind: VSphereMachineTemplate
metadata:
  annotations:
    cluster.x-k8s.io/cloned-from-groupkind: VSphereMachineTemplate.infrastructure.cluster.x-k8s.io
    cluster.x-k8s.io/cloned-from-name: vsphere-9.1.1865-worker-machinetemplate
    helm.sh/resource-policy: keep
    meta.helm.sh/release-name: vmsp-configs
    meta.helm.sh/release-namespace: vmsp-platform
  creationTimestamp: "2026-05-12T12:51:25Z"
  generation: 1
  labels:
    app.kubernetes.io/instance: vmsp-configs
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: vmsp-configs
    app.kubernetes.io/version: 9.1.1865
    cluster.x-k8s.io/cluster-name: aerosphere
    helm.sh/chart: vmsp-configs-9.1.1865
    helm.toolkit.fluxcd.io/name: vmsp-configs
    helm.toolkit.fluxcd.io/namespace: vmsp-platform
    topology.cluster.x-k8s.io/deployment-name: default
    topology.cluster.x-k8s.io/owned: ""
  name: aerosphere-default-j72pl
  namespace: vmsp-platform
  ownerReferences:
  - apiVersion: cluster.x-k8s.io/v1beta2
    kind: Cluster
    name: aerosphere
    uid: f2eca7f5-de56-4500-8a7c-bf295ef842d3
  resourceVersion: "23039"
  uid: 4bccfe6d-a9bb-46c2-aa73-a861cc6fd709
spec:
  template:
    spec:
      cloneMode: fullClone
      datacenter: Datacenter:datacenter-1001
      datastore: Datastore:datastore-672582
      diskGiB: 100
      folder: Folder:group-v683669
      memoryMiB: 24576
      namingStrategy:
        template: aerosphere-{{ trunc -5 .machine.name }}
      network:
        devices:
        - addressesFromPools:
          - apiGroup: ipam.cluster.x-k8s.io
            kind: InClusterIPPool
            name: aerosphere-global-ip-pool
          dhcp4: false
          dhcp6: false
          nameservers:
          - 192.168.1.5
          - 192.168.1.27
          networkName: DistributedVirtualPortgroup:dvportgroup-1069
      numCPUs: 12
      os: Linux
      powerOffMode: trySoft
      resourcePool: ResourcePool:resgroup-672691
      server: borealis.leaha.co.uk
      storagePolicyName: ""
      template: VirtualMachine:vm-683670
10.2.4 – <cluster-name>-cluster.yaml
apiVersion: cluster.x-k8s.io/v1beta2
kind: Cluster
metadata:
  creationTimestamp: "2026-05-12T12:51:24Z"
  finalizers:
  - cluster.cluster.x-k8s.io
  generation: 110
  labels:
    app.kubernetes.io/instance: vmsp-cluster-aerosphere
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: vmsp-cluster
    app.kubernetes.io/version: 9.1.1865
    cluster.x-k8s.io/cluster-name: aerosphere
    controlplane.vmsp.vmware.com/cloud-provider: vsphere
    helm.sh/chart: vmsp-cluster-9.1.1865
    helm.toolkit.fluxcd.io/name: vmsp-cluster-aerosphere
    helm.toolkit.fluxcd.io/namespace: vmsp-platform
    topology.cluster.x-k8s.io/owned: ""
  name: aerosphere
  namespace: vmsp-platform
  resourceVersion: "14160176"
  uid: f2eca7f5-de56-4500-8a7c-bf295ef842d3
spec:
  clusterNetwork:
    pods:
      cidrBlocks:
      - 198.18.0.0/17
    services:
      cidrBlocks:
      - 198.18.128.0/17
  controlPlaneEndpoint:
    host: 192.168.1.130
    port: 6443
  controlPlaneRef:
    apiGroup: controlplane.cluster.x-k8s.io
    kind: KubeadmControlPlane
    name: aerosphere-25ndd
  infrastructureRef:
    apiGroup: infrastructure.cluster.x-k8s.io
    kind: VSphereCluster
    name: aerosphere-9dwkk
  topology:
    classRef:
      name: vsphere-9.1.1865
    controlPlane:
      replicas: 1
    variables:
    - name: network
      value: DistributedVirtualPortgroup:dvportgroup-1069
    - name: credsSecretName
      value: aerosphere-vsphere-secret
    - name: dhcp6Enabled
      value: false
    - name: infraServerThumbprint
      value: 1A:65:C2:2A:F1:64:75:0C:74:BF:00:6A:A4:F1:E3:40:16:7A:78:61
    - name: datacenter
      value: Datacenter:datacenter-1001
    - name: dhcp4Enabled
      value: false
    - name: datastore
      value: Datastore:datastore-672582
    - name: storagePolicy
      value: ""
    - name: nodePrefix
      value: aerosphere
    - name: ipPool
      value:
      - apiGroup: ipam.cluster.x-k8s.io
        kind: InClusterIPPool
        name: aerosphere-global-ip-pool
    - name: nameservers
      value:
      - 192.168.1.5
      - 192.168.1.27
    - name: ntpServers
      value:
      - time.cloudflare.com
    - name: infraServerURL
      value: borealis.leaha.co.uk
    - name: resourcePool
      value: ResourcePool:resgroup-672691
    - name: folder
      value: Folder:group-v683669
    - name: addressFamily
      value: ipv4
    - name: vmTemplate
      value: VirtualMachine:vm-683670
    - name: dnsImageTag
      value: v1.12.4
    - name: etcdImageTag
      value: 3.6.5-0
    - name: imageRepository
      value: registry.vmsp-platform.svc.cluster.local:5000/images
    - name: kubeVipPodManifest
      value: |
        apiVersion: v1
        kind: Pod
        metadata:
          creationTimestamp: null
          name: kube-vip
          namespace: kube-system
        spec:
          containers:
          - args:
            - manager
            env:
            - name: vip_arp
              value: "true"
            - name: port
              value: "6443"
            - name: vip_interface
              value: ""
            - name: vip_cidr
              value: "32"
            - name: cp_enable
              value: "true"
            - name: cp_namespace
              value: kube-system
            - name: vip_ddns
              value: "false"
            - name: svc_enable
              value: "false"
            - name: svc_leasename
              value: plndr-svcs-lock
            - name: svc_election
              value: "true"
            - name: vip_leaderelection
              value: "false"
            - name: vip_leasename
              value: plndr-cp-lock
            - name: vip_leaseduration
              value: "15"
            - name: vip_renewdeadline
              value: "10"
            - name: vip_retryperiod
              value: "2"
            - name: vip_preserve_on_leadership_loss
              value: "false"
            - name: address
              value: ${CONTROL_PLANE_ENDPOINT_IP}
            - name: prometheus_server
              value: :2112
            image: registry.vmsp-platform.svc.cluster.local:5000/images/kube-vip:v1.0.2
            imagePullPolicy: IfNotPresent
            name: kube-vip
            resources:
              requests:
                cpu: 50m
                memory: 100Mi
              limits:
                cpu: 200m
                memory: 256Mi
            securityContext:
              capabilities:
                add:
                - NET_ADMIN
                - NET_RAW
            volumeMounts:
            - mountPath: /etc/kubernetes/admin.conf
              name: kubeconfig
            - mountPath: /etc/hosts
              name: etchosts
          hostNetwork: true
          volumes:
          - hostPath:
              path: /etc/kubernetes/admin.conf
              type: FileOrCreate
            name: kubeconfig
          - hostPath:
              path: /etc/kube-vip.hosts
              type: File
            name: etchosts
        status: {}
    - name: kubeApiServer
      value:
        cpuRequest: 250m
        defaultWatchCacheSize: "150"
        gogc: "85"
        gomemlimit: 3200MiB
        memoryLimit: 3500Mi
        memoryRequest: 512Mi
        watchCacheSizes: secrets#300,pods#300,deployments#150,statefulsets#75,daemonsets#75,replicasets#75,configmaps#150,endpoints#150,services#150,events#0,customresourcedefinitions#150,apiservices#75,leases#300,helmreleases#300,machines#200,machinedeployments#150,clusters#150,updaterequests#300,selfsubjectaccessreviews#500
    - name: controlPlaneIpAddr
      value: 192.168.1.130
    - name: numCPUs
      value: 4
    - name: memoryMiB
      value: 10240
    - name: diskGiB
      value: 100
    - name: taintControlPlane
      value: true
    - name: user
      value:
        passwordSecret:
          key: sshPassword
          name: aerosphere-ssh-password-secret
    version: v1.34.2
    workers:
      machineDeployments:
      - class: vsphere-9.1.1865-worker
        metadata:
          annotations:
            cluster.x-k8s.io/cluster-api-autoscaler-node-group-max-size: "20"
            cluster.x-k8s.io/cluster-api-autoscaler-node-group-min-size: "3"
        name: default
        variables:
          overrides:
          - name: numCPUs
            value: 12
          - name: memoryMiB
            value: 24576
          - name: diskGiB
            value: 100
status:
  conditions:
  - lastTransitionTime: "2026-05-28T02:54:24Z"
    message: ""
    observedGeneration: 110
    reason: Available
    status: "True"
    type: Available
  - lastTransitionTime: "2026-05-14T15:20:51Z"
    message: ""
    observedGeneration: 110
    reason: ProbeSucceeded
    status: "True"
    type: RemoteConnectionProbe
  - lastTransitionTime: "2026-05-25T08:30:51Z"
    message: ""
    observedGeneration: 110
    reason: Ready
    status: "True"
    type: InfrastructureReady
  - lastTransitionTime: "2026-05-12T12:51:36Z"
    message: ""
    observedGeneration: 2
    reason: Initialized
    status: "True"
    type: ControlPlaneInitialized
  - lastTransitionTime: "2026-05-28T02:54:24Z"
    message: ""
    observedGeneration: 110
    reason: Available
    status: "True"
    type: ControlPlaneAvailable
  - lastTransitionTime: "2026-05-25T13:19:28Z"
    message: ""
    observedGeneration: 110
    reason: Available
    status: "True"
    type: WorkersAvailable
  - lastTransitionTime: "2026-05-27T06:12:19Z"
    message: ""
    observedGeneration: 110
    reason: ReconcileSucceeded
    status: "True"
    type: TopologyReconciled
  - lastTransitionTime: "2026-05-12T12:51:31Z"
    message: ""
    observedGeneration: 110
    reason: NotRollingOut
    status: "False"
    type: RollingOut
  - lastTransitionTime: "2026-05-13T06:29:16Z"
    message: ""
    observedGeneration: 110
    reason: NotRemediating
    status: "False"
    type: Remediating
  - lastTransitionTime: "2026-05-26T07:09:22Z"
    message: ""
    observedGeneration: 110
    reason: NotScalingDown
    status: "False"
    type: ScalingDown
  - lastTransitionTime: "2026-05-24T22:17:55Z"
    message: ""
    observedGeneration: 110
    reason: NotScalingUp
    status: "False"
    type: ScalingUp
  - lastTransitionTime: "2026-05-28T02:54:24Z"
    message: ""
    observedGeneration: 110
    reason: Ready
    status: "True"
    type: ControlPlaneMachinesReady
  - lastTransitionTime: "2026-05-26T07:09:22Z"
    message: ""
    observedGeneration: 110
    reason: Ready
    status: "True"
    type: WorkerMachinesReady
  - lastTransitionTime: "2026-05-12T12:51:31Z"
    message: ""
    observedGeneration: 110
    reason: UpToDate
    status: "True"
    type: ControlPlaneMachinesUpToDate
  - lastTransitionTime: "2026-05-12T12:51:31Z"
    message: ""
    observedGeneration: 110
    reason: UpToDate
    status: "True"
    type: WorkerMachinesUpToDate
  - lastTransitionTime: "2026-05-12T12:51:31Z"
    message: ""
    observedGeneration: 110
    reason: NotPaused
    status: "False"
    type: Paused
  - lastTransitionTime: "2026-05-12T12:51:31Z"
    message: ""
    observedGeneration: 110
    reason: NotDeleting
    status: "False"
    type: Deleting
  controlPlane:
    availableReplicas: 1
    desiredReplicas: 1
    readyReplicas: 1
    replicas: 1
    upToDateReplicas: 1
  deprecated:
    v1beta1:
      conditions:
      - lastTransitionTime: "2026-05-25T08:30:51Z"
        status: "True"
        type: Ready
      - lastTransitionTime: "2026-05-12T12:51:36Z"
        status: "True"
        type: ControlPlaneInitialized
      - lastTransitionTime: "2026-05-12T12:51:36Z"
        status: "True"
        type: ControlPlaneReady
      - lastTransitionTime: "2026-05-25T08:30:51Z"
        status: "True"
        type: InfrastructureReady
      - lastTransitionTime: "2026-05-27T06:12:19Z"
        status: "True"
        type: TopologyReconciled
  initialization:
    controlPlaneInitialized: true
    infrastructureProvisioned: true
  observedGeneration: 110
  phase: Provisioned
  workers:
    availableReplicas: 3
    desiredReplicas: 3
    readyReplicas: 3
    replicas: 3
    upToDateReplicas: 3
10.2.5 – vmsp-platform.yaml
apiVersion: releases.vmsp.vmware.com/v1alpha1
kind: PackageDeployment
metadata:
  creationTimestamp: "2026-05-12T12:51:38Z"
  finalizers:
  - releases.vmsp.vmware.com/finalizer
  generation: 11
  labels:
    backup.vmsp.vmware.com/platform: "true"
    backup.vmsp.vmware.com/skip-restore: "true"
    component.vmsp.vmware.com/id: ae05f3e0-1b68-4a70-87c8-5a27bb70f9bc
    component.vmsp.vmware.com/name: vsp
    component.vmsp.vmware.com/type: vsp
    component.vmsp.vmware.com/version: 9.1.0.0.25370367
    packages.vcf.vmware.com/component: vsp
    packages.vcf.vmware.com/name: vmsp-platform
    packages.vcf.vmware.com/version: 9.1.0.0.25370367
  name: vmsp-platform
  namespace: vmsp-platform
  resourceVersion: "17906959"
  uid: ae3406ee-b815-4aa4-b89c-140aa56d7543
spec:
  packages:
  - name: envoyproxy-gateway
    version: v1.5.0-4
  - name: vmsp-global-config
    version: 9.1.1865
  - name: logging-operator
    version: 4.11.4-6
  - name: support-bundle
    version: 9.1.1865
  - name: reloader
    version: v1.4.12
  - name: cluster-api-installer
    version: v1.11.1
  - name: vsphere-cpi
    version: 1.34.0
  - name: gateway-api
    version: v1.3.0-7
  - name: antrea
    version: 2.4.2-1
  - name: coredns
    version: v1.12.4
  - name: vmsp-api
    version: 9.1.1865
  - name: trust-manager
    version: v0.20.3
  - name: vmsp-scripts
    version: v9.1.1865
  - name: seaweedfs
    version: 3.96.0-6
  - name: node-problem-detector
    version: v0.8.25
  - name: zalando-pg-operator
    version: v1.14.4-2
  - name: debug
    version: 1.0.8
  - name: etcd-defrag
    version: 0.37.0
  - name: flux2
    version: 2.17.1-1
  - name: cpa
    version: 1.9.1-1
  - name: argo-workflows
    version: 0.45.27-5
  - name: metrics-server
    version: 3.13.0-1
  - name: velero
    version: v1.16.2-9
  - name: vmsp-hooks
    version: 9.1.1865
  - name: kube-prometheus-stack
    version: 51.10.7-13
  - name: vmsp-backup
    version: 9.1.1865
  - name: namespace-config
    version: 0.1.10
  - name: kube-rbac-proxy
    version: v0.20.0
  - name: kyverno-policies
    version: 3.5.2-26
  - name: busybox
    version: 1.36.2-1
  - name: kyverno
    version: 1.15.2-4
  - name: ndc
    version: v9.1.1865
  - name: cluster-autoscaler
    version: 9.5.1-3
  - name: vsphere-csi
    version: 3.6.1-3
  - name: vmsp-configs
    version: 9.1.1865
  - name: kubernetes
    version: 1.0.8-2
  - name: vmsp-identity
    version: v9.1.1865
  - name: proxy-service
    version: 9.1.1865
  - name: metadata-service
    version: 9.1.1865
  - name: vmsp-etcd
    version: 3.6.5-4
  - name: vmsp-certs
    version: 9.1.1865
  - name: descheduler
    version: v0.33.0-4
  - name: vmsp-cluster
    version: 9.1.1865
  - name: zot
    version: v2.1.11-3
  - name: kube-vip
    version: v1.0.2-2
  - name: strimzi-kafka-operator
    version: v0.48.0-1
  - name: vmsp-agent
    version: v9.1.1865
  - name: synthetic-checker
    version: v9.1.1865
  - name: vmsp-operator
    version: v9.1.1865
  - name: cert-manager
    version: v1.18.2-5
  - name: argo-templates
    version: v9.1.1865
  - name: gatewayconfiguration
    version: 9.1.1865
  suspend: {}
  templateSelector: {}
  values:
    backups:
      destination: sftp
      full:
        enable: true
        schedule: 10 0 * * MON,TUE,WED,THU,FRI,SAT,SUN
      incremental:
        enable: false
        schedule: '*/15 * * * *'
      retention:
        maxAgeDays: 7
        maxBackups: -1
      storage:
        sftp:
          fingerprint: SHA256:LjIZbgd6sU5G1d4jIvimEJdRJpF4wrkT0xOGm8IJRDU
    ceip: {}
    cluster:
      generation: 0
      ha: "false"
      id: ae05f3e0-1b68-4a70-87c8-5a27bb70f9bc
      name: aerosphere
      nodePrefix: aerosphere
      type: management
      vmspVMMode: bootstrap
      worker:
        enabled: true
        machineType: management.large
        minReplicas: 3
        size: large
    depot:
      url: https://skyveil.leaha.co.uk/depot-service/content-gateway
    flags:
      assignNodesToVMGroup: true
      pdbs: "true"
    helmRepository:
      certSecretRef: registry-certificate
      insecure: false
    ingress:
      fleet:
        fqdn: skyveil.leaha.co.uk
        vips:
          ipv4:
          - 192.168.1.91
          ipv6: []
      instance:
        fqdn: windspire.leaha.co.uk
        vips:
          ipv4:
          - 192.168.1.92
          ipv6: []
      platform:
        fqdn: aerosphere.leaha.co.uk
        vips:
          ipv4:
          - 192.168.1.90
          ipv6: []
    logs:
      loginsight:
        host: mistral.leaha.co.uk
        port: 9543
        scheme: https
        sslVerify: false
      syslog:
        host: ""
        insecure: true
        transport: tcp
      type: loginsight
    network:
      addressFamily: ipv4
      dhcp: false
      dhcp6: false
      dnsServers:
      - 192.168.1.5
      - 192.168.1.27
      gateway:
        type: envoy
      ingress:
        vips:
          controlplane: 192.168.1.130
          primary: ""
      internal:
        cidr: 198.18.0.0/15
        cidrv6: ""
        clusterCIDR: 198.18.0.0/16
        clusterCIDRv6: ""
        serviceCIDR: 198.18.128.0/17
        serviceCIDRv6: ""
      ipPool:
        addresses:
        - 192.168.1.128/28
        excludedAddresses:
        - 192.168.1.90
        - 192.168.1.91
        - 192.168.1.92
        - 192.168.1.130
        gateway: 192.168.1.1
        prefix: "24"
      ipPoolv6:
        addresses: []
        excludedAddresses: []
        gateway: ""
        prefix: ""
      ntpServers:
      - time.cloudflare.com
    objectStorage:
      filerBacking: etcd
    profiles:
      name: small
    provider:
      vsphere:
        cluster: /Homelab/host/Cryosphere
        clusterId: ClusterComputeResource:domain-c671572
        datacenter: /Homelab
        datacenterId: Datacenter:datacenter-1001
        datastore: /Homelab/datastore/Triton
        datastoreId: Datastore:datastore-672582
        datastoreURL: ds:///vmfs/volumes/vsan:119e14b026f911f1-9185005056b9f29b/
        enriched: "true"
        folder: /Homelab/vm/vcf-management-services
        folderId: Folder:group-v683669
        host: borealis.leaha.co.uk
        insecureTLS: false
        network: /Homelab/network/vSphere-MGMT-1
        networkId: DistributedVirtualPortgroup:dvportgroup-1069
        port: "443"
        resourcePool: /Homelab/host/Cryosphere/Resources/VCF Appliances
        resourcePoolId: ResourcePool:resgroup-672691
        server: borealis.leaha.co.uk
        templateFolder: /Homelab/vm/vcf-management-services
        templateFolderId: Folder:group-v683669
        templateId: VirtualMachine:vm-683670
        thumbprint: 1A:65:C2:2A:F1:64:75:0C:74:BF:00:6A:A4:F1:E3:40:16:7A:78:61
        tlsCertificate: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUZBekNDQTJ1Z0F3SUJBZ0lKQU1kUWRtcWhSUTVXTUEwR0NTcUdTSWIzRFFFQkN3VUFNSUdiTVFzd0NRWUQKVlFRRERBSkRRVEVYTUJVR0NnbVNKb21UOGl4a0FSa1dCM1p6Y0dobGNtVXhGVEFUQmdvSmtpYUprL0lzWkFFWgpGZ1ZzYjJOaGJERUxNQWtHQTFVRUJoTUNWVk14RXpBUkJnTlZCQWdNQ2tOaGJHbG1iM0p1YVdFeEhUQWJCZ05WCkJBb01GR0p2Y21WaGJHbHpMbXhsWVdoaExtTnZMblZyTVJzd0dRWURWUVFMREJKV1RYZGhjbVVnUlc1bmFXNWwKWlhKcGJtY3dIaGNOTWpVeE1qQTFNVFUxTnpNeFdoY05NamN4TWpBMk1ETTFOek14V2pBc01SMHdHd1lEVlFRRApEQlJpYjNKbFlXeHBjeTVzWldGb1lTNWpieTUxYXpFTE1Ba0dBMVVFQmhNQ1ZWTXdnZ0dpTUEwR0NTcUdTSWIzCkRRRUJBUVVBQTRJQmp3QXdnZ0dLQW9JQmdRRGM4WmwxczVvUmZyL2h4SGVmZzlVWHFBWGR6b0UvY2pzUnpWSHkKZlEwZUt1cWdiOEdhZkE3WE83dG1FMDJCclBCelFmRUU1dzhPd1k3ekNTVEVPclErVlJ6a3FBRmdmbFA0V09WZApVZTYzY0NaaXFYRXBoN1h5b0JGK256ZklBbzF3eTZPL0Q0WEsydHBGVkQxWFo4TEdtZXgxei9Pd2FPVERSSjJsCi94U1kyMzB6ZE5iRjJveWJxL3FDVkFFSWtYMmdSRzF1ejY2dURlZlVoUThFYlpZbWUyNVFCekVya2xRWTFjVkkKZU42MlRPeWhUUEl1ZzcyVHptQk9qV1pkK0FtSzJLMHN0YnliVHpqSXRZczRjRC9LeUs1N0dJYWpGZjJNa1BiSQp3YjJIeUFMMGY4bmlLbXJqYUZZMUlLZk9sMmU1TVNDeGFuVzZNQ0FON1J4WkRpZG1OQmdCMnR3dlB0ZFI1bDFQClVtODJTYVRXNHZacXV6VTJWdkZHRnJKVXVVeHVhUEUyUVFnTENKc3NlODJtZFBoMkJGTk5NNS8yUnlTaDJ0WEYKWFpXWnp0dXBVZmdNVWdTc0xhQnAxUldvb1U5NEFiU29MSmVTbnZaK3d4Yy9aV2dZQndNU21sYjFDSmkwdGZtTgpkcld5dnJJNU9YMDM2QUlNQzRudmZOVTFmU01DQXdFQUFhT0J0ekNCdERBTEJnTlZIUThFQkFNQ0JlQXdId1lEClZSMFJCQmd3Rm9JVVltOXlaV0ZzYVhNdWJHVmhhR0V1WTI4dWRXc3dIUVlEVlIwT0JCWUVGTXNxRE0yNjlucFcKUWQ5RVZmSnQwSTYyZEtkb01COEdBMVVkSXdRWU1CYUFGRHdOTEtVMitKTGNKR01BLzBRYWFXOHlZdVgvTUVRRwpDQ3NHQVFVRkJ3RUJCRGd3TmpBMEJnZ3JCZ0VGQlFjd0FvWW9hSFIwY0hNNkx5OWliM0psWVd4cGN5NXNaV0ZvCllTNWpieTUxYXk5aFptUXZkbVZqY3k5allUQU5CZ2txaGtpRzl3MEJBUXNGQUFPQ0FZRUF3SUNPTzE2YS9rU1cKd0w5UnZyeHhldUxvWFI2TytNNUxoUm4wanZwbXo3SEROK1EydlFCSFlMYWhEQWQrVHc5dFRMRURyUUU0NWc0YQpzN1JNdDd3MlhMUEswQ3oxQWpPYUV2SkhZMGt6YzdyMVBKOXQvYzFvdTZOMTNVUklwWkZrSW9wUG5ZK2k1OVhQCnc3WDd5aGJiK21BRzdLVFRuY3BWRUxpcWgzN3VCeUhIdnBxYkhPQ1docXRvaFFvdmYvdUw1cm0wQm43ZW9UU3YKRXVJdG1OK3RnbVVyeXJ1K3liLzlaRGZ0SDRsa1JwbGprcDhhV0N1ZS8vRFZDQkZwUkdvcnJHS2tseklMRk4xSgpka3ZOaERlZUFrU3dwdVVsazAyK2tkV1UwVGNkZzdTZnpTa1FrS0NQYXlzU2llcWdQMzA3VVJOSmRiVitWdUhHCmN4TW4yZDlrc2ZLVmhWS0lJcE5RMnU3TUJYR0l3ZzV0UzEyaEJxVkVMTEYrRDg3QkpNSEovMmdOSzdFb2pQZ3YKMXZ0MHV2Tkl1LzBxYllwMitlOEJFVTI1MDIvTTU1c3NQelZqc1dBLzZ3VXYwZmZXWW1oUGVNV3JqajBtR0dMNwpJaS9sWngzelU0V043cDZvdkJIU3ZWVWxrNjFpQXJibUxad1FtTlAzYmgrOWFvSVBub2JrCi0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K
        vmGroup: Cryosphere_primary-az-vmgroup
    registry:
      server: registry.vmsp-platform.svc.cluster.local:5000
      service:
        enabled: true
        nodePort: 30000
        port: 5000
        type: ClusterIP
      storageType: pvc
    security:
      enforcePasswordPolicy: "true"
      passwordPolicy:
        complexityConstraint:
          minLength: 15
          minLowercase: 1
          minNumeric: 1
          minSpecial: 1
          minUppercase: 1
          passwordHistory: 5
        expirationConstraint:
          passwordExpirationDays: 365
        lockoutConstraint:
          lockoutEvaluationPeriod: 900
          lockoutMaxAuthFailures: 3
          lockoutPeriod: 900
        policyId: default
        policyInfo:
          description: VSP Default Password Policy
          policyName: VSP Default Password Policy
    sftp:
      directory: /mnt/Sol/General/Backups/VCF/
      host: 192.168.1.4
      port: "22"
      username: leaha
    ssh:
      publicKey: ""
    vcf:
      opsMgmt:
        host: ""
        username: ""
    zot:
      config:
        auth:
          enable: false
status:
  builtInsDigest: 84e657f5f4e5a4d87a2b37464b2f7347b12d376d21f31877aba3202e88f50fb1
  conditions:
  - lastTransitionTime: "2026-05-12T12:51:38Z"
    message: found templates for provided package
    observedGeneration: 1
    reason: MatchedTemplates
    status: "True"
    type: TemplatesIdentified
  - lastTransitionTime: "2026-05-31T20:31:46Z"
    message: 'failed to create or patch helmrelease vmsp-platform.node-problem-detector:
      Internal error occurred: failed calling webhook "validate.kyverno.svc-fail":
      failed to call webhook: Post "https://kyverno-svc.vmsp-policies.svc:443/validate/fail?timeout=10s":
      dial tcp 198.18.219.19:443: connect: connection refused'
    observedGeneration: 11
    reason: DeploymentFailed
    status: "False"
    type: Ready
  observedGeneration: 11
  packageDeployments:
  - name: antrea
    releases:
    - name: antrea
      namespace: vmsp-platform
      ready: "True"
      reason: UpgradeSucceeded
  - name: argo-templates
    releases:
    - message: 'Could not determine release state: failed to retrieve last release
        from storage: query: failed to query with labels: Get "https://192.168.1.130:6443/api/v1/namespaces/vmsp-platform/secrets?labelSelector=name%3Dargo-templates%2Cowner%3Dhelm":
        context deadline exceeded (Client.Timeout exceeded while awaiting headers)'
      name: argo-templates
      namespace: vmsp-platform
      ready: "False"
      reason: StateError
  - name: argo-workflows
    releases:
    - name: argo-workflows
      namespace: vmsp-platform
      ready: "True"
      reason: InstallSucceeded
  - name: cert-manager
    releases:
    - name: cert-manager
      namespace: vmsp-platform
      ready: "True"
      reason: UpgradeSucceeded
  - name: cluster-api-installer
    releases:
    - name: cluster-api-installer
      namespace: vmsp-platform
      ready: "True"
      reason: UpgradeSucceeded
  - name: cluster-autoscaler
    releases:
    - name: cluster-autoscaler
      namespace: vmsp-platform
      ready: "True"
      reason: InstallSucceeded
  - name: descheduler
    releases:
    - name: descheduler
      namespace: vmsp-platform
      ready: "True"
      reason: InstallSucceeded
  - name: envoyproxy-gateway
    releases:
    - name: envoyproxy-gateway
      namespace: vmsp-platform
      ready: "True"
      reason: UpgradeSucceeded
  - name: flux2
    releases:
    - name: fluxcd
      namespace: vmsp-platform
      ready: "True"
      reason: UpgradeSucceeded
  - name: gateway-api
    releases:
    - name: gateway-api
      namespace: vmsp-platform
      ready: "True"
      reason: UpgradeSucceeded
  - name: gatewayconfiguration
    releases:
    - name: gatewayconfiguration
      namespace: vmsp-platform
      ready: "True"
      reason: InstallSucceeded
  - name: kube-prometheus-stack
    releases:
    - name: kube-prometheus-stack
      namespace: vmsp-platform
      ready: "True"
      reason: InstallSucceeded
  - name: kube-vip
    releases:
    - name: kube-vip
      namespace: vmsp-platform
      ready: "True"
      reason: UpgradeSucceeded
  - name: kyverno
    releases:
    - name: kyverno
      namespace: vmsp-platform
      ready: "True"
      reason: UpgradeSucceeded
  - name: kyverno-policies
    releases:
    - name: kyverno-policies
      namespace: vmsp-platform
      ready: "True"
      reason: UpgradeSucceeded
    - message: 'Could not determine release state: failed to retrieve last release
        from storage: query: failed to query with labels: Get "https://192.168.1.130:6443/api/v1/namespaces/vmsp-platform/secrets?labelSelector=name%3Dvmsp-kyverno-policies%2Cowner%3Dhelm":
        context deadline exceeded (Client.Timeout exceeded while awaiting headers)'
      name: vmsp-kyverno-policies
      namespace: vmsp-platform
      ready: "False"
      reason: StateError
  - name: logging-operator
    releases:
    - name: logging-operator
      namespace: vmsp-platform
      ready: "True"
      reason: UpgradeSucceeded
  - name: metadata-service
    releases:
    - name: metadata-service
      namespace: vmsp-platform
      ready: "True"
      reason: InstallSucceeded
  - name: metrics-server
    releases:
    - message: 'Could not determine release state: failed to retrieve last release
        from storage: query: failed to query with labels: Get "https://192.168.1.130:6443/api/v1/namespaces/vmsp-platform/secrets?labelSelector=name%3Dmetrics-server%2Cowner%3Dhelm":
        context deadline exceeded (Client.Timeout exceeded while awaiting headers)'
      name: metrics-server
      namespace: vmsp-platform
      ready: "False"
      reason: StateError
  - name: ndc
    releases:
    - name: ndc
      namespace: vmsp-platform
      ready: "True"
      reason: InstallSucceeded
  - name: node-problem-detector
    releases:
    - name: node-problem-detector
      namespace: vmsp-platform
      ready: "True"
      reason: InstallSucceeded
  - name: proxy-service
    releases:
    - name: proxy-service
      namespace: vmsp-platform
      ready: "True"
      reason: InstallSucceeded
  - name: reloader
    releases:
    - name: reloader
      namespace: vmsp-platform
      ready: "True"
      reason: InstallSucceeded
  - name: seaweedfs
    releases:
    - name: seaweedfs
      namespace: vmsp-platform
      ready: "True"
      reason: UpgradeSucceeded
  - name: strimzi-kafka-operator
    releases:
    - name: strimzi-kafka-operator
      namespace: vmsp-platform
      ready: "True"
      reason: UpgradeSucceeded
  - name: support-bundle
    releases:
    - name: support-bundle
      namespace: vmsp-platform
      ready: "True"
      reason: InstallSucceeded
  - name: synthetic-checker
    releases:
    - name: hooks-server
      namespace: vmsp-platform
      ready: "True"
      reason: UpgradeSucceeded
    - name: synthetic-checker
      namespace: vmsp-platform
      ready: "True"
      reason: UpgradeSucceeded
  - name: trust-manager
    releases:
    - name: trust-manager
      namespace: vmsp-platform
      ready: "True"
      reason: UpgradeSucceeded
  - name: velero
    releases:
    - name: velero
      namespace: vmsp-platform
      ready: Unknown
      reason: DependencyNotReady
  - name: vmsp-agent
    releases:
    - message: 'Could not determine release state: failed to retrieve last release
        from storage: query: failed to query with labels: Get "https://192.168.1.130:6443/api/v1/namespaces/vmsp-platform/secrets?labelSelector=name%3Dvmsp-agent%2Cowner%3Dhelm":
        context deadline exceeded - error from a previous attempt: http2: client connection
        lost'
      name: vmsp-agent
      namespace: vmsp-platform
      ready: "False"
      reason: StateError
  - name: vmsp-api
    releases:
    - name: vmsp-api
      namespace: vmsp-platform
      ready: "True"
      reason: UpgradeSucceeded
  - name: vmsp-backup
    releases:
    - name: vmsp-backup
      namespace: vmsp-platform
      ready: "True"
      reason: UpgradeSucceeded
  - name: vmsp-certs
    releases:
    - name: vmsp-certs
      namespace: vmsp-platform
      ready: "True"
      reason: UpgradeSucceeded
  - name: vmsp-cluster
    releases:
    - name: vmsp-cluster-aerosphere
      namespace: vmsp-platform
      ready: "True"
      reason: UpgradeSucceeded
  - name: vmsp-configs
    releases:
    - name: vmsp-configs
      namespace: vmsp-platform
      ready: "True"
      reason: UpgradeSucceeded
  - name: vmsp-etcd
    releases:
    - name: vmsp-etcd
      namespace: vmsp-platform
      ready: "True"
      reason: UpgradeSucceeded
  - name: vmsp-global-config
    releases:
    - name: vmsp-global-config
      namespace: vmsp-platform
      ready: "True"
      reason: UpgradeSucceeded
  - name: vmsp-hooks
    releases:
    - name: vmsp-hooks
      namespace: vmsp-platform
      ready: "True"
      reason: UpgradeSucceeded
  - name: vmsp-identity
    releases:
    - message: 'Could not determine release state: unable to determine cluster state:
        [failed to determine if ServiceAccount is namespace scoped: failed to get
        restmapping: failed to get server groups: unexpected error when reading response
        body. Please retry. Original error: context deadline exceeded (Client.Timeout
        or context cancellation while reading body), failed to determine if ClusterRole
        is namespace scoped: failed to get restmapping: failed to get server groups:
        Get "https://192.168.1.130:6443/api?timeout=30s": context deadline exceeded
        - error from a previous attempt: http2: client connection lost, failed to
        determine if ClusterRole is namespace scoped: failed to get restmapping: failed
        to get server groups: Get "https://192.168.1.130:6443/api?timeout=30s": context
        deadline exceeded, Get "https://192.168.1.130:6443/apis/policy/v1/namespaces/vmsp-platform/poddisruptionbudgets/vmsp-identity?timeout=30s":
        context deadline exceeded, an empty namespace may not be set when a resource
        name is provided, Get "https://192.168.1.130:6443/apis/rbac.authorization.k8s.io/v1/clusterroles/vmsp-identity-manager-role?timeout=30s":
        context deadline exceeded - error from a previous attempt: http2: client connection
        lost, Get "https://192.168.1.130:6443/apis/rbac.authorization.k8s.io/v1/clusterroles/vmsp-identity-metrics-reader?timeout=30s":
        net/http: request canceled (Client.Timeout exceeded while awaiting headers),
        client rate limiter Wait returned an error: context deadline exceeded - error
        from a previous attempt: http2: client connection lost, Get "https://192.168.1.130:6443/apis/rbac.authorization.k8s.io/v1/clusterrolebindings/vmsp-identity-manager-rolebinding?timeout=30s":
        context deadline exceeded (Client.Timeout exceeded while awaiting headers),
        Get "https://192.168.1.130:6443/apis/rbac.authorization.k8s.io/v1/clusterrolebindings/vmsp-identity-proxy-rolebinding?timeout=30s":
        dial tcp 192.168.1.130:6443: connect: no route to host, Get "https://192.168.1.130:6443/apis/rbac.authorization.k8s.io/v1/namespaces/vmsp-platform/roles/vmsp-identity-leader-election-role?timeout=30s":
        dial tcp 192.168.1.130:6443: connect: no route to host, Get "https://192.168.1.130:6443/apis/rbac.authorization.k8s.io/v1/namespaces/vmsp-platform/rolebindings/vmsp-identity-leader-election-rolebinding?timeout=30s":
        dial tcp 192.168.1.130:6443: connect: no route to host, Get "https://192.168.1.130:6443/api/v1/namespaces/vmsp-platform/services/vmsp-identity-auth-service?timeout=30s":
        dial tcp 192.168.1.130:6443: connect: no route to host, Get "https://192.168.1.130:6443/api/v1/namespaces/vmsp-platform/services/vmsp-identity-webhook-service?timeout=30s":
        dial tcp 192.168.1.130:6443: connect: no route to host, Get "https://192.168.1.130:6443/apis/apps/v1/namespaces/vmsp-platform/deployments/vmsp-identity?timeout=30s":
        dial tcp 192.168.1.130:6443: connect: no route to host, Get "https://192.168.1.130:6443/apis/identity.vmsp.vmware.com/v1alpha1/namespaces/vmsp-platform/accessgrants/fleetbuilder-grants?timeout=30s":
        dial tcp 192.168.1.130:6443: connect: no route to host, Get "https://192.168.1.130:6443/apis/identity.vmsp.vmware.com/v1alpha1/namespaces/vmsp-platform/accessgrants/ops-logs-grants?timeout=30s":
        dial tcp 192.168.1.130:6443: connect: no route to host]'
      name: vmsp-identity
      namespace: vmsp-platform
      ready: "False"
      reason: StateError
  - name: vmsp-operator
    releases:
    - name: vmsp-operator
      namespace: vmsp-platform
      ready: "True"
      reason: UpgradeSucceeded
  - name: vmsp-scripts
    releases:
    - name: vmsp-scripts
      namespace: vmsp-platform
      ready: "True"
      reason: UpgradeSucceeded
  - name: vsphere-cpi
    releases:
    - name: vsphere-cpi
      namespace: vmsp-platform
      ready: "True"
      reason: UpgradeSucceeded
  - name: vsphere-csi
    releases:
    - name: vsphere-csi
      namespace: vmsp-platform
      ready: "True"
      reason: UpgradeSucceeded
  - name: zalando-pg-operator
    releases:
    - name: zalando-postgres-operator
      namespace: vmsp-platform
      ready: "True"
      reason: UpgradeSucceeded
  - name: zot
    releases:
    - name: zot-2
      namespace: vmsp-platform
      ready: "True"
      reason: UpgradeSucceeded
    - name: zot-1
      namespace: vmsp-platform
      ready: "True"
      reason: UpgradeSucceeded
  phase: Failed

Author

  • Leaha

    I am an avid techy and love solving problems and writing guides in my free time

    As a Senior Consultant at Xtravirt, a large part of my job is helping customers get the most out of their products and ensuring its all up to best practices
    My primary focus is in the datacenter with VMware and Veeam

    View all posts

Leave a Reply

Your email address will not be published. Required fields are marked *