VCF 9.1.1: One Host, One NFS Cluster, and a Validator That Wouldn’t Budge

Tested on VCF 9.1.1.0, September 2026

I had one spare R740xd after a rebuild and a simple plan: drop it into the management domain as a second cluster, back it with an NFS share off TrueNAS, done by lunch. SDDC Manager had a different plan. The Add Cluster wizard wants two hosts for external storage. So does the API. I have one host.

What follows is how I got it to one, the byte that stood in the way, and the two NFS mistakes I made on the way that had nothing to do with the byte.

The flags everyone recommends

If you run a small VCF lab you already know the two settings people reach for when they are short on hosts:

feature.vcf.vgl-29121.single.host.domain=true      # /home/vcf/feature.properties
bringup.mgmt.cluster.minimum.size=1                # domainmanager application-prod.properties

Both set, services restarted, appliance rebooted. The wizard still wanted two hosts. Fine, the UI does its own counting. But the API said the same thing:

INVALID_NUMBER_OF_MINIMUM_HOSTS
Minimum 2 hosts are required for vLCM cluster with external storage to be created.

Two settings that are supposed to unlock single host, and neither one moved this check. So I stopped setting properties and went to read the code.

Reading the validator

The check lives in V1ClusterValidator, tucked inside the nested jars in vcf-domain-manager.jar. SDDC Manager ships a full JDK, so javap is already on the box. Nothing to copy off, nothing to install.

Two things fell out of the disassembly. The validator does read feature flags. I could see it checking the ones for vSAN two node compute, LAG, IPv6 storage, vLCM host seeding. It does not read vgl-29121. The single host flag has no say on this path, which is exactly why nothing I set had changed anything.

The check itself, in validateMinHostsForExtStorage, is a number baked into the method:

71: invokeinterface List.size()
76: iconst_2
77: if_icmpge 149

That iconst_2 is the 2. No property reads it, no flag bends it. If I wanted one host, that byte had to say one.

One byte

Two copies of the class, same edit in each: iconst_2 (0x05) becomes iconst_1 (0x04). One byte apiece.

I wasn’t going to patch by hardcoded offset and pray. The script matches the byte sequence around the check (size(), then iconst_2, then the if_icmpge jumping 72 bytes on), and it bails if that sequence shows up anything other than once. On a build that doesn’t match, that is the behavior I want.

You patch the jar sitting on your own appliance, in place. Nothing is downloaded, nothing is shared. That keeps you on the exact build Broadcom gave you, and the byte check runs against your jar and no one else’s.

The one gotcha: Spring Boot needs those nested jars stored uncompressed or the loader won’t come up. Python’s zipfile keeps each entry’s original storage method when you hand the ZipInfo back to writestr, so the rebuilt jar stays laid out the way it expects.

#!/usr/bin/env python3
# LAB ONLY / UNSUPPORTED. Lowers the vLCM external storage cluster minimum (2 to 1)
# in SDDC Manager domainmanager. Verified on VCF 9.1.1.0. Aborts if the bytecode doesn't match.
import sys, zipfile, io

SRC = '/opt/vmware/vcf/domainmanager/vcf-domain-manager.jar'
OUT = sys.argv[1] if len(sys.argv) > 1 else 'vcf-domain-manager.jar.new'

T = {
 'BOOT-INF/lib/sddcmanager/domainmanager/clustermanager-is/clustermanager-is-rest-api-controller/libclustermanager-is-rest-api-controller.jar':
   'com/vmware/vcf/clustermanager/controller/v1/validation/V1ClusterValidator.class',
 'BOOT-INF/lib/sddcmanager/domainmanager/clustermanager-is/clustermanager-is-common/libclustermanager-is-common.jar':
   'com/vmware/vcf/clustermanager/services/validation/V1ClusterValidator.class',
}
PAT = bytes([0x01, 0x00, 0x05, 0xa2, 0x00, 0x48])   # invokeinterface size() | iconst_2 | if_icmpge +72

def patch_class(b):
    n = b.count(PAT)
    assert n == 1, f"pattern found {n} times, aborting (different build?)"
    i = b.index(PAT) + 2
    return b[:i] + b'\x04' + b[i+1:]

def rewrite(src, repl):
    out = io.BytesIO()
    with zipfile.ZipFile(src) as zin, zipfile.ZipFile(out, 'w') as zout:
        zout.comment = zin.comment
        for info in zin.infolist():
            data = zin.read(info.filename)
            if info.filename in repl:
                data = repl[info.filename](data)
                print('patched', info.filename)
            zout.writestr(info, data)
    return out.getvalue()

repl = {inner: (lambda c: lambda jb: rewrite(io.BytesIO(jb), {c: patch_class}))(c)
        for inner, c in T.items()}
open(OUT, 'wb').write(rewrite(SRC, repl))
print('wrote', OUT)

Check the byte flipped and the jar still opens before you touch anything live:

javap -p -c V1ClusterValidator.class | grep -A2 ' 71:'
# you want:  76: iconst_1

python3 -c "import zipfile; print(zipfile.ZipFile('vcf-domain-manager.jar.new').testzip())"
# you want:  None

Keep the stock jar first, then swap:

cp -p /opt/vmware/vcf/domainmanager/vcf-domain-manager.jar ./vcf-domain-manager.jar.orig
systemctl stop domainmanager
install -o vcf_domainmanager -g vcf -m 600 vcf-domain-manager.jar.new \
  /opt/vmware/vcf/domainmanager/vcf-domain-manager.jar
systemctl start domainmanager

domainmanager needs a minute or two. If something’s off, put the .orig jar back the same way and restart. I ran the validation again and it came back clean. That was a good moment.

Building the cluster over the API

The wizard still won’t budge, so the build is all curl and jq from the SDDC Manager shell. Commission the host into a network pool with an NFS network first, and make sure the export is actually up and writable, because that last part is where I tripped twice.

Step 1: token

SSO admin account. Put the JSON in single quotes so bash leaves the ! in your password alone:

TOKEN=$(curl -sk -X POST https://localhost/v1/tokens -H 'Content-Type: application/json' \
  -d '{"username":"[email protected]","password":"<your-password>"}' | jq -r .accessToken)
echo ${TOKEN:0:20}

You should see the front of a token. null means the login failed. Tokens die after about an hour, so if a later call 401s, rerun this one.

Step 2: the IDs

Host ID, domain ID, image ID:

HOST_ID=$(curl -sk -H "Authorization: Bearer $TOKEN" "https://localhost/v1/hosts?status=UNASSIGNED_USEABLE" \
  | jq -r '.elements[] | select(.fqdn=="<host-fqdn>") | .id')
DOMAIN_ID=$(curl -sk -H "Authorization: Bearer $TOKEN" https://localhost/v1/domains \
  | jq -r '.elements[] | select(.name=="<domain-name>") | .id')
echo "HOST=$HOST_ID DOMAIN=$DOMAIN_ID"

curl -sk -H "Authorization: Bearer $TOKEN" https://localhost/v1/personalities \
  | jq -r '.elements[] | "\(.personalityId)  \(.personalityName)"'

Grab the image that matches your other clusters and note its ID.

Step 3: the existing TEP pool

Reuse the domain’s host TEP pool instead of making a new one. Define your own range in the same subnet and NSX rejects it for overlapping, which is the next mistake I would have made:

NSX_ID=$(curl -sk -H "Authorization: Bearer $TOKEN" https://localhost/v1/nsxt-clusters | jq -r '.elements[0].id')
curl -sk -H "Authorization: Bearer $TOKEN" https://localhost/v1/nsxt-clusters/$NSX_ID/ip-address-pools \
  | jq '.. | objects | select(has("availableIpAddresses")) | {name, description, availableIpAddresses}'

Find the ESXi host overlay TEP pool and check it has a couple of free addresses.

Step 4: the spec

Save this as /root/cluster-spec.json. Keep it in /root, not /tmp, or a reboot eats it. I found that out the tedious way.

{
  "domainId": "<domain-id>",
  "deployWithoutLicenseKeys": true,
  "computeSpec": {
    "clusterSpecs": [{
      "name": "<cluster-name>",
      "clusterImageId": "<image-id>",
      "hostSpecs": [{
        "id": "<host-id>",
        "hostNetworkSpec": {
          "vmNics": [
            { "id": "vmnic4", "vdsName": "<vds-name>" },
            { "id": "vmnic5", "vdsName": "<vds-name>" }
          ]
        }
      }],
      "datastoreSpec": {
        "nfsDatastoreSpecs": [{
          "datastoreName": "<datastore-name>",
          "nasVolume": { "serverName": ["<nfs-ip>"], "path": "<nfs-export-path>", "readOnly": false }
        }]
      },
      "networkSpec": {
        "vdsSpecs": [{
          "name": "<vds-name>",
          "portGroupSpecs": [
            { "name": "<vds-name>-pg-mgmt", "transportType": "MANAGEMENT" },
            { "name": "<vds-name>-pg-vmotion", "transportType": "VMOTION" },
            { "name": "<vds-name>-pg-nfs", "transportType": "NFS" }
          ]
        }],
        "nsxClusterSpec": {
          "nsxTClusterSpec": {
            "geneveVlanId": <tep-vlan>,
            "ipAddressPoolSpec": { "name": "<existing-host-tep-pool>" }
          }
        }
      }
    }]
  }
}

Set vmnic4 and vmnic5 to whatever uplinks the host actually has. Then run it through jq once so a stray comma or leftover placeholder fails here instead of two calls later:

jq '{domainId, deployWithoutLicenseKeys}' /root/cluster-spec.json

Step 5: validate

curl -sk -X POST -H "Authorization: Bearer $TOKEN" -H 'Content-Type: application/json' \
  https://localhost/v1/clusters/validations -d @/root/cluster-spec.json \
  | jq '{resultStatus, failed: [.validationChecks[]? | select(.resultStatus!="SUCCEEDED") | .errorResponse | {errorCode, message}]}'

On a patched appliance resultStatus comes back SUCCEEDED with an empty failed list. The result is in the POST response, so there is nothing to poll.

Step 6: create and watch

curl -sk -X POST -H "Authorization: Bearer $TOKEN" -H 'Content-Type: application/json' \
  https://localhost/v1/clusters -d @/root/cluster-spec.json | tee /root/create.json | jq '{id, status}'

Follow it in the UI under Tasks, or from the shell:

TASK_ID=$(jq -r .id /root/create.json)
curl -sk -H "Authorization: Bearer $TOKEN" https://localhost/v1/tasks/$TASK_ID \
  | jq '{status, failed: [.subTasks[]? | select(.status=="FAILED") | .name]}'

When a subtask fails, fix the cause and hit Retry on the task in the UI. It resumes where it stopped instead of starting over.

The parts that actually went wrong

The byte was the easy bit. Everything after it was me.

First spec, I defined a fresh TEP pool in the same subnet as the domain’s existing one. NSX refused the overlap outright. That is why step 3 reuses the pool by name. Lesson learned before it cost me much.

Then this, which for a second had me convinced there was a second check to patch:

Cannot skip 1 ESXi Host(s) as only 0 ESXi host(s) would remain and the minimum is 1

It isn’t a limit. When a step fails on a single host cluster, the workflow tries to skip the bad host and carry on, and with one host there is nothing to carry on with. The message you want is the failure right before this one. Chase that, not this.

For me that earlier failure was NFS, and it went wrong twice.

The first time the datastore step hung for thirty seconds and quit. A permissions problem fails on the spot, so a clean thirty second stall means nobody is answering. From the SDDC Manager shell:

nc -zv <nfs-ip> 2049
nc -zv <nfs-ip> 111

Both said “Connection refused.” The NFS service on TrueNAS was off. That is the whole story. I had also turned off the NFS connectivity precheck earlier to get past validation, which is the deal you make with that bypass: the problem you skipped shows up later and says less.

Second time the mount took but the very next step, “Validate Datastore Availability,” died on “Failed to create file inside datastore.” ESXi could see the share and could not write to it. No root mapping on the export, so root was squashed to nobody. Set Maproot user and group to root on the share and it wrote fine.

After that the workflow went the distance: image remediation, HA, DRS, NSX host prep. A few minutes later the cluster was Active with one host next to the original three.

Then the next upgrade eats it

Every SDDC Manager upgrade lays down a fresh vcf-domain-manager.jar and the minimum is two again. The cluster you already built does not care; nothing about it leans on the patch once it exists. You only patch again if you want to build another external storage cluster on one host.

If you do, disassemble the method on the new build before running anything. The script refuses a jar whose bytes don’t match, which is the point of the check, but look at the method yourself rather than trusting that a later build lines up with this one.

So

You would never do this in production. You would buy a second host. But in a lab, where the wall is hardware and not sense, a single host NFS cluster in an existing domain is one byte away, once you know the flag everyone points you at doesn’t touch the check that’s actually stopping you.