VCF 9.1.1: When vCenter’s SFTP Backup Fails Before It Starts

A VCF 9.1.1 upgrade blocked by vcenter.deployment.migration_upgrade.no_recent_vcenter_backup.error, a vCenter SFTP backup that fails in 31 milliseconds without touching the network, and the workaround that actually ships.

vCenter 9.1 file-based backup fails over SFTP — and blocks the VCF 9.1.1 upgrade precheck

If you’re upgrading VMware Cloud Foundation to 9.1.1 and the vCenter upgrade dies at VCENTER_UPGRADE_PRECHECK, there’s a good chance you’re about to spend an evening on this. Here’s the short version so you don’t have to.

The symptom

SDDC Manager LCM fails the vCenter upgrade with three stacked errors. The top two are just the cascade:

com.vmware.vcf.error.update.vc.rdu.failed
vcenter.deployment.migration_upgrade.validation_failure_error
vcenter.deployment.migration_upgrade.no_recent_vcenter_backup.error

The third one is the actual cause:

The most recent file-based backup of the vCenter Server appliance being upgraded is over 72 hours old. This backup is considered outdated to continue with the upgrade.

vCenter 9.x enforces a hard precheck: a successful native file-based backup within the last 72 hours, or the migration-based upgrade won’t start.

Two things that trip people up immediately:

• Image-level backups don’t count. Veeam, VADP, snapshot-based — none of it registers. The precheck reads vCenter’s own backup history table. Only a VAMI file-based backup satisfies it.
• “Continue without backup” in SDDC Manager does not skip this. That option governs SDDC Manager’s own backup. The 72-hour check lives inside the vCenter upgrade precheck and there is no supported way to bypass it.

So: take a backup, retry the upgrade. Simple — unless your backup won’t run.

The actual problem

Manual backup to the existing SFTP target failed in 3 seconds, 0 bytes transferred. Same target that had been working fine on vCenter 9.0.2 two months earlier.

/var/log/vmware/applmgmt/backup.log tells the story:

Failed to authorize local host with command: sshpass ... ssh-copy-id ... Will proceed with backup through curl
Passwordless ssh tunnel is not working, falling back to CURL
Failed to create backup directory  Plugin error occurred. ErrCode: 200, Args: ()
BackupManager encountered an exception: Failed to create backup directory on backup server.

Note the timestamps on those last lines. Fallback to curl, then error — 31 milliseconds later. For comparison, an SSH connection to that same host in the same log takes ~245 ms just to connect and authenticate. The failure happens before a TCP session is established. Nothing ever reaches the SFTP server, despite the error message pointing squarely at it.

Also worth knowing: the directory named in the VAMI error (M_9.1.0.0300_<timestamp>_Manual) is never created. That path is what VAMI would have written. Don’t waste time checking the server for a partial transfer.

What it isn’t

I verified every one of these. Save yourself the hours:

• Server-side auth. The account authenticates fine.
• Path, chroot, permissions. sftp client connects, pwd shows the real home (no chroot), ls works, mkdir and rmdir succeed.
• Disk space on either end.
• Special characters in the password. The password contained ! twice. Not the issue.
• Proxy. The appliance ships an Envoy system proxy with HTTP_PROXY/HTTPS_PROXY/FTP_PROXY set to localhost ports. Two reasons this is a red herring: curl needs sftp_proxy or all_proxy to affect an sftp:// URL and neither is set, and the Envoy config had a null upstream anyway.
• known_hosts. A curl run from an interactive shell will fail with Could not find a known_hosts file if /root/.ssh/known_hosts doesn’t exist — but the plugin passes -k, which skips host key verification entirely. Tempting theory, wrong.
• The curl binary. CurlLib.py defines two paths, /usr/bin/curl and a curl_lazyloader.py. Only GetUploadCmd uses the lazyloader. Directory creation goes through plain /usr/bin/curl.

Here’s the kicker. Reconstruct the plugin’s exact command from CurlLib.py — same flags, same environment, same host — and it works:

/usr/bin/curl --fail -u 'user:REDACTED' --connect-timeout 120 -k --proxytunnel -v \
  sftp://sftp.example.local/home/backup/vCenter/sn_vcsa01.example.local/ -l

* Authentication complete
M_9.0.2.0_20260728-184150_
..
.
* Connection #0 to host sftp.example.local:22 left intact
rc=0

Authenticates, lists the directory, exit 0. The plugin running the same thing fails in 31 ms.

What I never proved

I didn’t capture the plugin’s actual curl invocation. The approach — wrap /usr/bin/curl in a logging shim, run one backup, read the trace, restore the binary — is sound, but the steps have to happen in that order, with the backup triggered between installing the wrapper and reading the log. I fumbled the sequence more than once and the trace never captured anything.

If you want to try it, the shape is:

# 1. Install
cp -p /usr/bin/curl /usr/bin/curl.real
printf '%s\n' '#!/bin/bash' \
  'echo "=== $(date -u +%FT%T.%3NZ) ARGS: $*" >> /root/curl-trace.log' \
  '/usr/bin/curl.real "$@" 2> >(tee -a /root/curl-trace.log >&2)' \
  'rc=$?; echo "=== rc=$rc" >> /root/curl-trace.log; exit $rc' > /usr/bin/curl
chmod 755 /usr/bin/curl

# 2. Trigger a manual backup in VAMI. Wait for it to fail. Touch nothing.

# 3. Read, then restore
cat /root/curl-trace.log
mv -f /usr/bin/curl.real /usr/bin/curl

Two gotchas: the log goes in /root, not /tmp, because the appliance management service may run under a private tmp namespace. And the password will appear in the ARGS line, so delete the trace afterward. Don’t leave a shell script sitting on /usr/bin/curl on a management vCenter — restore it before you do anything else.

The workaround

Stop fighting it. The SFTP-via-curl fallback is one code path among several, and the others are unaffected.

SMB — worked on the first attempt. 2.94 GB in 1 minute 51 seconds.

smb://nas.example.local/shares/backups/vCenter/sn_vcsa01.example.local/M_9.1.0.0300_..._Manual
Complete

Either way, uncheck stats, events, tasks and alarms. That data isn’t needed for the upgrade precheck and it’s the bulk of the transfer — the SEAT estimate on this appliance was ~1.6 GB raw before compression.

Once VAMI reports Succeeded, go straight back to LCM and retry. The 72-hour clock starts at backup completion, and a retry sitting queued will eat into it.

A note on the July backup

The same appliance backed up successfully to the same SFTP target on 9.0.2 in July. My read: back then the passwordless SSH tunnel worked, so the curl path was never exercised. The SFTP server has since been rebuilt with a newer OpenSSH and the backup account is now SFTP-only — no shell — which makes ssh-copy-id and ssh ... echo ~ fail, which forces the curl fallback, which is where this breaks.

If you want the SSH tunnel path back, give the backup account a real login shell, remove any ForceCommand internal-sftp that applies to it, and install the appliance’s /root/.ssh/id_rsa.pub into its authorized_keys. The plugin will then detect a working tunnel and skip curl entirely. That’s a real loosening of the SFTP account, so weigh it — I went with SMB instead.

One more thing in the log

Every backup run, successful or not, logged this:

Component size command "/etc/vmware/backup/component-scripts/vpxd/vpxd-backup-restore.py --size" failed. rc: 1

It’s not fatal — the job falls back to a full VCDB size estimate and carries on, and the SMB backup completed with this error present. But a vpxd backup script failing on a plain --size call isn’t healthy, and vpxd has to succeed during the real backup. Worth a look if your backup fails later in the job rather than at directory creation.

Takeaways

1. no_recent_vcenter_backup.error means exactly what it says, and there’s no way around it. Only native file-based backups count.
2. If your SFTP backup fails in ~3 seconds with 0 bytes, look at the duration before you look at the server. Sub-second failures are local.
3. “Failed to create backup directory on backup server” can be entirely client-side. The message is misleading.
4. SMB and NFS use different code paths. When SFTP misbehaves, switch rather than debug — especially with a 72-hour window running.
5. Schedule your file-based backups. A daily job to a target you’ve verified means this precheck is never the thing standing between you and an upgrade. It would also have surfaced this failure weeks earlier, in daylight, instead of mid-upgrade.

Point 5 is the one that actually matters. Everything above is an evening I didn’t need to spend.