We run NSX with Avi Load Balancer, and every so often the Avi Controller manages to lock the NSX admin account. Usually it is credential drift somewhere. The Controller keeps retrying, blows past the auth failure limit, and then the load balancer cannot talk to NSX at all.
There is a documented fix. You add the Controller IP to lockout_immune_addresses and its API calls stop counting against the lockout counter. One field. Five minutes.
It took me considerably longer than five minutes, because the procedure in the KB is what locked me out.
What the KB tells you to do
The doc is Prevent Password Lockout on Local Manager Nodes. GET the config to a file, patch it with sed, PUT it back:
curl -k -X GET -H "Content-Type: application/json" -u 'admin' \
https://localhost/api/v1/cluster/api-service > api-service.txt
sed -i '$i\,\"lockout_immune_addresses\":[ \"IP_Address\", \"IP_Address\", \"IP_Address\" ]' api-service.txt
curl -k -X PUT -H "Content-Type: application/json" -u 'admin' \
https://localhost/api/v1/cluster/api-service -d @api-service.txtI ran it more or less as written. Three separate things went sideways.
The sed only works by luck
sed '$i\...' inserts before the last line of the file. That gives you valid JSON only if the GET came back pretty printed with a lone } on the final line.
It does, on my build. But nothing about the API promises that, and the procedure does not check. Change NSX versions, stick a proxy in the path, have something negotiate compression differently, and the response comes back on one line. Now your insert has landed in front of the whole document and you are about to PUT garbage.
So check first, every time:
python3 -m json.tool api-service.txtIf that errors, throw the file away and start over.
The sed can also quietly wipe existing entries
This one did not bite me but it is the one I would actually worry about.
sed has no idea it is editing JSON. If lockout_immune_addresses is already in the config, the one liner just bolts a second copy of the key onto the end. Parsers generally take the last one, so whatever was in there before is gone. No error. Nothing in the output looks wrong.
If you are running Federation, your Local Managers probably already have the Global Manager addresses sitting in that list. Run the KB command and you have evicted the GM to make room for your load balancer. Congratulations, new lockout, worse than the old one.
Look before you touch it, and use something that parses JSON instead of doing string surgery on it:
python3 - <<'EOF'
import json
p = 'api-service.txt'
d = json.load(open(p))
existing = d.get('lockout_immune_addresses', [])
new = ["172.31.110.41"]
d['lockout_immune_addresses'] = sorted(set(existing) | set(new))
json.dump(d, open(p, 'w'), indent=2)
print(d['lockout_immune_addresses'])
EOFIt is about the same amount of typing. It merges instead of overwriting, and it blows up loudly on bad input rather than quietly on input that is valid but wrong.
The part that actually got me
Two things here and they compound.
First, IP_Address in that sed is a placeholder. Paste it as is and you now have three entries literally reading IP_Address. That is perfectly good JSON and NSX will accept the PUT without complaint.
Second, and this is the real problem: every curl -u 'admin' prompts you for a password interactively. If you copy a block of two commands out of a runbook and paste it into your terminal, the shell feeds the second command text straight into the first curl password prompt. You just submitted curl -k -X GET -u admin https://localhost/api/v1/cluster/api-service as your password.
Do that a few times while you are fixing the placeholder problem and you get this:
{"module_name":"common-services","error_message":"The credentials were incorrect or the account specified has been locked.","error_code":403}Defaults on 9.1:
vb-m01-nsx01a> get auth-policy api lockout-period
900 seconds
vb-m01-nsx01a> get auth-policy api max-auth-failures
5Five strikes, fifteen minutes.
And the bit I find genuinely funny in hindsight: localhost is not immune. My own verification curls, from a root shell on the Manager itself, are what locked the account. The procedure for preventing lockouts has no protection against causing one.
Spotting it quickly
Watch the byte count. The real config object is around 2 KB. The 403 body is 141 bytes.
100 2109 0 2109 0 0 121k 0 --:--:-- --:--:-- --:--:-- 128k
100 141 100 141 0 0 9379 0 --:--:-- --:--:-- --:--:-- 10071I lost a few minutes here because I was piping the GET into grep lockout, got nothing back, and assumed my write had not taken. It had. The grep found nothing because it was grepping an error message. Ask for the status code instead:
curl -k -X GET -u 'admin' https://localhost/api/v1/cluster/api-service \
-o /tmp/verify.txt -w '\nHTTP %{http_code}\n'
cat /tmp/verify.txtGetting back in
You wait. That is the whole recovery.
Every auth attempt during the lockout resets the 900 second clock, so hammering it makes things worse. Stop, wait it out, then make one attempt with a password you are certain about.
You are not locked out of the appliance, only the API, so from your root shell:
su adminNo password challenge, and you land in the NSX CLI where the get auth-policy commands live. It is a restricted shell so do not bother trying curl from in there, I did and it just tells you the command does not exist. exit back to root when your fifteen minutes are up.
How I would do it now
# fetch
curl -k -X GET -H "Content-Type: application/json" -u 'admin' \
https://localhost/api/v1/cluster/api-service > api-service.txt
# see what is already in there
python3 -m json.tool api-service.txt | grep -A5 lockout
# merge
python3 - <<'EOF'
import json
p = 'api-service.txt'
d = json.load(open(p))
d['lockout_immune_addresses'] = sorted(
set(d.get('lockout_immune_addresses', [])) | {"172.31.110.41"}
)
json.dump(d, open(p, 'w'), indent=2)
print(d['lockout_immune_addresses'])
EOF
# push
curl -k -X PUT -H "Content-Type: application/json" -u 'admin' \
https://localhost/api/v1/cluster/api-service -d @api-service.txtOne command at a time. Wait for each password prompt to finish before you send the next line. If you take nothing else from this, take that.
You do not need a separate GET to verify, because the PUT hands the stored object back to you:
"lockout_immune_addresses" : [ "172.31.110.41" ],
"_last_modified_user" : "admin",
"_revision" : 1_revision going from 0 to 1 is your proof it committed, and _last_modified_user flipping from system to admin is a nice second signal. That response comes from the server after the write, so it is trustworthy even if a later GET falls over.
While we are on _revision: it is optimistic concurrency. The PUT sends back whatever value your GET returned, so if anything else modifies the api-service config in between, you will get a mismatch. Always GET fresh. Do not reuse a file from an earlier session.
Which addresses belong in the list
Worth thinking about for a minute, because getting it wrong means you are doing this again.
Every Avi Controller node IP plus the cluster VIP. On a 3 node Controller cluster, outbound calls to NSX can come from the individual node addresses even though the VIP handles inbound. Immunize only the VIP and you will still get locked out by whichever node was doing the talking. Check what is actually connecting:
netstat -an | grep ':443' | grep <your-controller-subnet>Use the address NSX sees, not the one configured on the Controller. If there is NAT in between those are not the same thing, and the immune check runs against the source address NSX observes.
On scope: this is cluster wide config, so one PUT covers every Manager node. It does not replicate across Federation sites though, so each Local Manager cluster needs doing separately.
Should localhost be immune?
I thought about it, given that local curls are what got me. Adding 127.0.0.1 means routine API poking from the Manager shell stops counting against you.
The downside is that local brute force against admin would no longer lock the account. Realistically that only helps someone who has already got a shell on the NSX Manager, and if that has happened you have bigger things to deal with. I left it off anyway, mostly because I do not want to explain the deviation at audit time. Running one command at a time solves the same problem for free.
The bit nobody puts in the runbook
lockout_immune_addresses lives in ApiServiceConfig. It survives reboots, it survives upgrades, and it is completely invisible day to day. Nothing in the UI shows it. Nothing alerts you when it goes stale.
Rebuild the Avi Controller, or re-IP it, and this entry is silently wrong. Everything keeps working until it does not, and when it does break the symptom is a locked out admin account and a load balancer that has stopped talking to NSX. Nothing about that points you at a config field you set a year and a half ago.
Go add it to the Controller rebuild runbook while you still remember why it is there.
Tested on 9.1.0.0200.25524172. Lab addresses in the examples, swap in your own.
