How to protect ESXi from root account lockout
I have a VMWare ESXi instance, running version 6.0.0. Our staff was locked out of the ESXi thick client (the "vSphere Client" Windows application) for a substantial amount of time today. We were presented with an "incorrect username or password" error message when attempting to log in. After some research we determined that we were being locked out of our own ESXi host due to v6.0's root lockout feature which locks down the account for a set amount of time (default: 2 minutes) after 3 consecutive failed password attempts. It seems that the attacker continued for some hours until finally relenting. At that point we were able to log in ourselves using the root account.
We are a bit confused about why this could happen. The server is hosted in a fairly large and reputable data centre and is a true dedicated instance. However, said facility wants to charge rather excessive rates to put this VM server behind a hardware firewall. So we have been relying on the built-in firewall of ESXi.
On the Configuration -> Security Profile -> Firewall section, we have the following services (which are defined by default) to be IP restricted, to only allow our office IP:
- SSH Server
- vSphere Web Access
- vSphere Web Client
- vsanvp
- vMotion
Despite this, it seems the attacker is still able to at least get through and somehow trigger a 'wrong password' error because the server's ESXi event log shows numerous lines like this:
Remote access for ESXi local user account 'root'
has been locked for 120 seconds after 563 failed
login attempts.
This is despite the fact that only our office IP is authorized, and we know that nobody here is initiating this.
What are we doing wrong?
a) You shouldn't be using the .net/Windows client, it goes away entirely with 6.5 which is imminent and VMware have been strongly urging users to move away from it for literally years now.
b) I'm unclear, are you all logging into the host directly, i.e. without a vCenter and if so are you logging in as root?
c) It appears you've not put the host into strict lockdown mode - I'd disable SSH too, as a service and in the firewall.
I read a lot of "Disable SSH" or "Malicious usertrying to brute force"... if your ESXi aren't exposed to internet (untrust), and if SSH is disabled, this is likely not the cause if your lockouts.
I've had all ESXi host getting regularly locked out in the same fashion.
After logs investigation, it appeared to be caused by a vendor script (Lenovo=IBM):
/etc/cim/lenovo/refresh.sh
I resolved the issue by:
- Removing the call to the scrip from crontab (/var/spool/cron/crontab/root)
- Removing the vendor directory scripts (/etc/cim/lenovo/)
- Removing Lenovo extensions (https://support.lenovo.com/au/en/solutions/ht502599)
Note1: Removing only extensions (3.) did not resolve my issue.
Note2: Your vendor might be different from mine (Dell, HP) so the directory would be different, check the cron file.
The root cause of the issue was: I moved from a vendor image (https://my.vmware.com/web/vmware/details?productId=352&downloadGroup=OEM-ESXI55U3A-LENOVO) to a generic ESXi 6.5 image. In the process extensions and scripts weren't cleaned by VMware Update Manager.
