1. Access ESXi console with remote or direct KVM or ILO
2. Login to ESXi console (F2) with root account (it won't be locked at this level)
3. Navigate to Troubleshooting Options
4. In first line you should see either Disable or Enable ESXi Shell. If its Enable, hit enter once.
5. Then use combination Alt+F1 to switch to ESXi shell
6. Login with root account (it won't be locked at this level)
7. Run following command. This will show number of failed login attempts
pam_tally2 --user root
8. To reset the failures, run following command
pam_tally2 --user root --reset
Now you can access the ESXi with shell and Web UI (vSphere Client).
Run 'pam_tally2 --user root' couple of times to check if there are new failed login attempts.