Add user filtering with AllowUsers option in sshd_config file:
AllowUsers johndoe@192.168.1.* admin2@192.168.1.* otherid1 otherid2
This allows johndoe and admin2 only from 192.168.1.* addresses and otherid1, otherid2 from anywhere.
In larger environments lots of connections are made by the Plug-In backups. It is recommended to tune the network settings of the invol...