While reading the system journal of my server, I noticed a bunch of evil looking entries and finally found in the Python tool fail2ban a satisfying but not an overall answer to the problem.

The system journal can be inspected by:

$ sudo journalctl -r

Feb 07 14:29:45 dummy sshd[763]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=XXX.XX.X.XXX  user=root
Feb 07 14:29:45 dummy sshd[764]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=YY.YYY.YY.YY  user=root
Feb 07 14:29:47 dummy sshd[760]: error: PAM: Authentication failure for root from XXX.XX.X.XXX
Feb 07 14:29:47 dummy sshd[758]: error: PAM: Authentication failure for root from YY.YYY.YY.YY
Feb 07 14:29:48 dummy sshd[765]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=YY.YYY.YY.YY  user=root
Feb 07 14:29:48 dummy sshd[766]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=XXX.XX.X.XXX  user=root
Feb 07 14:29:50 dummy sshd[758]: error: PAM: Authentication failure for root from YY.YYY.YY.YY
Feb 07 14:29:50 dummy sshd[760]: error: PAM: Authentication failure for root from XXX.XX.X.XXX
Feb 07 14:29:50 dummy sshd[767]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=YY.YYY.YY.YY  user=root
Feb 07 14:29:51 dummy sshd[768]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=XXX.XX.X.XXX  user=root
Feb 07 14:29:52 dummy sshd[758]: error: PAM: Authentication failure for root from YY.YYY.YY.YY
Feb 07 14:29:52 dummy sshd[758]: Received disconnect from YY.YYY.YY.YY port 58773:11:  [preauth]
Feb 07 14:29:52 dummy sshd[758]: Disconnected from authenticating user root YY.YYY.YY.YY port 58773 [preauth]
Feb 07 14:29:52 dummy sshd[760]: error: PAM: Authentication failure for root from XXX.XX.X.XXX
Feb 07 14:29:53 dummy sshd[760]: Postponed keyboard-interactive for root from XXX.XX.X.XXX port 19193 ssh2 [preauth]
Feb 07 14:29:53 dummy sshd[769]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=XXX.XX.X.XXX  user=root
Feb 07 14:29:56 dummy sshd[760]: error: PAM: Authentication failure for root from XXX.XX.X.XXX
Feb 07 14:29:56 dummy sshd[760]: Failed keyboard-interactive/pam for root from XXX.XX.X.XXX port 19193 ssh2
Feb 07 14:29:56 dummy sshd[760]: Postponed keyboard-interactive for root from XXX.XX.X.XXX port 19193 ssh2 [preauth]

Two hosts (later I noticed a few more), lets call them XXX.XX.X.XXX and YY.YYY.YY.YY for now, are permanently trying to log in to my server as root.

Using an arbitrary IP-Address lookup service, I was told be attacked from hosts in Asia. The country was reported as well, but it does not matter for this blog post. In my opinion the worth of this information is doubtful anyways.

But for now, I write about what I did and what prevented greater damage so far:

A good decision at server setup was to only permit logins via public-key cryptography and not passwords. Especially root should never be able to login by a plain ssh command! To achieve this, the following entries are set in /etc/ssh/sshd_config:

PermitRootLogin no
PubkeyAuthentication yes
PasswordAuthentication no

This is in my opinion the most important layer of defense against brute force attacks. But as my system journal reveals, this is by far not enough. I don’t want my server to waste time with brute force attacks. It should simply ignore any communication with them.

A promising approach without writing black- or whitelists on my own is the Python software fail2ban. Like the name suggests, too many failed login attempts (= brute force attacks) automatically result in a permanent ban from my server. In general I consulted three sources for the fail2ban setup:

First one has to install the software. openSUSE makes this task pretty comfortable by typing

sudo zypper in fail2ban

Then I edited as root the configuration file /etc/fail2ban/jail.local:

# Do all your modifications to the jail's configuration in jail.local!

[DEFAULT]
bantime = 1d

# SSH servers
[sshd]
enabled = true
port    = ssh
logpath = %(sshd_log)s
backend = %(sshd_backend)s

and finally started the already available service:

sudo systemctl start fail2ban.service

Now each suspicious login behavior will result in a one day 1d ban. To verify this theory, there are several ways to check. The first method is to read the fail2ban log:

$ sudo cat /var/log/fail2ban.log

2019-02-04 16:39:40,379 fail2ban.server         [2885]: INFO    --------------------------------------------------
2019-02-04 16:39:40,379 fail2ban.server         [2885]: INFO    Starting Fail2ban v0.10.3.fix1
2019-02-04 16:39:40,385 fail2ban.database       [2885]: INFO    Connected to fail2ban persistent database '/var/lib/fail2ban/fail2ban.sqlite3'
2019-02-04 16:39:40,386 fail2ban.database       [2885]: WARNING New database created. Version '2'
2019-02-07 14:31:02,652 fail2ban.server         [2885]: INFO    Shutdown in progress...
2019-02-07 14:31:02,652 fail2ban.server         [2885]: INFO    Stopping all jails
2019-02-07 14:31:02,652 fail2ban.database       [2885]: INFO    Connection to database closed.
2019-02-07 14:31:02,652 fail2ban.server         [2885]: INFO    Exiting Fail2ban
2019-02-07 14:31:02,737 fail2ban.server         [907]: INFO    --------------------------------------------------
2019-02-07 14:31:02,737 fail2ban.server         [907]: INFO    Starting Fail2ban v0.10.3.fix1
2019-02-07 14:31:02,739 fail2ban.database       [907]: INFO    Connected to fail2ban persistent database '/var/lib/fail2ban/fail2ban.sqlite3'
2019-02-07 14:31:02,740 fail2ban.jail           [907]: INFO    Creating new jail 'sshd'
2019-02-07 14:31:02,747 fail2ban.jail           [907]: INFO    Jail 'sshd' uses systemd {}
2019-02-07 14:31:02,748 fail2ban.jail           [907]: INFO    Initiated 'systemd' backend
2019-02-07 14:31:02,748 fail2ban.filter         [907]: INFO      maxLines: 1
2019-02-07 14:31:02,773 fail2ban.filtersystemd  [907]: INFO    [sshd] Added journal match for: '_SYSTEMD_UNIT=sshd.service + _COMM=sshd'
2019-02-07 14:31:02,773 fail2ban.filter         [907]: INFO      maxRetry: 5
2019-02-07 14:31:02,774 fail2ban.filter         [907]: INFO      encoding: UTF-8
2019-02-07 14:31:02,774 fail2ban.actions        [907]: INFO      banTime: 86400
2019-02-07 14:31:02,774 fail2ban.filter         [907]: INFO      findtime: 600
2019-02-07 14:31:02,776 fail2ban.jail           [907]: INFO    Jail 'sshd' started
2019-02-07 14:31:02,796 fail2ban.filter         [907]: INFO    [sshd] Found YY.YYY.YY.YY - 2019-02-07 14:21:03
2019-02-07 14:31:02,797 fail2ban.filter         [907]: INFO    [sshd] Found XXX.XX.X.XXX - 2019-02-07 14:21:04
... many many lines like this ...
2019-02-07 14:31:02,976 fail2ban.actions        [907]: NOTICE  [sshd] Ban YY.YYY.YY.YY
2019-02-07 14:31:03,016 fail2ban.actions        [907]: NOTICE  [sshd] Ban XXX.XX.X.XXX
2019-02-07 14:31:03,022 fail2ban.actions        [907]: WARNING [sshd] YY.YYY.YY.YY already banned
2019-02-07 14:31:03,022 fail2ban.actions        [907]: WARNING [sshd] XXX.XX.X.XXX already banned
...

Et voilà, XXX.XX.X.XXX and YY.YYY.YY.YY misbehaved and were banned! :smile: To get even more trust and understanding in the fail2ban mechanism, one can take a look at iptables, the system tool, where fail2ban adds appropriate entries:

$ sudo iptables -L

...
Chain f2b-sshd (1 references)
target  prot  opt  source        destination
REJECT  all   --   YY.YYY.YY.YY  anywhere    reject-with icmp-port-unreachable
REJECT  all   --   XXX.XX.X.XXX  anywhere    reject-with icmp-port-unreachable
...

My system is of no such importance, that I dare write about this attack to learn from it. Maybe (and in case of critical systems hopefully) other system administrators know better and can teach me how to cope with such a delicate and mean situation.

All in all I hope not to be part of a botnet or similar right now and that attackers no longer find my sever to be a patient listener :wink: