Securing a public server while it's a WIP

System Administration
, ,
1

So, I have some semi real world troubleshooting to do.. I find that I am unable to ssh into my Hetzner vps. It hangs for a while and eventually returns a connection timeout. I’ve double checked the configuration on the client side and it has not changed. I haven’t tried in about 3 days, but it was working fine. I have managed to get in through the web console and, with ss -tunlp, can see that sshd is indeed listening on my custom ssh port (12345 for the sake of this discussion).

systemctl status shows that sshd.service is running and sshd.socket is disabled. That is a mystery because I had left it enabled. Does it only enable on connection?

firewall-cmd --list-all still shows 12345/tcp in ports just as I had left it.

Aha.. I missed the lower levels of the OSI model. I have the Hetzner firewall (yes, I have 2 firewalls) pinned down to only allowing traffic from the IP address that my ISP doled out. That IP has changed. Updating it solved the problem.

Well, that means that whomever got my old IP address technically had access to my server if they had the ssh keys, which I never share. What is a better way to secure this VPS while I am piecing things together? Or am I overkilling it, meaning keeping my SSH keys secure is sufficient? Or maybe pay the extra money for a static IP?

2

TL;DR: Forget about firewalling SSH; just disable password authentication. If you have higher-than-average security demands, consider setting up a jump host.

SSH is a very robust piece of software that you really can trust (if you keep it up to date). Disabling password authentication is the only important security measure necessary. With public key authentication only, you’re good.

Assuming there’s no vulnerability in SSH, even running on the standard port 22, there’s nothing an attacker can do to break in. They don’t have your private key – they’re out and stay out.

I concede that there is a practical reason to use a different port than 22, and that’s that you’ll have less log entries of those unsuccessful break-in attempts. People will still find the port if they scan your machine, but that’ll be only a fraction of the overall script kiddie population.

In my humble opinion, paying for a static client IP address just for a firewall rule is overkill in almost any case. For other services, it might make sense, but not for SSH.

That doesn’t mean some pesky security assessment company will not complain about an open SSH port when they check your infrastructure. In that case, there is a solution that’s similar to your approach, but more flexible and convenient.

If you run a single machine that only does SSH, its security posture will be super. Run a nightly SSH package update, and you’re golden. You can then use this machine as a so-called “jump host” or “bastion host”: You SSH first to this machine, and from there, you hop to the actual destination host. On that destination host, you can then lock down SSH to only accept connections from the jump host’s IP address. Mind you, you can do that in /etc/ssh/sshd_config; no firewall rules necessary.

And speaking of built-in, here comes the kicker: SSH is jump-host-aware! That means you can configure your SSH client to always access your destination hosts via the jump host automatically:

Host jumphost
  HostName jumphost.example.com
 
Host servicehost
  HostName servicehost.example.com
  ProxyJump jumphost

This saves you from always having to SSH twice to get where you want.

3

Thanks much. I was wondering if the extra paranoia was really worth it with the firewalls. I had already disabled password auth, root access via ssh and changed the port, things that I have done before. Still it’s good to know what is enough.

Jump hosting is good to know about when I do get to the point of needing a security assessment.

4

I appreciate this. I’ve heard of jump hosts before but it never really clicked until now. Thanks.