Introduction

The public internet is like a hurricane and your webserver is like sticking your head outside in the middle.

This was what I was told by someone when I noticed that my fairly new server was getting hit by script kiddies. He was trying to assure me that everything was fine.

Unfortunately, I had made a few mistakes when setting up the server, so I was vulnerable. He, and others, helped me fix the problems, and when I was done, he said,

Once that’s done, memorize all the procedures you did and help anyone who hasn’t seen the light.

I am not any good at memorization, so I am writing this post/tutorial instead.

Prerequisites

This tutorial assumes a basic set of knowledge:

1. The user must be comfortable using a command-line interface.
2. The user must be comfortable in their operating system of choice.
3. The user must know how to install software in their operating system of choice.
4. The user must know what root is.
5. The user must be running a system that is Unix-like.
6. The user must know the basics of using ssh, including creating and setting up keys.
7. The user must also know how to restart a service on their server.

Also, the first thing that every new sysadmin needs to understand, in my opinion, is ports. Before you attempt to run your own webserver, make sure you understand what ports are and their basic usages.

Conventions

In this post, I will use a # character at the beginning of all commands that must be run as root, and a $ character for commands that must be run as the user that will be set up during the process.

Change root Password

Assuming you are not trying to run a server out of your house, which is not a good idea, you are probably renting a server from a cloud provider. And when you receive access, you are probably going to be given root access through ssh with a password.

So log into your server, probably as root. Then, change the password for root:

# passwd root

When prompted, enter a new password. When I first made my server, I generated a random password that was over 100 characters long.

Setting Up a Firewall

We have to set up a firewall to block access to all of the ports that you won’t use, so let’s do that.

First, pick a random port between 1000 and 65535. Then, install nftables on your server. In Arch, you can do that with the following command:

# pacman -S nftables

Then, open the file /etc/nftables.conf and insert the following contents:

#!/usr/bin/nft -f
# ipv4/ipv6 Simple & Safe Firewall
# you can find examples in /usr/share/nftables/

table inet filter {
  chain input {
    type filter hook input priority 0; policy drop;

    # allow established/related connections
    ct state {established, related} accept

    # early drop of invalid connections
    ct state invalid drop

    # allow from loopback
    iifname lo accept

    # allow icmp
    ip protocol icmp accept
    ip6 nexthdr icmpv6 accept

    # allow ssh and web
    tcp dport { 22, 80, 443, <random_port_number> } accept

    # everything else
    counter reject with icmpx type port-unreachable
  }
  chain forward {
    type filter hook forward priority 0;
    drop
  }
  chain output {
    type filter hook output priority 0;
  }

}

where <random_port_number> is changed to the random number you just picked.

Then enable and start the nftables service. The details of doing so will depend on the OS you picked, but for Arch Linux, using systemd, the commands are:

# systemctl enable nftables
# systemctl start nftables

Locking Down ssh

The next thing we need to do is to get rid of root ssh access.

Add a New User

Next, you need to add a new user:

# useradd -a <username>

Be sure to replace <username> as appropriate.

Next, set a password for the user:

# passwd <username>

Check that you can login through ssh as the user that you have just created, with the password that you have set.

Give the User sudo Permissions

Next, we are going to give that user the ability to run sudo.

First, install sudo. In Arch, that is as simple as:

# pacman -S sudo

But it may be different for your server.

Next, edit the sudoers file. The command to do so is:

# visudo

On Arch, for some reason, you have to use nano to edit the file, like so:

# EDITOR=nano visudo

In that file you will find two lines that say:

## Uncomment to allow members of group wheel to execute any command
# %wheel ALL=(ALL) ALL

The above two lines might substitute sudo for wheel.

Uncomment the line, like so:

## Uncomment to allow members of group wheel to execute any command
%wheel ALL=(ALL) ALL

Now, before you save the file and leave, take note of whether it says wheel or sudo. That’s important later.

Save the file and close it. If it closes successfully, you can move on.

Now we have to add the user to the group that can run sudo. If the file you just edited used wheel, run the following command:

# usermod -a -G wheel <username>

and if it used sudo, run the following command:

# usermod -a -G sudo <username>

In both cases, replace <username> as appropriate.

Once that is done, use your user’s ssh session to run the following command:

$ sudo su -

When prompted, enter your user’s password.

You should get a root shell. If you did, run the following command:

# exit

You have successfully given sudo access to your user.

Make ssh Use Public Keys

Make sure you have, on your home computer, already set up an ssh key pair.

Once that is done, login to your user on the server, not root.

Run the following commands:

$ mkdir .ssh
$ touch .ssh/authorized_keys
$ chmod -R 700 .ssh
$ chmod 600 .ssh/authorized_keys

Those four commands create and set up a file that will hold your public key, which will allow you to ssh into the server without a password.

On your home computer, find the file that holds the public key of the key pair you just set up. It should be at ~/.ssh/<key_name>.pub.

Open that file in a text editor.

Copy the text in that file, and then on your user’s ssh session, run the following:

$ nano .ssh/authorized_keys

Paste the text you just copied from your public key. Save the file, and close it.

Next, on your root ssh session (open a new one if you already closed the previous one), open the /etc/ssh/sshd_config file in a text editor.

You should see a line that contains either:

PubKeyAuthentication yes

or:

PubKeyAuthentication no

Make sure it says:

PubKeyAuthentication yes

Save the file, and close it.

Restart the ssh service on your server. On Arch Linux, which uses systemd, the command to do so is:

# systemctl restart sshd

Log out of your user’s ssh session. Then try to ssh back in.

If it asks you for a password, something went wrong, but if it doesn’t, you have successfully set up your ssh keys for logging into the server.

Disable Password Login on ssh

Once you have completed the above, you can now try to disable password login on ssh.

On your root ssh session, open the /etc/ssh/sshd_config file in a text editor. Find the line that says something like:

ChallengeResponseAuthentication no

Make sure it looks exactly like the above (i.e., it’s not commented out and it does not say yes).

Then, find the line that says something like:

PasswordAuthentication yes

Make sure it says:

PasswordAuthentication no

and that it is not commented out.

Save and close the file, and then restart the ssh service. Again, on Arch Linux, the command is:

# systemctl restart sshd

Log out of your user’s ssh session and try logging back in again. If it fails, something went wrong, and you should not continue until it works.

Disable root Login on ssh

When you have completed the above, in your root ssh session, open the /etc/ssh/sshd_config file in a text editor again and find the line that says something like:

PermitRootLogin yes

Make sure it says:

PermitRootLogin no

and that it is not commented out.

Save and close the file, and restart the ssh service.

Do NOT close your root ssh session yet! First, log out of your user’s session, and log back in. If you can get in, and successfully get a root shell from running the following command:

$ sudo su -

then you are ready to log out of your root ssh session.

Switch ssh Port

It is a good idea to not use the standard port 22 for ssh, so we will fix that.

In an ssh session, run the following command:

$ sudo su -

When you have a root shell, open the /etc/ssh/sshd_config file again and find the line that looks like:

# Port 22

Uncomment it and set it to the random port number you picked in the firewall section.

Then restart the ssh service.

Now, do NOT log off your current ssh session. Instead, open a new session with the following command:

$ ssh -p <port_number> <username>@<server_ip>

making sure to replace <port_number>, <username>, and <server_ip> appropriately.

If you successfully log in, you can log out the first session. From now on, that will be the command you need to run to ssh into your server. I would suggest setting an alias.

Set Up an ssh Tarpit

Next, we are going to set up endlessh as an ssh tarpit on port 22.

Log into your server by ssh, install git, and clone the endlessh repo. You should be able to build and install it with these commands:

$ make
$ sudo make install

Then set up endlessh as a service with your init system. On Arch, this is done by setting up a systemd service.

On Arch, create and open the file /etc/systemd/system/endlessh.service. Put the following contents into it:

[Unit]
Description = an ssh tarpit
After = network.target

[Service]
ExecStart = /usr/local/bin/endlessh -s

[Install]
WantedBy = multi-user.target

Then save and close the file. Then create and open /etc/endlessh/config and insert the following contents:

# The port on which to listen for new SSH connections.
Port 22

# The endless banner is sent one line at a time. This is the delay
# in milliseconds between individual lines.
Delay 10000

# The length of each line is randomized. This controls the maximum
# length of each line. Shorter lines may keep clients on for longer if
# they give up after a certain number of bytes.
MaxLineLength 32

# Maximum number of connections to accept at a time. Connections beyond
# this are not immediately rejected, but will wait in the queue.
MaxClients 4096

# Set the detail level for the log.
#   0 = Quiet
#   1 = Standard, useful log messages
#   2 = Very noisy debugging information
LogLevel 1

# Set the family of the listening socket
#   0 = Use IPv4 Mapped IPv6 (Both v4 and v6, default)
#   4 = Use IPv4 only
#   6 = Use IPv6 only
BindFamily 0

These, plus the -s arg to endlessh makes it listen on port 22 and sends the logs to syslog.

Enable and start the service with the following commands:

$ sudo systemctl enable endlessh
$ sudo systemctl start endlessh

You can run the following command to see if the service started successfully:

sudo journalctl -xe

Setting Up a Server Program

The next thing to do is to set up your server program. I use nginx, and I used a service, nginxconfig.io, to generate the configs for nginx and my websites.

Unfortunately, because my websites are simple, I can’t help you with your own website; it could be entirely different. I just suggest taking the time to learn how to configure your server program of choice.

However, for the sake of the visitors to your website, be sure to set up https! That was why we left port 443 open. (Port 80 is for insecure http.) In fact, if you can set up your server to automatically redirect all http traffic to https, you should.

Miscellaneous

These are items I added after this post was published:

• Add fail2ban to ensure attackers don’t get many chances.

Conclusion

After all this, you should have a server that will withstand attempts by the hurricane of relatively unskilled bad actors on the Internet. However, it probably will not withstand attempts by three-letter agencies.

How to do that is well above my skill level.